Unmanned aerial vehicle cluster formation decision-making and hunting method based on reinforcement learning

Abstract

The invention relates to an unmanned aerial vehicle cluster formation decision making and hunting method based on reinforcement learning, and belongs to the technical field of multi-agent formation control and decision making. The expected position corresponding to each agent and the control instruction of the dynamic model of the multi-agent are obtained, the central node does not need to quickly realize the expected formation configuration given by the decision-making system in a distributed mode, and the formation control problem of the multi-agent is solved; secondly, the formation of the multiple agents can be visually manipulated to adapt to dynamically changing tasks, and the formation manipulation problem of the multiple agents is solved; according to an actual confrontation task, learning and evolutionary online autonomous decision-making of the multi-agent system is realized, and the formation autonomous decision-making problem of the multi-agent system is solved; and finally, the interpretability and safety of learning decision are enhanced by utilizing a control model mechanism, and the interpretability and safety problems in the formation strategy learning process are considered.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent formation control and decision-making, and in particular to a UAV cluster formation decision-making and encirclement method based on reinforcement learning. Background Art

[0002] Clustering and intelligence are important trends in the development of future system confrontation and multi-platform combat technology. The core technology involved is multi-agent control and decision-making. Multi-agent control and decision-making technology can organize a group of individuals with simple structures and limited capabilities to evolve through reasonable information interaction, and emerge complex organizational structures and intelligent behaviors, thereby completing tasks that traditional single platforms cannot complete.

[0003] The realization of these complex tasks undoubtedly depends on the formation decision and control methods of multi-agent systems. The main problem that formation decision solves is what kind of formation configuration the cluster should need in order to complete a specific task; while formation control studies how to achieve the desired formation configuration given by the formation decision system safely (collision avoidance within the formation, environmental threat avoidance), controllable (formation shape maintenance, communication connection maintenance), and efficient (fastest convergence speed, optimal position allocation). At present, there are relatively many studies on the distributed control problem of drone clusters, while there are fewer studies on distributed formation decision problems, which are of course more challenging. Considering the dynamic network topology of drone clusters, the application of intelligent learning algorithms for formation decision-making needs to solve the "dimensionality curse" and dynamic local information dimension problems, and there are also few studies on combining intelligent learning algorithms such as supervised learning and reinforcement learning with collaborative control mechanism models to improve the guidance and interpretability of group decision results. Considering the dynamic expansion requirements of networked multi-agent systems and the complexity and variability of tasks, formation control and decision-making methods should have typical characteristics such as distributed self-organization, scalable number of platforms, dynamic reconfiguration of network topology, and learnable and evolutionary strategies. This undoubtedly makes the distributed formation control and decision-making of multi-agents a very challenging technical problem. Summary of the invention

[0004] In view of the above problems, the present invention provides a UAV swarm formation decision and encirclement method based on reinforcement learning. The present invention establishes a consistency negotiation update equation and a multi-agent dynamics model, obtains the expected position corresponding to each agent and the control instructions of the multi-agent dynamics model, and does not require a central node to quickly implement the expected formation configuration given by the decision system in a distributed manner, thereby solving the formation control problem of the multi-agent; secondly, the formation of the multi-agent can be intuitively manipulated to adapt to dynamically changing tasks, thereby solving the formation manipulation problem of the multi-agent; according to actual confrontation tasks, such as the encirclement problem, the learnable and evolutionary online autonomous decision-making of the multi-agent system is realized, thereby solving the formation autonomous decision-making problem of the multi-agent; finally, the control model mechanism is used to enhance the interpretability and security of learning decisions, and the interpretability and security issues in the formation strategy learning process are considered.

[0005] The present invention provides a UAV swarm formation decision and encirclement method based on reinforcement learning, comprising:

[0006] Step 1: determining formation parameters of a multi-agent formation, and obtaining a reference position of the multi-agent formation based on the formation parameters of the multi-agent formation;

[0007] Preferably, the formation parameter expression of the multi-agent formation is:

[0008]

[0009] Among them, θ is the formation parameter of the multi-intelligence formation, p c ∈ 2 Indicates the center position of the formation, which is a real number; is the formation orientation, ζ is the neighbor distance, β is the deformation angle, and T is the transposition;

[0010] Preferably, the multi-agent formation includes translation, rotation, scaling and deformation actions; the multi-agent formation has a better ability to cope with changes in the environment or tasks.

[0011] In one embodiment of the present invention, for a roundup task, the multi-agent formation is an open encirclement formation and / or a closed encirclement formation;

[0012] Preferably, the expression of the reference position of the multi-agent formation in step 1 is:

[0013]

[0014] Among them, p i is the reference position of the ith agent, q i is the displacement of the reference position of the ith agent relative to the reference position of the first agent.

[0015] When i=1, qi is the displacement of the reference position of the ith agent relative to the reference position of the first agent;

[0016] When i≠1, i=2,3…I, I is the total number of agents, ψ i is the direction of the reference position of the ith agent relative to the reference position of the i-1th agent.

[0017] Step 2: Establish a multi-agent formation parameter decision model; set constraints for the multi-agent formation parameter decision model;

[0018] Obtain local observation information of multi-agent formations;

[0019] Based on the multi-agent formation parameter decision model, constraint conditions and local observation information of the multi-agent formation, a consensus negotiation update equation is established;

[0020] Preferably, the consistency negotiation update equation expression in step 2 is:

[0021]

[0022] in, is the updated formation parameter of the ith agent, π i Represents the decision model of the formation parameters of the ith agent. According to the local observation information of the ith agent, it outputs the update rate of the formation parameters, s a is the state of the attacker; o(·) is the local observation information function of the agent; c θ is the weight coefficient of consistency negotiation, is the estimated formation parameter of the ith agent, is the estimated formation parameter of the jth agent adjacent to the ith agent, is the neighbor set of the ith agent as the defender.

[0023] Preferably, the constraint condition in step 2 is expressed as:

[0024]

[0025] Among them, θ * is the optimal value of the formation parameter if and only if When the equality sign holds, is a collection of defensive agents.

[0026] Step 3: updating the formation parameters of each agent based on the consistency negotiation update equation to obtain updated formation parameters of each agent;

[0027] Based on the updated formation parameters of each agent, the corresponding expected position is obtained;

[0028] Step 4: Establish a multi-agent dynamics model;

[0029] Setting control instructions for the multi-agent dynamics model based on the desired positions of each agent;

[0030] Preferably, the multi-agent dynamics model in step 4 includes a dynamics model of a fully constrained multi-agent and a dynamics model of a non-fully constrained multi-agent;

[0031] Further, the control instructions of the dynamics model of the nonholonomic constrained multi-agent include linear velocity instructions and angular velocity instructions;

[0032] Furthermore, the expression of the dynamic model of the complete constrained multi-agent is:

[0033]

[0034] in, is the velocity vector of the position change of the ith agent, v i is the linear velocity of the ith agent, C d is the resistance coefficient of the agent, u i is the control instruction of the ith agent, Indicates that the control instruction of the i-th agent has a maximum value constraint, Represents the maximum value of the agent's control input.

[0035] Furthermore, the expression of the dynamic model of the non-holonomic constrained multi-agent is:

[0036]

[0037] Among them, v i is the linear velocity of the ith agent, ω i is the angular velocity of the i-th agent, is the projection of the position change speed of the ith agent in the x direction, is the projection of the position change speed of the i-th agent in the y direction, is the speed of the angle change of the i-th agent, The direction of the i-th agent.

[0038] Preferably, the expression of the control instruction of the multi-agent dynamics model in step 4 is:

[0039]

[0040] Among them, k p is the weight coefficient corresponding to the position error, is the expected position of the ith agent as the defender, k v is the weight coefficient corresponding to the speed error, is the movement speed of the center of the formation.

[0041] Step 5: Establish a reinforcement learning framework and a learning environment model; embed the learning environment model into the reinforcement learning framework to obtain a reinforcement learning method;

[0042] Determine learning samples for reinforcement learning methods;

[0043] Determine the loss function of the multi-agent formation parameter decision model;

[0044] Using a reinforcement learning method, a learning sample and a loss function of the multi-agent formation parameter decision model to train the multi-agent formation parameter decision model, and obtain a final multi-agent formation parameter decision model;

[0045] Preferably, the learning environment model includes action and state space, reward return and learning objectives;

[0047] Furthermore, the action space described in step 5 is the update rate of each component of the formation parameter θ ;

[0048] The state space is selected based on local observations of multiple agents, including estimating formation parameters and attacker status s a ;

[0049] The reward return is the reward brought by each state transfer; the reward return includes the final reward and the immediate reward;

[0050] The instant rewards include enveloping angle rewards, alignment angle rewards, average distance rewards and control energy rewards;

[0051] Furthermore, the learning objectives include model learning objectives, evaluation learning objectives and action learning objectives;

[0052] Furthermore, the expression of the learning objective is:

[0053]

[0054] Among them, ω * is the model learning objective, ψ * To evaluate the learning objectives, φ * is the action learning target, and t represents the learning time.

[0055] Furthermore, the learning samples include: model learning samples, evaluation learning samples and action learning samples;

[0056] Furthermore, the model learning sample expression is:

[0057]

[0058] in, is the model replay sample corresponding to the kth reference position of the attacker agent, For a j The state of the attacker agent at the kth reference position, For a j The displacement of an attacking agent from the kth reference position to the nearest attacking agent, For a j The minimum displacement of an attacking agent from the kth reference position to the encirclement formation, For a j The state of an attacker agent at the k+1th reference position.

[0059] The expression for evaluating the learning sample is:

[0060]

[0061] in, is the evaluation replay sample corresponding to the kth reference position of the attacker agent, q is the evaluation replay, s k is the kth reference position of the attacker agent, a k is the action of the attacker agent at the kth reference position, s k+1 is the k+1th reference position of the attacker agent, R(s k ,a k ) is the reward for the state transition of the attacker agent to the kth reference position.

[0062] The expression of the action learning sample is:

[0063]

[0064] in, is the action replay sample of the kth reference position of the attacker agent, is the first action of the attacker agent at the kth reference position.

[0065] Furthermore, the loss function of the formation parameter decision model includes a model learning loss function, an evaluation learning loss function and an action learning loss function.

[0066] Furthermore, the model learning loss function expression is:

[0067]

[0068] Among them, J m is the loss function of the model learning, is any sample in the mini-batch of samples learned by the model, corresponding to the reference position of the attacker agent, is a small batch sample, For the sample, is collected from the actual environment j The state of the attacker agent at the k+1th reference position;

[0069] Furthermore, the loss function expression of the evaluation learning is:

[0070]

[0071] Among them, J q To evaluate the learning loss function, is a small batch of samples for evaluating learning, the samples correspond to the reference position, q ψ Target evaluation network, γ is the discount rate used to balance short-term and long-term rewards, To evaluate the network for independent goals, For independent target strategy networks.

[0072] Furthermore, the expression of the action learning loss function is:

[0073]

[0074] in, is the policy gradient loss function, is a small batch of policy gradient samples corresponding to the reference position, π φ The target policy network.

[0075] Step 6: When performing the encirclement and capture mission, the multi-agent acts as a defender, encircles the attacking cluster based on the final multi-agent formation parameter decision model, and makes the attacking cluster enter the safe area.

[0076] Preferably, the roundup task in step 6 includes a roundup phase and a transportation phase;

[0077] Preferably, the specific steps of encircling the attacking cluster based on the final formation parameter decision model include:

[0078] Step 61, when the distance between the attacking clusters is relatively far, multiple agents enter the round-up phase; in the round-up phase, each agent inputs the final formation parameter decision model respectively, and outputs the formation parameters of each agent;

[0079] Step 62: Neighboring agents negotiate formation parameters based on the consensus negotiation update equation to obtain updated formation parameters of each agent.

[0080] Step 63: input the updated formation parameters of each agent into the multi-agent dynamics model, output the control instructions of each agent; and obtain the corresponding agent state based on the control instructions of each agent;

[0081] Step 64: the attacking cluster makes corresponding state transitions according to the states of each agent to obtain an updated state of the attacking cluster;

[0082] Step 65, determining whether the updated state of the attacking cluster is enclosed in the environment model, if so, proceeding to the next step, if not, repeating steps 62-64 until the updated state of the attacking cluster is enclosed in the environment model;

[0083] Step 66: After the updated state of the attacking cluster is enclosed in the environmental model, it enters the transportation phase; based on the speed obstacle method, the vector of the center speed of the multi-agent formation is optimized to maintain the closed encirclement formation while avoiding environmental obstacles;

[0084] Furthermore, the transportation phase in step 66 includes multiple transportation tasks; the multiple transportation tasks include avoiding environmental obstacles, preventing the attacking cluster from escaping, and transporting the attacking cluster to a designated safe area.

[0085] Compared with the prior art, the present invention has at least the following beneficial effects:

[0086] (1) This invention proposes a parametric formation control method that can flexibly manipulate the formation of multi-agents, further enhancing the application potential of multi-agent systems;

[0087] (2) The present invention establishes a multi-agent formation decision framework based on parameterized formations, which can achieve more efficient and generalizable multi-agent roundups;

[0088] (3) The present invention designs a model-based reinforcement learning method that combines the idea of ​​model predictive control for roundup strategy learning, which has higher sample efficiency and better interpretability and guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0090] Figure 1 (a)-(b) are schematic diagrams of problem scenario settings in the capture phase and the transportation phase in an embodiment of the present invention;

[0091] Figure 2Schematic diagram of the overall framework of the multi-agent formation decision and round-up method in an embodiment of the present invention;

[0092] Figure 3 (a)-(b) are schematic diagrams of parameterized encirclement formation designs of open encirclement formation and closed encirclement formation in embodiments of the present invention;

[0093] Figure 4 A schematic diagram of a multi-agent roundup based on formation parameter decision-making in an embodiment of the present invention;

[0094] Figure 5 Schematic diagram of a formation parameter decision algorithm architecture based on model reinforcement learning in an embodiment of the present invention;

[0095] Figure 6 A schematic diagram of a schematic diagram of a non-escapable angle in an embodiment of the present invention;

[0096] Figure 7 A schematic diagram of an alignment angle schematic diagram in an embodiment of the present invention;

[0097] Figure 8 A schematic diagram of the algorithm execution and sample collection process of the learning method proposed in the embodiment of the present invention;

[0098] Fig. 9 (a)-(b) are schematic diagrams of a strategy network loss value curve and an evaluation network loss value curve during training in an embodiment of the present invention;

[0099] Fig.10 (a)-(b) are schematic diagrams of cumulative discounted rewards and winning rate curves during the training process in an embodiment of the present invention;

[0100] Fig.11 (a)-(b) are schematic diagrams of simulation comparison and verification results of the round-up phase in an embodiment of the present invention and the traditional model predictive control;

[0101] Fig.12 (a)-(b) are schematic diagrams of formation parameter decision instructions and formation parameter change curves during the simulation process in an embodiment of the present invention;

[0102] Fig.13 (a)-(b) are schematic diagrams of speed curves of the defender and the attacker during the simulation process in an embodiment of the present invention;

[0103] Fig.14 (a)-(d) are schematic diagrams of numerical generalization simulation results in an embodiment of the present invention;

[0104] Fig.15 A schematic diagram of the composition of the experimental platform in an embodiment of the present invention;

[0105] Fig.16 A schematic diagram of a multi-robot capture experiment result in an embodiment of the present invention;

[0106] Fig.17 (a)-(c) are schematic diagrams of the robot many-to-many driving experiment data in an embodiment of the present invention. DETAILED DESCRIPTION

[0107] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0108] A specific embodiment of the present invention, as Figure 1-17 , a UAV swarm formation decision and encirclement method based on reinforcement learning is disclosed;

[0109] In order to illustrate the effectiveness of the method proposed in the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment, and the specific implementation steps are as follows:

[0110] The present invention provides a UAV swarm formation decision and encirclement method based on reinforcement learning, comprising:

[0111] Step 1: determining formation parameters of a multi-agent formation, and obtaining a reference position of the multi-agent formation based on the formation parameters of the multi-agent formation;

[0112] Preferably, the formation parameter expression of the multi-agent formation is:

[0113]

[0114] Among them, θ is the formation parameter of the multi-intelligence formation, p c ∈ 2 Indicates the center position of the formation, which is a real number; is the formation orientation, ζ is the neighbor distance, β is the deformation angle, and T is the transposition;

[0115] Preferably, the multi-agent formation includes translation, rotation, scaling and deformation actions; the multi-agent formation has a better ability to cope with changes in the environment or tasks.

[0116] In one embodiment of the present invention, for a roundup task, the multi-agent formation is an open encirclement formation and / or a closed encirclement formation;

[0117] Preferably, the expression of the reference position of the multi-agent formation in step 1 is:

[0118]

[0119] Among them, p i is the reference position of the ith agent, q i is the displacement of the reference position of the ith agent relative to the reference position of the first agent.

[0120] When i=1, q i is the displacement of the reference position of the ith agent relative to the reference position of the first agent;

[0121] When i≠1, i=2,3…I, I is the total number of agents, ψ i is the direction of the reference position of the ith agent relative to the reference position of the i-1th agent.

[0122] Step 2: Establish a multi-agent formation parameter decision model; set constraints for the multi-agent formation parameter decision model;

[0123] Obtain local observation information of multi-agent formations;

[0124] Based on the multi-agent formation parameter decision model, constraint conditions and local observation information of the multi-agent formation, a consensus negotiation update equation is established;

[0125] Preferably, the consistency negotiation update equation expression in step 2 is:

[0126]

[0127] in, is the updated formation parameter of the ith agent, π i Represents the decision model of the formation parameters of the ith agent. According to the local observation information of the ith agent, it outputs the update rate of the formation parameters, s a is the state of the attacker; o(·) is the local observation information function of the agent; c θ is the weight coefficient of consistency negotiation, is the estimated formation parameter of the ith agent, is the estimated formation parameter of the jth agent adjacent to the ith agent, is the neighbor set of the ith agent as the defender.

[0128] Preferably, the constraint condition in step 2 is expressed as:

[0129]

[0130] Among them, θ *is the optimal value of the formation parameter if and only if When the equality sign holds, is a collection of defensive agents.

[0131] Step 3: updating the formation parameters of each agent based on the consistency negotiation update equation to obtain updated formation parameters of each agent;

[0132] Based on the updated formation parameters of each agent, the corresponding expected position is obtained;

[0133] Step 4: Establish a multi-agent dynamics model;

[0134] Setting control instructions for the multi-agent dynamics model based on the desired positions of each agent;

[0135] Preferably, the multi-agent dynamics model in step 4 includes a dynamics model of a fully constrained multi-agent and a dynamics model of a non-fully constrained multi-agent;

[0136] Further, the control instructions of the dynamics model of the nonholonomic constrained multi-agent include linear velocity instructions and angular velocity instructions;

[0137] Furthermore, the expression of the dynamic model of the complete constrained multi-agent is:

[0138]

[0139] in, is the velocity vector of the position change of the ith agent, v i is the linear velocity of the ith agent, C d is the resistance coefficient of the agent, u i is the control instruction of the ith agent, Indicates that the control instruction of the i-th agent has a maximum value constraint, Represents the maximum value of the agent's control input.

[0140] Furthermore, the expression of the dynamic model of the non-holonomic constrained multi-agent is:

[0141]

[0142] Among them, v i is the linear velocity of the ith agent, ω i is the angular velocity of the i-th agent, is the projection of the position change speed of the ith agent in the x direction, is the projection of the position change speed of the i-th agent in the y direction, is the speed of the angle change of the i-th agent, The direction of the i-th agent.

[0143] Preferably, the expression of the control instruction of the multi-agent dynamics model in step 4 is:

[0144]

[0145] Among them, k p is the weight coefficient corresponding to the position error, is the expected position of the ith agent as the defender, k v is the weight coefficient corresponding to the speed error, is the movement speed of the center of the formation.

[0146] Step 5: Establish a reinforcement learning framework and a learning environment model; embed the learning environment model into the reinforcement learning framework to obtain a reinforcement learning method;

[0147] Determine learning samples for reinforcement learning methods;

[0148] Determine the loss function of the multi-agent formation parameter decision model;

[0149] Using a reinforcement learning method, a learning sample and a loss function of the multi-agent formation parameter decision model to train the multi-agent formation parameter decision model, and obtain a final multi-agent formation parameter decision model;

[0150] Preferably, the learning environment model includes action and state space, reward return and learning objectives;

[0151] Furthermore, the action space described in step 5 is the update rate of each component of the formation parameter θ

[0152] The state space is selected based on local observations of multiple agents, including estimating formation parameters and attacker status s a ;

[0153] The reward return is the reward brought by each state transfer; the reward return includes the final reward and the immediate reward;

[0154] The instant rewards include enveloping angle rewards, alignment angle rewards, average distance rewards and control energy rewards;

[0155] Furthermore, the learning objectives include model learning objectives, evaluation learning objectives and action learning objectives;

[0156] Furthermore, the expression of the learning objective is:

[0157]

[0158] Among them, ω * is the model learning objective, ψ * To evaluate the learning objectives, φ * is the action learning target, and t represents the learning time.

[0159] Furthermore, the learning samples include: model learning samples, evaluation learning samples and action learning samples;

[0160] Furthermore, the model learning sample expression is:

[0161]

[0162] in, is the model replay sample corresponding to the kth reference position of the attacker agent, For a j The state of the attacker agent at the kth reference position, For a j The displacement of an attacking agent from the kth reference position to the nearest attacking agent, For a j The minimum displacement of an attacking agent from the kth reference position to the encirclement formation, For a j The state of an attacker agent at the k+1th reference position.

[0163] The expression for evaluating the learning sample is:

[0164]

[0165] in, is the evaluation replay sample corresponding to the kth reference position of the attacker agent, q is the evaluation replay, s k is the kth reference position of the attacker agent, a k is the action of the attacker agent at the kth reference position, s k+1 is the k+1th reference position of the attacker agent, R(s k ,a k ) is the reward for the state transition of the attacker agent to the kth reference position.

[0166] The expression of the action learning sample is:

[0167]

[0168] in, is the action replay sample of the kth reference position of the attacker agent, is the first action of the attacker agent at the kth reference position.

[0169] Furthermore, the loss function of the formation parameter decision model includes a model learning loss function, an evaluation learning loss function and an action learning loss function.

[0170] Furthermore, the model learning loss function expression is:

[0171]

[0172] Among them, J m is the loss function of the model learning, is any sample in the mini-batch of samples learned by the model, corresponding to the reference position of the attacker agent, is a small batch sample, For the sample, is collected from the actual environment j The state of the attacker agent at the k+1th reference position;

[0173] Furthermore, the loss function expression of the evaluation learning is:

[0174]

[0175] Among them, J q To evaluate the learning loss function, is a small batch of samples for evaluating learning, the samples correspond to the reference position, q ψ Target evaluation network, γ is the discount rate used to balance short-term and long-term rewards, To evaluate the network for independent goals, For independent target strategy networks.

[0176] Furthermore, the expression of the action learning loss function is:

[0177]

[0178] in, is the policy gradient loss function, is a small batch of policy gradient samples corresponding to the reference position, π φ The target policy network.

[0179] Step 6: When performing the encirclement and capture mission, the multi-agent acts as a defender, encircles the attacking cluster based on the final multi-agent formation parameter decision model, and makes the attacking cluster enter the safe area.

[0180] Preferably, the roundup task in step 6 includes a roundup phase and a transportation phase;

[0181] Preferably, the specific steps of encircling the attacking cluster based on the final formation parameter decision model include:

[0182] Step 61, when the distance between the attacking clusters is relatively far, multiple agents enter the round-up phase; in the round-up phase, each agent inputs the final formation parameter decision model respectively, and outputs the formation parameters of each agent;

[0183] Step 62: Neighboring agents negotiate formation parameters based on the consensus negotiation update equation to obtain updated formation parameters of each agent.

[0184] Step 63: input the updated formation parameters of each agent into the multi-agent dynamics model, output the control instructions of each agent; and obtain the corresponding agent state based on the control instructions of each agent;

[0185] Step 64: the attacking cluster makes corresponding state transitions according to the states of each agent to obtain an updated state of the attacking cluster;

[0186] Step 65, determining whether the updated state of the attacking cluster is enclosed in the environment model, if so, proceeding to the next step, if not, repeating steps 62-64 until the updated state of the attacking cluster is enclosed in the environment model;

[0187] Step 66: After the updated state of the attacking cluster is enclosed in the environmental model, it enters the transportation phase; based on the speed obstacle method, the vector of the center speed of the multi-agent formation is optimized to maintain the closed encirclement formation while avoiding environmental obstacles;

[0188] Furthermore, the transportation phase in step 66 includes multiple transportation tasks; the multiple transportation tasks include avoiding environmental obstacles, preventing the attacking cluster from escaping, and transporting the attacking cluster to a designated safe area.

[0189] The present invention also includes step 7, simulation platform and simulation results;

[0190] 71. Simulation scene setting

[0191] The simulation parameter settings are shown in the following table; the parameter settings of the proposed reinforcement learning method are shown in Table 1; simulation training is carried out with 8 defenders and 3 attackers as the benchmark scenario.

[0192] Table 1 Simulation scenario parameter settings

[0193]

[0194] Table 2 The parameter settings in the learning algorithm are as follows

[0195]

[0196] 72. Simulation training results

[0197] The policy network is a traditional fully connected neural network, which consists of a normalization layer, two hidden layers and an anti-normalization layer, and finally outputs the updated formation parameters.

[0198] During the simulation training process, the loss function value curve of the formation parameter decision model is as follows Fig. 9 As shown. As the training progresses, the loss function value of the formation parameter decision model gradually decreases. It gradually decreases from a maximum of about 0.4 to below 0.05, which shows that the direct output of the formation parameter decision model gradually converges to the action output after PSO algorithm optimization. In the subsequent simulation after the training is completed, if there are high real-time requirements for control instruction decisions, the action optimization process under the model predictive control framework in the execution phase is not necessary, and the formation parameter decision model can also be used directly to make formation parameter update decisions. Subsequent simulations and experiments will verify this point.

[0199] Fig.10 The discounted cumulative reward and win rate curve for each round in the training process are given. The cumulative discounted reward is an important indicator to measure the performance of the strategy, and maximizing the cumulative discounted reward is also the starting point for the design of the reinforcement learning algorithm. During the training process, a penalty of -1 is added to each immediate reward to guide the agent to complete the capture of the target as soon as possible. Therefore, when the capture task fails or takes a long time, the cumulative discounted reward will show a large negative value; as the win rate continues to increase and the capture time continues to decrease during the training process, the cumulative discounted reward will gradually increase. Fig.10 (a) The cumulative discounted rewards shown are still processed by moving exponential average with a smoothing factor of 0.9.

[0200] Another indicator that can more directly reflect the performance of a strategy is the task winning rate. Fig.10 (b) The winning rate shown is obtained by moving window averaging, with a window size of 20, that is, each point on the curve is the winning rate calculated by the winning and losing results of the next 20 rounds. At the beginning, since no data has been collected for learning and training, the strategy at this stage is equivalent to the traditional model predictive control method without the learning evolution mechanism. It can be seen that the winning rate is about 60% at this time, indicating that the learning method proposed in this chapter can provide acceptable strategy performance at the beginning of training instead of relying entirely on random trial and error for learning. As the simulation training process proceeds, the model network, evaluation network and policy network are gradually optimized, and the winning rate can be stably maintained at 100% within 40 rounds.

[0201] 73. Simulation verification of comparative method

[0202] The proposed formation decision method based on reinforcement learning is used as a comparison method for the control of the benchmark formation parameter decision model. The simulation scenario uses a total of 8 defenders and 3 attackers, and there are two polygonal obstacle areas and one circular obstacle area in the environment (defenders cannot pass but attackers can); the defender needs to surround all attackers in a closed formation and transport them all to a safe area for the mission to be successful.

[0203] Fig.11 The simulation results of the two methods are given, where the large figure on the left is the trajectory diagram of the entire process, and the small figures on the right are snapshots of the scene at different times. Fig.11 (a) is the simulation result of using the traditional model predictive control method to make formation parameter decisions. Due to the obvious error in the prediction model of the attacker's state transition and insufficient time for the PSO algorithm to fully optimize the decision-making action, in this simulation, although the defenders were able to present a strategy of collaboratively trapping the attackers in an encirclement formation, the defenders formed an encirclement trend too early, causing two of the attackers to successfully escape the encirclement formation and eventually enter the protection area, resulting in the failure of our mission. Fig.11 (b) The simulation result of using the policy network trained by the proposed method to make formation parameter decisions. In this simulation, the defender can use the characteristics of the attackers approaching the protection area to lure them further into the encirclement formation and then quickly close the formation, eventually successfully trapping all the attackers inside the encirclement formation.

[0204] Fig.12 The decision instructions and formation parameter change curve of the formation parameter decision model in the simulation process are given. All defenders estimate the formation parameter θ$ based on their own local observations. i and attacker states, using an independent formation parameter decision model for decision making. There are random differences in the initial formation parameters of each defender, so their decision instructions are not exactly the same at the beginning. However, this difference is quickly eliminated as the negotiation of the formation parameters converges. During the simulation, the translation instruction fluctuates greatly when the attacking cluster is close enough and is about to complete the encirclement; after the encirclement is completed, the translation instruction guides the encircling formation to move in a uniform straight line to the safe area. The deformation instruction increases slowly and then quickly in the early defense stage, thereby completing the encirclement of the attacking cluster. After that, the deformation parameter β remains unchanged at the maximum value.

[0205] Fig.13The speed change curves of all defenders and all attackers during the collaborative expulsion simulation are shown. The speed of the defenders never exceeds their maximum speed of 2m / s, while the speed of the attackers can reach 2.5m / s. During the encirclement process, the speed change process of different defenders is always continuous and stable, and in the expulsion phase after the encirclement is completed, the speed of all defenders is basically the same. After the attackers are enclosed in the encirclement formation, the speed amplitude and direction fluctuate violently. This is because the offensive cluster is still trying to break free from the encirclement formation and go to the protection area while being forced to transport to the safe area.

[0206] 74. Quantitative generalization simulation verification

[0207] This method makes decisions based on formation parameters without directly using global states, so it is suitable for distributed systems, and the learned formation parameter decisions can be flexibly applied to a variety of capture and expulsion tasks with different numbers of defenders and attackers. This reuse of formation parameter decisions not only does not require redesign of the strategy scheme, but the parameters of the strategy network can also be used directly without retraining. The figure below shows the performance of the strategy trained using 8 defenders and 3 attackers, directly applied to other capture and expulsion scenarios with different numbers of enemies and friends.

[0208] Fig.14 (a)-(d) correspond to the task scenarios of a decrease in the number of attackers (8 vs. 2), an increase in the number of attackers (8 vs. 4), a decrease in both the number of defenders and attackers (6 vs. 2), and an increase in both the number of defenders and attackers (10 vs. 5). In the simulation, each defender uses an independent pre-trained network to make decisions on the formation parameter update based on its own observations. The instructions after the policy network decision no longer go through the PSO algorithm optimization step and are directly used for the formation parameter update. The simulation results show that the pre-trained network can successfully complete the task for task scenarios with different numbers of enemies and friends. In the encirclement phase, the attacking cluster can be completely confined to the encirclement formation; in the transportation phase, the encirclement formation can be kept intact and all attackers can be successfully transported to the designated safe area.

[0209] Fixed-dimensional decision instructions can realize the formation control of different numbers of intelligent agents, which verifies that the method proposed in the present invention is independent of the number of intelligent agents and has good scalability and application prospects.

[0210] The present invention further comprises step 8, robot experimental verification: designing a robot physical experiment to verify the effectiveness of the method of the present invention;

[0211] 81. Introduction to the experimental system

[0212] More mobile robots were used in the experiment, including 6 defense robots and 2 attack robots. The motion capture system was used for positioning, and the command was distributed based on the ROS2 robot operating system. In order to make the encirclement formation more intuitive, adjacent defense robots were connected with elastic ropes, which can be seen in the figure. Fig.15 The experimental program was written in Python, the reinforcement learning library was TensorFlow 2, the computer system was Ubuntu 18.04, and the robot operating system version was ROS2foxy.

[0213] 82. Multi-robot roundup experiment

[0214] This part of the work designs a physical experiment to verify the application of the proposed distributed formation decision method in the actual multi-robot encirclement and expulsion problem. The mobile robot is a wheeled robot with non-holonomic constraints, and its control instructions are linear velocity and angular velocity. The maximum linear velocity and maximum angular velocity of the defender are set to v d =0.15m / s and ω d =1.2rad / s; for the attacker, v a =0.2m / s and ω a =3.6rad / s.

[0215] like Fig.16 As shown in the figure, the six robots marked in red below are defenders; the two robots with blue lights above are attackers; the cyan area below represents the protection area; the green area above represents the safe area; the defenders are connected by a rope net. Two attackers tried to invade the protection area below, but were blocked by the defender formation. The defenders updated the formation parameters through distributed decision-making and formed an encirclement formation around the attackers. Then, all attackers were completely confined inside the closed encirclement formation through formation deformation. After that, the transportation phase began, and the defenders transported all attackers to the safe area by moving the encirclement formation.

[0216] Fig.17 The data analysis results of the entire experiment are given. Sub-figure (a) is the decision instruction for the update rate of formation parameters of all defenders. It can be seen that except for the slight differences in the decisions made by different defenders for the formation parameter update when the formation parameters did not converge at the beginning, the subsequent decision results are basically consistent. Sub-figure (b) is the formation parameters understood by each defender during the experiment. In sub-figure (c), from top to bottom are the defender linear speed v d 、Defender angular velocity ω d , attacker linear speed v a , attacker's angular velocity ω a and the encirclement formation convergence error e fThe defender's speed never exceeds 0.15 m / s and converges to zero after reaching the safe zone. It can be seen from the experiment that the method of the present invention can successfully complete the task.

[0217] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A UAV swarm formation decision and encirclement method based on reinforcement learning, characterized in that: include: Step 1: determining formation parameters of a multi-agent formation, and obtaining a reference position of the multi-agent formation based on the formation parameters of the multi-agent formation; Step 2: Establish a multi-agent formation parameter decision model to obtain local observation information of the multi-agent formation; Based on the multi-agent formation parameter decision model and the local observation information of the multi-agent formation, a consensus negotiation update equation is established; Step 3: updating the formation parameters of each intelligent agent based on the consistency negotiation update equation to obtain the updated formation parameters of each intelligent agent; and obtaining the corresponding expected position based on the updated formation parameters of each intelligent agent; Step 4: Establish a multi-agent dynamics model; set control instructions for the multi-agent dynamics model based on the expected positions corresponding to each agent; Step 5: Design reinforcement learning methods and determine learning samples; Determine the loss function of the multi-agent formation parameter decision model; use the reinforcement learning method, the learning sample and the loss function of the multi-agent formation parameter decision model to train the multi-agent formation parameter decision model to obtain the final multi-agent formation parameter decision model; Step 6: When performing the round-up mission, the attacking cluster is rounded up based on the final multi-agent formation parameter decision model, and the attacking cluster is brought into a safe area.

2. The drone swarm formation decision and encirclement method according to claim 1 is characterized in that: The expression of the reference position of the multi-agent formation in step 1 is: Among them, p i is the reference position of the ith agent, p c Indicates the center position of the formation, q i is the displacement of the reference position of the ith agent relative to the reference position of the first agent, i = 1, 2, 3…I, I represents the total number of agents.

3. The method for determining the formation of drone swarms and capturing drones according to claim 1, characterized in that: The specific steps of establishing the consistency negotiation update equation in step 2 include: Establish a multi-agent formation parameter decision model; set constraints for the multi-agent formation parameter decision model; obtain local observation information of the multi-agent formation; The consistency negotiation update equation is established based on the multi-agent formation parameter decision model, constraints and local observation information of the multi-agent formation.

4. The method for determining the formation of drone swarms and capturing drones according to claim 1, characterized in that: The consistency negotiation update equation expression in step 2 is: in, is the updated formation parameter of the ith agent, π i Represents the decision model of the formation parameters of the ith agent. According to the local observation information of the ith agent, it outputs the update rate of the formation parameters, s a is the state of the attacker; o(·) is the local observation information function of the agent; c θ is the weight coefficient of consistency negotiation, is the estimated formation parameter of the ith agent, is the estimated formation parameter of the jth agent adjacent to the ith agent, is the neighbor set of the ith agent as the defender.

5. The method for determining the formation of drone swarms and encircling them according to claim 1, characterized in that: The multi-agent dynamics model described in step 4 includes a dynamics model of a fully constrained multi-agent and a dynamics model of a non-fully constrained multi-agent.

6. The method for determining the formation of drone swarms and encircling them according to claim 2, characterized in that: The expression of the control instruction of the multi-agent dynamics model in step 4 is: Among them, u i is the control instruction of the ith agent, k p is the weight coefficient corresponding to the position error, is the expected position of the ith agent as the defender, p i is the reference position of the ith agent, k v is the weight coefficient corresponding to the speed error, is the moving speed of the center of the formation, v i is the linear velocity of the ith agent, C d is the drag coefficient of the agent.

7. The method for determining the formation of drone swarms and encircling them according to claim 1, characterized in that: The specific steps of designing the reinforcement learning method and determining the learning samples described in step 5 include: A reinforcement learning framework and a learning environment model are established; the learning environment model is embedded in the reinforcement learning framework to obtain a reinforcement learning method.

8. The method for determining the formation of drone swarms and encircling them according to claim 1, characterized in that: The loss function of the formation parameter decision model includes a model learning loss function, an evaluation learning loss function and an action learning loss function.

9. The method for determining the formation of drone swarms and capturing drones according to claim 1, characterized in that: The specific steps of encircling the attacking cluster based on the final formation parameter decision model described in step 6 include: Step 61, when the distance between the attacking clusters is relatively far, multiple agents enter the round-up phase; in the round-up phase, each agent inputs the final formation parameter decision model respectively, and outputs the formation parameters of each agent; Step 62: Neighboring agents negotiate formation parameters based on the consensus negotiation update equation to obtain updated formation parameters of each agent. Step 63: input the updated formation parameters of each agent into the multi-agent dynamics model, output the control instructions of each agent; and obtain the corresponding agent state based on the control instructions of each agent; Step 64: the attacking cluster makes corresponding state transitions according to the states of each agent to obtain an updated state of the attacking cluster; Step 65, determining whether the updated state of the attacking cluster is enclosed in the environment model, if so, proceeding to the next step, if not, repeating steps 62-64 until the updated state of the attacking cluster is enclosed in the environment model; Step 66: After the updated state of the attacking cluster is enclosed in the environmental model, it enters the transportation phase; based on the speed obstacle method, the vector of the center speed of the multi-agent formation is optimized to maintain a closed encirclement formation while avoiding environmental obstacles.

10. The drone swarm formation decision and encirclement method according to claim 9, characterized in that: The transportation phase described in step 66 includes multiple transportation tasks; the multiple transportation tasks include avoiding environmental obstacles, preventing the attacking cluster from escaping, and transporting the attacking cluster to a designated safe area.

Images