A Multi-Pursuit and Evasion Game Method for UAVs Based on PSO-M3DDPG

Abstract

The present invention provides a multi - to - multi pursuit - evasion game method for unmanned aerial vehicles (UAVs) based on PSO - M3DDPG. Specifically, the present invention: constructs a UAV motion model and a task assignment model; constructs reward functions for pursuit UAVs and evasion UAVs, introduces the PSO algorithm to initialize the parameters of the neural network model, and uses the PSO - M3DDPG algorithm to train the initial pursuit - evasion game model of both the pursuit and evasion UAVs. Eventually, convergence is achieved to obtain the final model of the UAV multi - to - multi pursuit - evasion game. The final model of the UAV multi - to - multi pursuit - evasion game can be used to output the strategies of the pursuit UAVs in real time. The present invention introduces the PSO algorithm to avoid unreasonable initial values, combines the M3DDPG algorithm with the min - max method to improve the robustness and generalization ability of the algorithm. At the same time, a multi - agent adversarial learning method is introduced to solve the computational difficulties brought by solving the optimization min - max objective.

Description

Technical Field

[0001] The present invention relates to the fields of drone pursuit and escape game and reinforcement learning, and in particular to a drone many-to-many pursuit and escape game method. Background Art

[0002] With the upgrading of the technology industry and the improvement of intelligence, actual application scenarios have become more complex. Many important applications require interaction between intelligent agents. During the training process, each intelligent agent is changing and the environment becomes unstable. Therefore, traditional single-agent reinforcement learning algorithms are not applicable.

[0003] To address these issues and enable agents to learn stable strategies, researchers proposed the MADDPG algorithm, based on the DDPG algorithm. This algorithm incorporates information about other agents into the critic network and employs a "centralized training, distributed execution" approach. This algorithm is suitable for cooperative environments as well as mixed environments where competition and cooperation coexist. However, agents trained with the MADDPG algorithm are often fragile and sensitive to their environment, especially when their behavior could be detrimental to other agents, such as the risk of collision in a pursuit-and-escape environment. These agents can easily become trapped in poor local optima.

[0004] Therefore, how to provide a multi-to-multi pursuit and escape game method for drones based on game reinforcement learning that can reduce the sensitivity of the intelligent agent to the environment and provide pursuit and escape strategies in real time and intelligently is a problem that technical personnel in this field urgently need to solve. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this paper provides a multi-to-multi drone pursuit and escape game method based on PSO-M3DDPG. This method improves the robustness and generalization of the algorithm. Researchers proposed the PSO-M3DDPG algorithm by combining the MADDPG algorithm with the minimax method. They also introduced the multi-agent adversarial learning (MAAL) method to address the computational difficulties associated with solving the minimax optimization objective. This design optimizes the network training process, prevents agents from falling into local optima, and improves the pursuit effect.

[0006] In order to achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0007] S1: Construct a multi-to-multi virtual pursuit and escape game scenario for drones;

[0008] S2: Construct a UAV motion model; set the three maneuvering modes of the escaping UAV to be linear motion, simple curve motion, and intelligent escape motion, with increasing complexity;

[0009] S3: Build a task allocation model; design advantage function and objective function to solve the optimal task allocation;

[0010] S4: Build an information interaction model for drones;

[0011] S5: Construct reward functions for pursuit drones and escaping drones;

[0012] S6: Introduce the PSO algorithm to perform a certain degree of search and learning on the initial weight values of the deep neural network model, obtain a relatively optimal solution in the overall optimization process, and initialize the neural network model parameters;

[0013] S7: Use the PSO-M3DDPG algorithm to train the initial pursuit-evasion game model of the drones on both sides of the pursuit and escape. Eventually, it converges to obtain the final model of the multi-to-multi pursuit-evasion game of drones;

[0014] S8: The final model of the multi-to-multi pursuit-evasion game of drones can be used to output the strategies of pursuit drones in real time;

[0015] Input the state information of the pursuit and escape drones into the neural network of the final multi-to-multi evasion game model of drones in real time, perform task allocation, obtain the action selection of the pursuit and escape drones, and control the pursuit drones to complete the pursuit task of the escaping drones.

[0016] In step S2, for the multi-to-multi pursuit and escape task of drones, three escape strategies with gradually increasing levels of intelligence are designed, and the strategy training for the pursuit and escape tasks of our drones is carried out respectively. The expressions for the three maneuvering methods of the escaping drones are as follows:

[0017] Linear motion:

[0018]

[0019] Curvilinear motion:

[0020]

[0021] a v = 0 a θ =(θ' target -θ target ) / Δt

[0022] Intelligent escape motion:

[0023]

[0024] θ tar-uav = arctan((y target -y center ) / (x target -x center ))

[0025]

[0026] where Δt is the step size; x target , y target are the coordinates of the escaping drone, and x′ target , y′ target are the coordinates of the escaping drone at the next moment; v target , θ target are the speed magnitude and direction of the escaping drone. The direction is the angle between the speed direction and the positive X-axis. v′ target , θ′ target are the speed magnitude and direction of the escaping drone at the next moment; a v , a θ are the linear acceleration and angular acceleration of the escaping drone; k and m are parameters affecting the curvature of the curve, k = 1, 2, 3...; m can take any value; x center , y center are the geometric center coordinates of the pursuing drone swarm; θ tar-uav is the angle between the line connecting the escaping drone and the geometric center of the pursuing drone swarm within the detection range and the positive x-axis of the map. x i is the x-coordinate of the position of the i-th drone, and y i is the y-coordinate of the position of the i-th drone, i = 1, 2, 3,..., n, where n is the number of drones.

[0027] The task assignment model includes a dominance function and an objective function. The task assignment flag a ij is the drone task assignment flag, indicating whether the i-th drone executes the j-th task:

[0028]

[0029] When the number of pursuing drones is equal to the number of escaping drones, the following 0-1 programming is performed:

[0030]

[0031] When the number of pursuing drones is greater than the number of escaping drones, the following 0-1 programming is performed:

[0032]

[0033] The position of the pursuing drone is The position of the escaping drone is The speed ratio of the pursuing drone and the escaping drone The coordinates of the center of the Apollonius circle can be obtained as The radius of the Apollonius circle is The smaller the Apollonius circle formed by the pursuer and the escapee, the better, and the larger the abscissa of the point on the circle closest to the Y-axis, the better. The dominance function is defined as follows:

[0034]

[0035] The overall objective function is as follows:

[0036]

[0037] Therefore, the optimal task allocation is as follows:

[0038]

[0039] The UAV information interaction model is that each UAV can only interact with the three nearest UAVs within its communication range.

[0040] The reward function includes two types of guiding global rewards in terms of distance and direction, as well as two types of local rewards for capture success and mission failure. The reward function is set as follows:

[0041]

[0042]

[0043] where r golbal is the global reward; is the reward in terms of distance; is the reward in terms of direction; r local is the local reward; r final is the reward value for a UAV to complete the pursuit of a single escaping UAV; r bound represents the boundary reward and punishment; r team is the sub-pursuit team reward; r danger is the reward and punishment value for collisions between pursuit UAVs; dis represents the relative distance at the current moment; dis_ represents the relative distance at the next moment; represents the angle between the velocity vector of the pursuit UAV and the line connecting the positions of the pursuit and escaping UAVs; β and γ are hyperparameters representing weight coefficients; α danger represents the weight coefficient, i represents the i-th UAV, is the global reward of the i-th UAV, represents the local reward of the i-th UAV, is the reward in terms of distance, is the reward in terms of direction, is the reward value for the i-th UAV to complete the pursuit of a single escaping UAV, is the boundary reward and punishment of the i-th UAV, is the sub-pursuit team reward of the i-th UAV, is the reward and punishment value for collisions between the i-th UAV and other pursuit UAVs, dis irepresents the relative distance between the i-th UAV and the target UAV at the current moment, represents the relative distance between the i-th UAV and the target UAV at the next moment, d ij represents the relative distance between the i-th UAV and the j-th UAV, d danger represents the minimum collision distance between pursuit UAVs, d safe represents the minimum safe distance between pursuit UAVs.

[0044] The PSO algorithm is used to optimize and solve the initial weight values of the deep neural network model. The specific steps are as follows:

[0045] Set relevant parameters c1, c2, w and the number of particles, initialize the population, and continuously evaluate and optimize the particles in the population. That is, randomly initialize a group of particles in the feasible solution space. Each particle is a feasible solution to the optimization problem, and an objective function determines a fitness value for it. Each particle will move in the solution space, and its movement direction and distance at the next moment are determined by the velocity. The particle will follow the current optimal particle and finally obtain the optimal solution through generation-by-generation search; calculate the velocity and position of each particle at the new moment according to the following formula:

[0046] v id (t + 1) = wv id (t) + c1r1(p i (t) - x id (t)) + c2r2(g(t) - x id (t))

[0047] x id (t + 1) = x id (t) + v id (t + 1)

[0048] For the i-th particle, its position is represented as X i = (x i0 , x i1 , …, x iDim ), and its velocity is represented as V i = (v i0 , v i1 , …, v iDim ), i = 1, 2, …, N, where N is the number of particles and Dim is the dimension of the particle. x id (t) represents the position of particle i at the current moment t, and the subscript d represents the particle dimension. x id (t + 1) represents the position of particle i at the next moment t + 1. v id (t) represents the velocity of particle i at the current moment t, and the subscript d represents the particle dimension. v id(t + 1) represents the velocity of particle i at the next moment t + 1, and p i (t) represents the optimal solution of particle i before time t, and g(t) represents the historical optimal solution of the entire particle swarm before time t; w is the inertia weight, controlling the change of particle velocity; r1 and r2 are random numbers between [0, 1], serving as control weights; c1 and c2 are learning factors. When c1 = c2 = 2, it represents the random acceleration weights for the particle to advance towards its own and the global optimal values;

[0049] Assume that all other agents act in the most unfavorable way for themselves, forming a minimum learning objective J M (θ i ):

[0050]

[0051] s represents the agent's environmental state under the state space ρ, and a i represents the policy network of the i-th agent under which the selected action, and θ i is the parameter of the policy network, and O i is the observation information of the i-th agent, is the centralized state function of the i-th agent;

[0052] Update the critic network by minimizing the estimation error, update the actor network using the sampled policy gradient, optimize the parameters, and use the method of minimizing the loss value and gradient descent to optimize the weights and bias parameters of the critic network and the weights and bias parameters of the actor network respectively. Finally, construct the optimal policy network as follows:

[0053]

[0054]

[0055] y k is the objective function of the critic network, and L(θ i ) is the loss function;

[0056] The update method of the target network is as follows:

[0057] θ′ target ← τθ eval +(1 - τ)θ target

[0058] τ represents the soft update coefficient, indicating the proportion of new elements.

[0059] In the M3DDPG algorithm, a multi-agent adversarial learning method is adopted to solve the min-max objective. That is, when other agents all take actions that are most unfavorable to itself, the optimal strategy for the agent to execute the pursuit task is solved; by constructing a local linear function to approximate the non-linear state value function, the gradient descent method is used to replace the inner-loop minimization method to approximately solve the objective, that is, to solve the local linear function. The supplementary formula is as follows:

[0060]

[0061] where a′ k is the action taken by the k-th agent through the selection of the policy network at state o k , is the action that makes the Q value of the j-th agent decrease fastest. The hyperparameter γ is the discount factor, and ε j≠i is the introduced interference, which makes the actions of other agents perform worst. A group of perturbations ε is defined. By linearizing the Q function a perturbation value ε j is sought, and locally approximated in the gradient direction for the action a * that reduces the Q value the most. Let a * =a + ε. By superimposing the interference, the actions of other agents that reduce the Q value the most are made, and an adverse environment is constructed to learn a more robust strategy. Then the Q function is linearized, and the expression is as follows:

[0062]

[0063] where α represents an adjustable coefficient that can affect the step size of the gradient descent solution.

[0064] The beneficial effect of the present invention is to provide a multi-agent pursuit-evasion game method for unmanned aerial vehicles based on PSO-M3DDPG. The PSO-M3DDPG algorithm provided by the present invention is improved and developed on the basis of the MADDPG algorithm. The PSO algorithm is introduced to avoid the unreasonableness of the initial value. Combined with the M3DDPG algorithm of the min-max method, the robustness and generalization ability of the algorithm are improved. At the same time, the multi-agent adversarial learning (MAAL) method is introduced to solve the computational difficulties brought by solving the optimized min-max objective. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is the flowchart of the multi-agent pursuit-evasion game method for unmanned aerial vehicles based on PSO-M3DDPG provided by the present invention.

[0066] Figure 2 is the schematic diagram of the simulation scene of the multi-agent pursuit-evasion game for unmanned aerial vehicles provided by the present invention;

[0067] Figure 3 Schematic diagram of the motion model of the escape UAV provided by the present invention; Figure 3 In Figure (a) of [], the motion model diagram of the escape UAV moving in a uniform straight line, Figure 3 In Figure (b) of [], the motion model diagram of the escape UAV moving in a simple curve, Figure 3 In Figure (c) of [], the intelligent escape motion model diagram of the escape UAV.

[0068] Figure 4 Schematic diagram of the cooperative pursuit of the pursuit UAV provided by the present invention;

[0069] Figure 5 Schematic diagram of the information interaction model of the UAV provided by the present invention;

[0070] Figure 6 Flowchart of the PSO algorithm provided by the present invention;

[0071] Figure 7 Initial situation diagram of the multi - to - multi pursuit - evasion game task of the UAV provided by the present invention; Figure 7 In Figure (a) of [], the two - to - two pursuit - evasion task of the UAV, Figure 7 In Figure (b) of [], the four - to - two pursuit - evasion task of the UAV.

[0072] Figure 8 Comparison diagram of the change curves during the training process of some parameters of the "actor_eval network" when the escape UAV moves in a straight line provided by the present invention;

[0073] Figure 9 Trajectory diagram of the trained model controlling the pursuit UAV to pursue the escape UAV moving in a straight line provided by the present invention; Figure 9 In Figure (a) of [], the trajectory diagram of the two - to - two pursuit - evasion task of the UAV, Figure 9 (b) is the trajectory diagram of the four - to - two pursuit - evasion task of the UAV.

[0074] Figure 10 Comparison diagram of the individual and global reward curves when the escape UAV moves in a straight line during the training process in the multi - to - multi pursuit - evasion task provided by the present invention; Figure 10 In Figure (a) of [], the individual reward, Figure 10 To Figure (b) is the global reward.

[0075] Figure 11 Comparison diagram of the change curves during the training process of some parameters of the "actor_eval network" when the escape UAV moves in a simple curve provided by the present invention;

[0076] Figure 12 Trajectory diagram of the trained model controlling the pursuit UAV to pursue the escape UAV moving in a simple curve provided by the present invention; Figure 12Figure (a) of [ID] is the trajectory diagram of the two - on - two pursuit - evasion mission of drones. Figure 12 Figure (b) of [ID] is the trajectory diagram of the four - on - two pursuit - evasion mission of drones.

[0077] Figure 13 This is the comparison diagram of the individual and global reward curves when the escaping drones adopt simple curve motion during the training process in the multi - on - multi pursuit - evasion mission provided by the present invention; Figure 13 Figure (a) of [ID] is the individual reward. Figure 13 Figure (b) of [ID] is the global reward.

[0078] Figure 14 This is the comparison diagram of the change curves during the training process of some parameters of the "actor_eval network" when the escaping drones adopt intelligent escape motion provided by the present invention;

[0079] Figure 15 This is the trajectory diagram of the trained model controlling the pursuit drones to pursue the escaping drones with intelligent escape motion provided by the present invention; Figure 15 Figure (a) of [ID] is the trajectory diagram of the two - on - two pursuit - evasion mission of drones. Figure 15 Figure (b) of [ID] is the trajectory diagram of the four - on - two pursuit - evasion mission of drones.

[0080] Figure 16 This is the comparison diagram of the individual and global reward curves when the escaping drones adopt intelligent escape motion during the training process in the multi - on - multi pursuit - evasion mission provided by the present invention; Figure 16 Figure (a) of [ID] is the individual reward. Figure 16 Figure (b) of [ID] is the global reward. Detailed implementation manners

[0081] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0082] As Figure 1 , the embodiment of the present invention discloses a multi - on - multi pursuit - evasion game method for drones based on PSO - M3DDPG, including the following steps:

[0083] S1: Construct a multi - on - multi virtual pursuit - evasion game scenario for drones.

[0084] S2: Construct a drone motion model. Set three maneuvering modes for the escaping drones: linear motion, simple curve motion, and intelligent escape motion, with the complexity increasing gradually;

[0085] S3: Construct a task allocation model. Design the advantage function and the objective function to solve the optimal task allocation situation;

[0086] S4: Build a drone information interaction model;

[0087] S5: Construct the reward functions for the pursuit drones and the escaping drones;

[0088] S6: Introduce the PSO algorithm to conduct a certain degree of search and learning on the initial weight values of the deep neural network model to obtain a better solution in the overall optimization process and initialize the neural network model parameters;

[0089] S7: Using the PSO-M3DDPG algorithm, the initial pursuit-escape game model of the two drones is trained and eventually converges to obtain the final model of the many-to-many pursuit-escape game of drones;

[0090] S8: The final model of the multi-to-multi UAV pursuit-escape game can output the strategy for chasing UAVs in real time.

[0091] The above steps are described in detail below.

[0092] S1: Construct a multi-to-multi virtual pursuit and escape game scenario for drones. Figure 2 As shown, the battlefield situation is observed from the perspective of the pursuit drone, where the blue drone is the pursuit drone e1, the red drone is the escape drone p1, and the d E is the safe distance of the escaping drone. When the distance between the chasing drone and the escaping drone is less than d E At that time, it was believed that the escaping drone had been successfully pursued.

[0093] S2: Determine the environmental constraints of the pursuit drone and the escaping drone in the pursuit-escape game scenario, and set three maneuvering modes for the escaping drone: linear motion, simple curve motion, and intelligent escape motion, with increasing complexity. The motion modeling of the pursuit drone is as follows: Figure 3 As shown in the figure, three escape strategies with gradually increasing intelligence levels are designed, where Δt is the step length; x target 、y target is the coordinate of the escaping drone; v target ,θ target is the speed and direction of the escaping drone, and the direction is the angle between the speed direction and the positive direction of the X axis. The motion expression is as follows:

[0094] For the multi-to-multi drone pursuit mission, we designed three escape strategies with gradually increasing intelligence levels, and trained our own drone pursuit mission strategies respectively. The expressions are as follows:

[0095] Linear motion:

[0096]

[0097] Curved motion:

[0098]

[0099]

[0100] a v = 0 a θ =(θ' target - θ target ) / Δt

[0101] Intelligent escape movement:

[0102]

[0103] θ tar-uav = arctan((y target - y center ) / (x target - x center ))

[0104]

[0105] where Δt is the step size; x target , y target are the coordinates of the escaping UAV; v target , θ target are the speed magnitude and direction of the escaping UAV, and the direction is the angle between the speed direction and the positive direction of the X-axis; a v , a θ are the linear acceleration and angular acceleration of the escaping UAV; v' target , θ' target are the speed magnitude and direction of the escaping UAV at the next moment; x' target , y' target are the coordinates of the escaping UAV at the next moment; k, m are parameters affecting the bending degree of the curve; x center , y center are the geometric center coordinates of the pursuing UAV cluster; θ tar-uav is the angle between the escaping UAV and the line connecting the geometric center of the pursuing UAV cluster within the detection range.

[0106] S3: Build a task allocation model. Make the following assumptions:

[0107]

[0108] Task allocation mainly has two cases. One is that the number of pursuing UAVs is equal to the number of escaping UAVs, i.e., (n = m); the other is that the number of pursuing UAVs is more than the number of escaping UAVs, i.e., (n > m).

[0109] When n = m, perform 0-1 programming:

[0110]

[0111] When n > m, the UAVs cooperate in pursuit as Figure 4 . Perform 0-1 programming:

[0112]

[0113] Assume the position of the pursuing drone is The position of the escaping drone is The speed ratio of the two The coordinates of the center of the Apollonius circle can be obtained as The radius of the Apollonius circle is Define the advantage function as follows:

[0114]

[0115] The overall objective function is as follows:

[0116]

[0117] The optimal task assignment is as follows:

[0118]

[0119] S4: Build a UAV information interaction model. Each UAV has a fixed communication range, and UAVs can communicate with each other only within the communication range. Assume that each UAV can only interact with the three nearest UAVs within its communication range. The information interaction model is as Figure 5 shown.

[0120] S5: Construct the reward functions for the pursuing UAV and the escaping UAV respectively; for the completion of the pursuit task, two guiding global rewards in terms of distance and direction and two local rewards for capture success and mission failure are designed; for the cooperation requirements among the pursuit teams, two local rewards for forming a pursuit team and collision between UAVs are designed.

[0121] The reward functions are set as follows:

[0122]

[0123] where r golbal is the global reward; is the reward in terms of distance; is the reward in terms of direction; r local is the local reward; r final is the reward value for a UAV to complete the pursuit of a single escaping UAV; r bound represents the boundary reward and punishment; r team is the sub - pursuit team reward; r dangerThe reward and punishment value for collisions between pursuit drones; dis represents the relative distance at the current moment; dis_ represents the relative distance at the next moment; φ represents the angle between the velocity vector of the pursuit drone and the line connecting the positions of both the pursuit and evasion drones; β and γ are hyperparameters representing weight coefficients; α danger represents the weight coefficient.

[0124] S6: Introduce the PSO algorithm to conduct a certain degree of search and learning on the initial weight values of the deep neural network model to obtain a relatively optimal solution in the overall optimization process. Initialize the neural network model parameters according to the relatively optimal solution, and then perform detailed optimization learning through the gradient descent of the neural network to obtain the optimal solution. The PSO algorithm flowchart is as Figure 6 shown.

[0125] represents the position of the particle at time t, represents the position of the particle at the next moment, represents the velocity of the particle at time t, represents the velocity of the particle at the next moment, represents the optimal solution of the particle before time t, represents the historical optimal solution of the entire particle swarm before time t.

[0126] For the i-th particle, its position is represented as X i =(x i0 ,x i1 ,…,x iDim ) and its velocity is represented as V i =(v i0 ,v i1 ,…,v iDim ) where i = 1, 2, …, N, N is the number of particles, and Dim is the dimension of the particle. Each particle determines its own velocity based on its own experience and the experience of the group to adjust its own trajectory and move closer to the optimal point. Different particles obtain their individual fitness values through the corresponding objective function, and analyze and evaluate the quality of individuals based on the individual fitness values. The velocity and position update formulas of the particles are as follows:

[0127] v id (t + 1)=wv id (t)+c1r1(p i (t)-x id (t))+c2r2(g(t)-x id (t))

[0128] x id (t + 1)=x id (t)+v id (t + 1)

[0129] In the above formula: w is the inertia weight, which controls the change of particle velocity; r1 and r2 are random numbers between [0, 1], which are used to control the weight; c1 and c2 are learning factors, representing the random acceleration weights for the particle to advance towards its own and the global optimal values.

[0130] S7: Use the PSO-M3DDPG algorithm to train the initial pursuit-evasion game model of the UAVs on both sides of the pursuit-evasion. The M3DDPG algorithm adopts the weight structure obtained by the PSO algorithm for the gradient descent update mode of deep reinforcement learning, and continuously updates the "critic_eval" network and "actor_eval" network of the UAVs until final convergence to obtain the optimal solution. After the training is completed, the final model of the multi-to-multi pursuit-evasion game of the UAVs is obtained. The algorithm flow chart of the PSO-M3DDPG algorithm is as Figure 1 shown.

[0131] Assume that all other agents perform the most unfavorable behaviors to themselves, force each agent to execute a relatively good strategy, and optimize the cumulative reward of the agent, thus forming the minimum learning objective J M (θ i ).

[0132]

[0133] s represents the agent's environmental state under the state space ρ, a i represents the action selected by the i-th agent's policy network under, θ i is the parameter of the policy network, O i is the observation information of the i-th agent, the centralized state function of the i-th agent;

[0134] Update the critic network by minimizing the estimation error, and update the actor network using the sampled policy gradient to optimize the parameters. The loss function and optimization formula are as follows:

[0135]

[0136] The model adopts a deep network framework, including a "real" and an "estimated" dual-network model. Therefore, the target network update method is as follows:

[0137] θ target ←τθ eval +(1 - τ)θ target

[0138] During the training process, the basic M3DDPG algorithm and the improved PSO-M3DDPG algorithm are used as the decision-making units of the pursuit drones to train the model for the multi-drone multi-pursuit and evasion task. At the beginning of each round of training, the initial states of the drones are randomly initialized. The different initial scenarios of the multi-drone multi-pursuit and evasion task are as follows Figure 7 as shown

[0139] Next, the present invention conducts a simulation experiment on the final model of the multi-drone multi-pursuit and evasion game obtained by the above method:

[0140] In a 500*500 simulated environment scenario, multiple pursuit drones and multiple evasion drones are randomly generated. The speed magnitude and direction of the drones are randomly generated within their respective value ranges. It is set that the evasion drones perform straight-line motion, simple curve motion, and intelligent evasion motion, and the maximum number of iterations per round is set to 800. The network parameters are trained until convergence.

[0141] The convergence of the improved algorithm is studied. The changes in some parameter values of the actor_eval network are respectively as Figure 8 、 Figure 11 、 Figure 14 shown. It can be seen from the figure that, as can be seen from the above figure, at the beginning of the training of the M3DDPG algorithm, due to the random initialization of the neural network parameters according to the normal distribution, local optimal solutions are likely to appear in the pursuit and evasion decision-making process; while in the training of the PSO-M3DDPG algorithm, due to the use of the PSO algorithm for pre-training of the neural network parameters, a better solution space is obtained, and the overall parameter optimization amplitude is smaller during the neural network update, and the convergence speed is accelerated. As the learning process deepens, the neural network parameters gradually approach the optimal value until convergence, reaching a stable state, and a stable behavior decision-making model for the drone swarm is obtained.

[0142] The actual performance of the improved algorithm is studied. The PSO-M3DDPG algorithm and the M3DDPG algorithm are respectively used to train the multi-drone multi-pursuit and evasion task, and the average individual round reward of the drones and the overall reward value of the drone swarm are recorded for each training round. Specifically, they are respectively as Figure 10 、 Figure 13 、 Figure 16 shown. It can be seen from the figure that as the training process progresses, the reward value slowly rises. The initial reward value of the PSO-M3DDPG algorithm is higher than that of the M3DDPG algorithm, and the overall learning efficiency and the final convergence result are significantly better than those of the M3DDPG algorithm, indicating that the PSO algorithm is used to search and learn the neural network parameters to a certain extent, obtaining a better initial solution, promoting the learning process of the neural network, accelerating the algorithm convergence speed and obtaining a better convergence result.

[0143] S8: The final model of the multi - to - multi pursuit - evasion game using drones can output the strategies of the pursuit drones in real time.

[0144] After training, to test the effect of the model, the pursuit - evasion model was used to control the pursuit - evasion drones in real time for simulation experiments. After 1000 rounds of experiments, the results proved that the pursuit drones could successfully capture the evading drones. The simulation experiment trajectory diagrams of the evading drones under different maneuvering methods are respectively as Figure 9 、 Figure 12 、 Figure 15 shown. Figure 9 Figure (a) of Figure 9 is the trajectory diagram of the two - to - two drone pursuit - evasion mission. Figure 9 Figure (b) is the trajectory diagram of the four - to - two drone pursuit - evasion mission. Figure 12 The left and right figures of Figure 12 are the trajectory diagrams of the pursuit missions when the evading drone performs uniform linear motion at different speeds and directions. Figure 12 The left and right sides of Figure 15 Figure (a) of Figure 15 is the trajectory diagram of the two - to - two drone pursuit - evasion mission. Figure 15 Figure (b) is the trajectory diagram of the four - to - two drone pursuit - evasion mission.

[0145] The pursuit drone swarm was trained for pursuit - evasion missions with evading drones performing simple linear motion, simple curve motion, and intelligent evasion motion, and the performance of the trained neural network model was verified. The converged artificial neural network was used as the behavior decision unit of the pursuit drone swarm, and multi - to - multi drone pursuit - evasion experiments were carried out under different numbers and different initial states, and the trajectory diagrams of the pursuit drone swarm were obtained for analysis. From Figure 9 、 Figure 12 、 Figure 15 it can be seen that the trajectory diagrams of multi - to - multi pursuit - evasion missions with different numbers of drones are respectively shown. It can be seen from the figures that for targets with simple linear motion, simple curve motion, and intelligent evasion motion, swarms of drones with different numbers have effectively completed the pursuit - evasion missions and performed well.

Claims

1. A multi - to - multi pursuit - evasion game method for unmanned aerial vehicles based on PSO - M3DDPG, characterized in that It includes the following steps: S1: Construct a multi - to - multi virtual pursuit - evasion game scenario for unmanned aerial vehicles (UAVs); S2: Construct a UAV motion model; Set three maneuvering modes for the evading UAVs, namely linear motion, simple curve motion, and intelligent evasion motion, with increasing complexity in sequence; S3: Construct a task assignment model; Design an advantage function and an objective function to solve the optimal task assignment situation; S4: Build a UAV information interaction model; S5: Construct reward functions for the pursuing UAVs and the evading UAVs; S6: Introduce the PSO algorithm to conduct a certain degree of search and learning on the initial weight values of the deep neural network model, obtain a relatively optimal solution in the overall optimization process, and initialize the neural network model parameters; S7: Use the PSO - M3DDPG algorithm to train the initial pursuit - evasion game model of both the pursuing and evading UAVs, and finally converge to obtain the final model of the multi - to - multi UAV pursuit - evasion game; S8: The final model of the multi - to - multi UAV pursuit - evasion game can be used to output the strategies of the pursuing UAVs in real - time; Input the state information of the pursuing and evading UAVs into the neural network of the final multi - to - multi UAV evasion game model in real - time for task assignment, obtain the action selection of the pursuing and evading UAVs, and control the pursuing UAVs to complete the pursuit task of the evading UAVs.

2. The multi - to - multi UAV pursuit - evasion game method based on PSO - M3DDPG according to claim 1, wherein: In step S2, for the multi - to - multi UAV pursuit - evasion task, three escape strategies with gradually increasing intelligence levels are designed, and the strategy training of the pursuit - evasion tasks of our UAVs is carried out respectively. The expressions of the three maneuvering modes of the evading UAVs are as follows: Linear motion: Curve motion: a v = 0a θ =(θ′ target - θ target ) / Δt Intelligent evasion motion: θ tar-uav = arctan((y target - y center ) / (x target - x center )) where Δt is the step size; x target , y target are the coordinates of the escaping drone, and x′ target , y′ target are the coordinates of the escaping drone at the next moment; v target , θ target are the magnitude and direction of the velocity of the escaping drone. The direction is the angle between the velocity direction and the positive direction of the X-axis. v′ target , θ′ target are the magnitude and direction of the velocity of the escaping drone at the next moment; a v , a θ are the linear acceleration and angular acceleration of the escaping drone; k and m are parameters affecting the degree of curvature of the curve, k = 1, 2, 3...; m can take any value; x center , y center are the geometric center coordinates of the pursuing drone swarm; θ tar-uav is the angle between the line connecting the escaping drone and the geometric center of the pursuing drone swarm within the detection range and the positive direction of the x-axis of the map, x i is the x-coordinate of the position of the i-th drone, and y i is the y-coordinate of the position of the i-th drone, i = 1, 2, 3,..., n, where n is the number of drones.

3. The multi - to - multi UAV pursuit - evasion game method based on PSO - M3DDPG according to claim 1, wherein: The task assignment model includes an advantage function and an objective function, and a task assignment flag a ij is a drone task assignment flag, indicating whether the i-th drone executes the j-th task: When the number of pursuing UAVs is equal to the number of evading UAVs, the following 0 - 1 programming is carried out: When the number of pursuing UAVs is greater than the number of evading UAVs, the following 0 - 1 programming is carried out: The position of the pursuing drone is The position of the escaping drone is The speed ratio of the pursuing drone and the escaping drone The coordinates of the center of the Apollonius circle can be obtained as The radius of the Apollonius circle is The smaller the Apollonius circle formed by the pursuer and the escapee, the better, and the larger the abscissa of the point on the circle closest to the Y-axis, the better. Define the advantage function as follows: The overall objective function is as follows: Therefore, the optimal task assignment is as follows:

4. The multi - to - multi UAV pursuit - evasion game method based on PSO - M3DDPG according to claim 1, wherein: The UAV information interaction model is that each UAV can only interact with the three nearest UAVs within its communication range.

5. The multi - to - multi UAV pursuit - evasion game method based on PSO - M3DDPG according to claim 1, wherein: The reward function includes two types of guiding global rewards in terms of distance and direction, as well as two types of local rewards for capture success and task failure. The reward function is set as follows: where r golbal is the global reward; is the reward for distance; is the reward for direction; r local is the local reward; r final is the reward value for the UAV to complete the pursuit of a single escaping UAV; r bound represents the boundary reward and punishment; r team is the sub - pursuit team reward; r danger is the reward and punishment value for collisions between pursuit UAVs; dis represents the relative distance at the current moment; dis_ represents the relative distance at the next moment; represents the angle between the velocity vector of the pursuit UAV and the line connecting the positions of the pursuit and escaping UAVs; β and γ are hyperparameters representing weight coefficients; α danger represents the weight coefficient, i represents the i - th UAV, is the global reward of the i - th UAV, represents the local reward of the i - th UAV, is the reward for distance, is the reward for direction, is the reward value for the i - th UAV to complete the pursuit of a single escaping UAV, is the boundary reward and punishment of the i - th UAV, is the sub - pursuit team reward of the i - th UAV, is the reward and punishment value for collisions between the i - th UAV and other pursuit UAVs, dis i represents the relative distance between the i - th UAV and the target UAV at the current moment, represents the relative distance between the i - th UAV and the target UAV at the next moment, d ij represents the relative distance between the i - th UAV and the j - th UAV, d danger represents the minimum collision distance between pursuit UAVs, d safe represents the minimum safe distance between pursuit UAVs.

6. The multi - to - multi UAV pursuit - evasion game method based on PSO - M3DDPG according to claim 1, wherein: The PSO algorithm is used to optimize and solve the initial weight values of the deep neural network model. The specific steps are as follows: Set relevant parameters c1, c2, w and the number of particles, initialize the population, and continuously evaluate and optimize the particles in the population. That is, randomly initialize a group of particles in the feasible solution space. Each particle is a feasible solution to the optimization problem, and a fitness value is determined for it by the objective function. Each particle will move in the solution space, and its movement direction and distance at the next moment are determined by the velocity. The particle will follow the current optimal particle and finally obtain the optimal solution through generation-by-generation search. Calculate the velocity and position of each particle at the new moment according to the following formula: v id (t + 1) = wv id (t) + c1r1(p i (t) - x id (t)) + c2r2(g(t) - x id (t)) x id (t + 1) = x id (t) + v id (t + 1) For the i-th particle, its position is represented as X i =(x i0 , x i1 , …, x iDim ), and its velocity is represented as V i =(v i0 , v i , …, v iDim ), where i = 1, 2, …, N, N is the number of particles, Dim is the dimension of the particles, x id (t) represents the position of particle i at the current time t, the subscript d represents the particle dimension, x id (t + 1) represents the position of particle i at the next time t + 1, v id (t) represents the velocity of particle i at the current time t, the subscript d represents the particle dimension, v id (t + 1) represents the velocity of particle i at the next time t + 1, p i (t) represents the optimal solution of particle i before time t, g(t) represents the historical optimal solution of the entire particle swarm before time t; w is the inertia weight, controlling the change of particle velocity; r1, r2 are random numbers between [0, 1], serving as control weights; c1, c2 are learning factors, with c1 = c2 = 2, representing the random acceleration weights for the particle to advance towards its own and the global optimal values; Assume that all other agents act in the most unfavorable way for themselves, forming a minimum learning objective J M (θ i ): s represents the agent's environmental state under the state space ρ, a i represents the i-th agent's policy network the action selected under, θ i is the policy network parameter, O i the observation information of the i-th agent, the centralized state function of the i-th agent; Update the critic network by minimizing the estimation error, update the actor network using the sampled policy gradient, optimize the parameters, and use the method of minimizing the loss value and gradient descent to optimize the weights and bias parameters of the critic network and the weights and bias parameters of the actor network respectively. Finally, construct the optimal policy network as follows: y k is the objective function of the critic network, and L(θ i ) is the loss function; The target network is updated as follows: θ′ target ←τθ eval +(1 - τ)θ target τ represents the soft update coefficient, indicating the proportion of new elements.

7. The method for multi - to - multi pursuit - evasion game of unmanned aerial vehicles based on PSO - M3DDPG according to claim 1, wherein: In the M3DDPG algorithm, a multi-agent adversarial learning method is adopted to solve the minimax objective, that is, to find the optimal strategy for the agent to perform the pursuit task when all other agents take the most unfavorable actions against it; by constructing a local linear function to approximate the non-linear state value function, and using the gradient descent method to replace the inner-loop minimization method to approximately solve the objective, that is, to solve the local linear function. The supplementary formula is as follows: where a′ k is the action taken by the k-th agent selected through the policy network at state o k , and is the action that makes the Q-value of the j-th agent decrease fastest. The hyperparameter γ is the discount factor, and ε j≠i is the introduced interference that makes the actions of other agents perform worst. Define a set of perturbations ∈, and seek an interference value ε by linearizing the Q-function j , and locally approximate in the gradient direction for the behavior a * that reduces the Q-value the most * to perform interference, let a = a + ε. By superimposing the interference, make the actions of other agents reduce the Q-value the most, construct an adverse environment to learn a more robust policy, and then linearize the Q-function Where α represents an adjustable coefficient that can affect the step size of gradient descent solution.

Images