A multi-to-one UAV hunting game method based on ME-DDPG

Abstract

This paper provides a multi-to-one drone pursuit game method based on ME-DDPG. This method incorporates a directional strategy calculated using gradient descent into the set of strategies learned by the DDPG algorithm. This method uses neural network training to derive a pursuit strategy for drones. This method improves the local convergence problem caused by insufficient exploration and enhances the learning efficiency of the algorithm. Simulation experiments have also verified the success rate of multi-to-one drone pursuit tasks. By incorporating gradient descent as a training action for drone pursuit, this method improves the convergence rate of the model. The trained pursuit game model exhibits good transferability and can be directly applied to multi-to-one drone pursuit tasks in various scenarios and situations, achieving a high success rate in drone pursuit.

Description

Technical Field

[0001] The present invention relates to the fields of drone pursuit game, differential strategy and reinforcement learning technology, and in particular to a many-to-one drone pursuit game method. Background Art

[0002] The multi-drone pursuit problem is a typical differential game problem in air combat. Due to the conflicting objective functions of both parties, it can be viewed as a multi-agent dynamic game system. The optimal control strategy for both aircraft depends on the interests of the pursuing and escaping parties. Traditional differential game methods face modeling difficulties and weak transferability of agent interactions in environments with a large number of agents, a large state space, and complex real-time decision-making. In contrast, reinforcement learning (RL) has attracted considerable attention in the field of complex systems and multi-agent systems due to its excellent learning performance. Agents modeled using RL can learn from their environment and other agents, thereby improving their behavior. However, in the multi-agent context, RL lacks targeted strategies and consideration of the impact of the adversarial game between the two parties, resulting in poor convergence and rationality. Therefore, this paper effectively combines game theory with RL algorithms and applies them to the drone pursuit problem. By replacing the optimal solution with the equilibrium solution in game theory to obtain an effective strategy, the convergence efficiency is improved while also making the strategy reasonable; at the same time, the characteristic of the reinforcement learning method that the intelligent agent improves the strategy by interacting with the environment and does not require the construction of an environmental model also solves the problem of modeling difficulties in game algorithms.

[0003] However, in the multi-agent domain, reinforcement learning lacks targeted strategies and consideration of the impact of the adversarial nature of the game, resulting in poor convergence and rationality. Therefore, a new approach has been proposed to effectively combine game theory with reinforcement learning algorithms for the drone pursuit problem. By replacing the optimal solution with the equilibrium solution in game theory to obtain an effective strategy, this approach improves convergence efficiency while also making the strategy more rational. Furthermore, the reinforcement learning method's characteristic of agents improving their strategies through interaction with the environment, without requiring a model of the environment, addresses the modeling challenges inherent in game algorithms.

[0004] Therefore, how to provide a multi-to-one UAV pursuit game method based on game reinforcement learning that takes into account kinematic constraints and can obtain a more real-time and intelligent pursuit strategy is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, the present invention provides a multi-to-one drone pursuit game method based on ME-DDPG. The present invention provides a Mixed Experienced Deep Deterministic Policy Gradient (ME-DDPG) algorithm. This algorithm adds a directional strategy calculated by the gradient descent method to the strategy set explored and learned by the DDPG algorithm. Through neural network training, the pursuit strategy for our drone is obtained, thereby improving the local convergence problem caused by insufficient exploration of our drone and improving the learning efficiency of the algorithm. The success rate of the multi-to-one drone pursuit mission has also been verified in simulation experiments.

[0006] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0007] S1: Construct a multi-to-one virtual pursuit game scenario for drones;

[0008] S2: Determine the nonholonomic motion constraints and environmental range constraints of the pursuing and escaping drones in the pursuit game scenario;

[0009] S3: Construct reward functions for both the pursuing and escaping drones. The reward functions evaluate the execution of a certain action by both drones in a specific state and output a reward value.

[0010] S4: Introducing the ME-DDPG algorithm, and combining the nonholonomic motion constraint, the environmental range constraint, and the reward function, constructing the initial pursuit game models of the pursuing and fleeing drones respectively;

[0011] S5: Train the initial pursuit game model for both the pursuing and escaping drones, obtain interaction data during the training process, and use this interaction data to continuously update the drones' neural network. After training is complete, the final model for the many-to-one pursuit game is obtained. Input the state information of the pursuing and escaping drones into the model's neural network in real time to determine the action choices of the pursuing drone. This allows multiple pursuing drones to complete the pursuit of the escaping drone with a high success rate.

[0012] S6: The final model of the many-to-one drone hunting game can output the strategy for hunting drones in real time.

[0013] The nonholonomic motion constraints for both the pursuing drone and the escaping drone in the pursuit game scenario are the same, and are expressed as follows:

[0014]

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] Among them, Q t is the matrix of the drone's information at time t; x t and y t is the position coordinate of the UAV at time t; v t is the speed of the UAV at time t; α t is the azimuth of the UAV at time t; and is the linear acceleration and angular acceleration of the UAV at time t; △t is the simulation step length; Q t+1 is the matrix of the drone's information at time t+1; x t+1 and y t+1 is the position coordinate of the UAV at time t+1; v t+1 is the speed of the UAV at time t+1; α t+1 is the azimuth of the UAV at time t+1; x p ,y p To hunt down the drone's location; x e ,y e is the location of the escaping drone; v p ,α p To chase the drone speed; v e ,α e Speed ​​for escaping drones.

[0022] The expression of the environmental range constraint condition is:

[0023]

[0024] In the above formula: [x min ,x max ] and [y min ,y max ] is the boundary of the virtual environment, x min ,x max is the minimum and maximum value of the horizontal coordinate of the virtual environment; y min ,y max is the minimum and maximum value of the vertical coordinate of the virtual environment; x min ,x max ,ymin ,y max A rectangular virtual environment is formed, and both the pursuit drone and the escape drone are generated and move in the virtual environment; p ,y p To hunt down the drone's location; x e ,y e is the position of the escaping drone. When the pursuit drone or the escaping drone flies out of any environment boundary, the pursuit mission is considered to have failed. When the distance between the pursuit drone and the escaping drone is less than the safe distance d of the escaping drone, the pursuit mission is considered to have failed. E When the pursuit was successful.

[0025] The reward function for chasing the drone includes the ending reward, immediate reward and collision reward, and the expression is:

[0026]

[0027] Among them, r i is the reward function for chasing drone i; For the ending reward of hunting down drone i; is the immediate reward function for chasing drone i; is the collision reward function of drone i;

[0028] End Reward The expression is:

[0029]

[0030] Among them, d fight The safe distance at which the escaping drone will not be captured; when the distance between the pursuing drone and the t Less than the safety distance d fight When the pursuit drone successfully captures the escaping drone, a reward of 60 is given to the pursuit drone, and this round ends. When the pursuit drone flies out of the scene boundary, the pursuit mission is considered to have failed, and this round ends based on the reward of -30 for the pursuit drone. When the escaping drone flies out of the scene boundary, the pursuit mission is considered to have failed, a reward of -30 is given to the pursuit drone, and this round ends.

[0031] Instant Rewards The expression is:

[0032]

[0033] Among them, γ and β are reward weight coefficients, both ranging from 0 to 1; dis is the distance between the pursuit drone i and the escaping drone at the current moment; dis_ is the distance between the pursuit drone i and the escaping drone at the next moment; φ is the angle between the velocity direction of the pursuit drone i and the position line between the pursuit drone i and the escaping drone.

[0034] Collision reward function The expression is:

[0035]

[0036] Among them, d ij is the distance between the chasing drone i and the chasing drone j; d danger To hunt down dangerous distances between drone individuals; d safe The safe distance between individual drones for pursuit; β danger is the reward weight coefficient, ranging from 0 to 1. When the distance between the two pursuing drones is less than the danger distance, a collision is considered to have occurred, and the reward value is -20 based on the drone. When the distance between the two pursuing drones is between the danger distance and the safe distance, the drone is given an appropriate negative reward value according to the formula. When the distance between the two pursuing drones is greater than the safe distance, the collision reward value is 0.

[0037] In step S5, the process of training the initial pursuit game model of the pursuing and escaping drones includes:

[0038] At the beginning of each game, the positions, speeds, and directions of multiple chasing drones and one escaping drone are randomly initialized.

[0039] The action of the pursuit drone is selected as the output of the actor neural network or calculated using the gradient descent method, and noise is superimposed. When the pursuit drone does not appear within the detection range of the escaping drone, the escaping drone performs random movements. When the pursuit drone appears within the detection range of the escaping drone, the escaping drone will fly towards the area with weak pursuit force and perform maneuvering flight.

[0040] The information of each step of the drone hunting (s, a, r, s') is saved for network training, where s is the state information of the drone at this moment, a is the action taken by the drone at this moment, r is the reward value of the drone at this moment, and s' is the state information of the drone at the next moment after taking the action;

[0041] After N training rounds episode After rounds of game training, the final pursuit game model of the pursuit and escape agents is obtained, N episode The round value is greater than 1000;

[0042] The action of chasing the drone is selected as the output of the actor neural network or calculated using the gradient descent method, and then noise is superimposed. The noise superposition method is OU noise, and the expression is:

[0043] dz t =θ(μ-z t)dt+ρdW t

[0044] Among them, z t is the state variable, t represents time; μ is the mean; θ is the rate parameter of mean reversion; ρ is the volatility parameter; W t The generated OU noise will gradually decrease over time, which can meet the demand for exploring new strategies in the process of interaction between the UAV and the environment.

[0045] The update process of the ME-DDPG algorithm is:

[0046] The ME-DDPG algorithm contains four neural networks, including two types of neural networks, namely actor network and critic network. The actor network outputs the action selection of the drone, and the critic network evaluates the output of the actor network; the actor network and the critic network each contain two types of networks, namely eval network and target network. The eval network is a real-time updated network that outputs real-time values. The update of the target network lags behind the eval network and outputs an evaluation value. The four neural networks are actor_eval strategy reality network, actor_target strategy estimation network, critic_eval value reality network, and critic_target strategy estimation network. _target value estimation network, the "actor_eval" policy estimation network and the "actor_target" policy reality network used to select actions; the "critic_eval" value estimation network and the "critic_target" value reality network used to evaluate action selection; during the training process, the parameters of the actor_eval policy reality network and the critic_eval value reality network are updated in real time with each step of training, and the parameters of the actor_target policy estimation network and the critic_target value estimation network use a soft update strategy, which follows the parameters of the eval neural network every 100 rounds and assists the training process of the eval neural network.

[0047] The state space and action space of the drone correspond to the input and output of the neural network respectively. The state space of the drone, with dimension s_dim of 20, includes the state information of the pursuing drone i itself, the state information of the escaping drone, and the state information of the three drones adjacent to the pursuing drone i. The expression is:

[0048] S=[x i ,y i ,v i ,α i ,x e ,y e ,ve ,α e ,x j ,y j ,v j ,α j ,x k ,y k ,v k ,α k ,x l ,y l ,v l ,α l ]

[0049] Among them, S is the state space of the UAV; x i ,y i ,v i ,α i The position, speed and direction of the hunting drone i; x e ,y e ,v e ,α e is the position, speed and direction of the escaping drone; x j ,y j ,v j ,α j ,x k ,y k ,v k ,α k ,x l ,y l ,v l ,α l The position, speed and direction of the three adjacent drones to hunt drone i.

[0050] The action space of the drone has a dimension a_dim of 2, including the linear acceleration and angular acceleration of the pursuit drone, and the expression is:

[0051] A=[δ v ,δ a ]

[0052] Among them, A is the UAV action space; δ v is the linear acceleration of the pursuit drone; δ a Angular acceleration for hunting drones.

[0053] In the neural network of the ME-DDPG algorithm, the actor network inputs the state information of the pursuit drone, and the input dimension is the dimension s_dim of the state space, that is, 20; the output is the decision action of the pursuit drone, and the output dimension is the dimension a_dim of the action space, that is, 2. The critic network inputs the current state information and behavior information of the pursuit drone, and the dimension is s_dim+a_dim, that is, 22; the output is the value judgment of the action taken by the actor network at the current moment, and the dimension is 1. The hidden layers of the four neural networks are all 4-layer fully connected neural networks.

[0054] The present invention provides a multi-to-one drone pursuit game method based on ME-DDPG, taking into account the nonholonomic constraints, velocity constraints, and acceleration constraints in drone kinematics. The ME-DDPG algorithm provided by the present invention improves upon the DDPG algorithm by incorporating gradient descent as a training action option for drone pursuit, thereby increasing the model's convergence speed. The trained pursuit game model exhibits excellent transferability and can be directly applied to multi-to-one drone pursuit missions in various scenarios and situations, achieving a high drone pursuit success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flowchart of the ME-DDPG-based drone many-to-one pursuit game method provided by the present invention;

[0056] Figure 2 A schematic diagram of a multi-to-one drone pursuit game simulation scenario provided by the present invention;

[0057] Figure 3 A schematic diagram of the escape strategy of the escaping drone provided by the present invention;

[0058] Figure 4 The initial situation diagram of the many-to-one pursuit game simulation of drones provided by the present invention;

[0059] Figure 5 Flowchart of the ME-DDPG algorithm provided by the present invention;

[0060] Figure 6 A graph showing the changes in some parameters of the actor_eval network during the training process provided by the present invention;

[0061] Figure 7 This is a graph showing the cumulative reward for hunting down drones under different p values ​​provided by the present invention;

[0062] Figure 8 The simulated trajectory diagram of the pursuit drone controlled by the trained model in four different situations provided by the present invention to pursue the escaping drone;

[0063] Figure 9 This is a comparison chart of the mission success rates of the ME-DDPG algorithm provided by the present invention in different training rounds for the many-to-one drone pursuit game. DETAILED DESCRIPTION

[0064] The present invention will be further described below with reference to the accompanying drawings and examples.

[0065] like Figure 1 As shown, the embodiment of the present invention discloses a multi-to-one UAV hunting game method based on ME-DDPG, comprising the following steps:

[0066] S1: Construct a multi-to-one virtual pursuit game scenario for drones;

[0067] S2: Determine the nonholonomic motion constraints and environmental range constraints of the pursuing drone and the escaping drone in the pursuit game scenario;

[0068] S3: Construct a reward function for hunting drones. The reward function is used to evaluate the execution of a certain action by the hunting drone in a specific state and output a reward value.

[0069] S4: Introducing the ME-DDPG algorithm and combining the nonholonomic motion constraint, the environmental range constraint, and the reward function to construct an initial pursuit game model for the drone;

[0070] S5: Train the initial pursuit game model for the drone, obtain interaction data during the training process, and use the interaction data to continuously update the drone's neural network. After the training is completed, the final model of the drone's many-to-one pursuit game is obtained;

[0071] S6: The final model of the many-to-one drone hunting game can output the strategy for hunting drones in real time.

[0072] The above steps are described in detail below.

[0073] S1: Construct a multi-to-one virtual pursuit game scenario for drones. Figure 2 As shown, the blue drones are pursuit drones p1, p2, ...p i ,...p n , the red drone is the escape drone e,d fight A safe distance to escape the drone.

[0074] When the distance d between all the chasing drones and the escaping drone is i When both are smaller than the safe distance of the escaping drone, the chasing drone is considered to have successfully completed the pursuit mission.

[0075] S2: Non-holonomic motion constraints and environmental range constraints are applied to the pursuit and escaping drones. The motion of the drone is described by the position coordinates x, y, velocity magnitude v, and velocity direction α. ​​The motion expression is as follows:

[0076]

[0077]

[0078]

[0079]

[0080] In the above formula: Q t is the matrix of the drone's information at time t; x t and y t is the position coordinate of the UAV at time t; v t is the speed of the UAV at time t; α t is the azimuth of the UAV at time t; and is the linear acceleration and angular acceleration of the UAV at time t; △t is the simulation step length; Q t+1 is the matrix of the drone's information at time t+1; x t+1 and y t+1 is the position coordinate of the UAV at time t+1; v t+1 is the speed of the UAV at time t+1; α t+1 is the azimuth of the UAV at time t+1; x p ,y p To hunt down the drone's location; x e ,y e is the location of the escaping drone; v p ,α p To chase the drone speed; v e ,α e Speed ​​for escaping drones.

[0081] The situation of chasing drones Figure 4 As shown in Figure 2, based on the UAV motion constraints and environmental range constraints, the acceleration, velocity, and position of the pursuit UAV and the escaping UAV are restricted as follows:

[0082]

[0083]

[0084]

[0085]

[0086] In the above formula, the speed of the pursuit drone is between [1, 2.5], the speed of the escaping drone is between [1.2, 2.2], the acceleration range of the pursuit drone line is [-1, 1], and the angular acceleration range is [x min ,x max ] and [y min ,y max ] is the boundary of the virtual environment. When the pursuit drone or the escaping drone flies out of the environment boundary, the pursuit mission is considered to have failed. When the distance between the pursuit drone and the escaping drone is less than a certain range, the pursuit is considered to have been successful.

[0087] During the pursuit game, the states of the pursuing and escaping drones can be determined by their position, velocity, and direction. The drone state space, with dimension s_dim of 20, includes the state information of the pursuing drone i itself, the state information of the escaping drone, and the state information of the three drones adjacent to the pursuing drone i. The expression is:

[0088] S=[x i ,y i ,v i ,α i ,x e ,y e ,v e ,α e ,x j ,y j ,v j ,α j ,x k ,y k ,v k ,α k ,x l ,y l ,v l ,α l ]

[0089] Among them, S is the state space of the UAV; x i ,y i ,v i ,α i The position, speed and direction of the hunting drone i; x e ,y e ,v e ,α e is the position, speed and direction of the escaping drone; x j ,y j ,v j ,α j ,x k ,y k ,v k ,α k,x l ,y l ,v l ,α l The position, speed and direction of the three adjacent drones to hunt drone i.

[0090] The motion control of the pursuit drone is controlled by linear acceleration and angular acceleration, which respectively change the speed and direction of the drone. The linear acceleration and angular acceleration are output as an action pair. The action space of the drone has a dimension a_dim of 2, including the linear acceleration and angular acceleration of the pursuit drone. The expression is:

[0091] A=[δ v ,δ a ]

[0092] Among them, A is the UAV action space; δ v is the linear acceleration of the pursuit drone; δ a Angular acceleration for hunting drones.

[0093] S3: Construct a reward function for chasing drones. The reward function for chasing drones includes the ending reward, immediate reward, and collision reward, and the expression is:

[0094]

[0095] Among them, r i is the reward function for chasing drone i; For the ending reward of hunting down drone i; is the immediate reward function for chasing drone i; is the collision reward function of drone i.

[0096] End Reward e i nd The expression is:

[0097]

[0098] Among them, d fight The safe distance at which the escaping drone will not be captured; when the distance between the pursuing drone and the t Less than the safety distance d fight When the pursuit drone successfully captures the escaping drone, it is considered that the pursuit drone has successfully captured the escaping drone, and a reward of 60 is given to the pursuit drone, and this round ends. When the pursuit drone flies out of the scene boundary, the pursuit mission is considered to have failed, and a reward of -30 is given to the pursuit drone, and this round ends. When the escaping drone flies out of the scene boundary, the pursuit mission is considered to have failed, and a reward of -30 is given to the pursuit drone, and this round ends.

[0099] Instant Rewards The expression is:

[0100]

[0101] Among them, α and β are reward weight coefficients; dis is the distance between the pursuit drone i and the escaping drone at the current moment; dis_ is the distance between the pursuit drone i and the escaping drone at the next moment; φ is the angle between the velocity direction of the pursuit drone i and the position line between the pursuit drone i and the escaping drone.

[0102] Collision reward function The expression is:

[0103]

[0104] Among them, d ij is the distance between the chasing drone i and the chasing drone j; d danger To hunt down dangerous distances between drone individuals; d safe The safe distance between individual drones for pursuit; β danger is the reward weight coefficient. When the distance between the two pursuing drones is less than the danger distance, a collision is considered to have occurred, and the drone is given a -20 reward. When the distance between the two pursuing drones is between the danger distance and the safe distance, the drone is given an appropriate negative reward according to the formula. When the distance between the two pursuing drones is greater than the safe distance, the collision reward is 0.

[0105] S4: Introduce the ME-DDPG algorithm to construct the initial pursuit game model of the drones of both pursuit and escape parties.

[0106] like Figure 3 At the beginning of each game, the positions, speeds, and directions of the chasing and escaping drones are randomly initialized;

[0107] In the DDPG algorithm, the output of the agent during training is the output of the actor neural network or a random selection method is used to select an action. The ME-DDPG algorithm provided by the present invention, based on the DDPG algorithm, integrates the idea of ​​differential games. Calculations have shown that the pursuit direction of the drone follows the direction of motion of the escaping drone, thereby replacing the random selection method of action in DDPG training. That is, the action of chasing the drone is selected as the output of the actor neural network or calculated using the gradient descent method. There is a 1-p probability of selecting the output of the actor neural network as the action of chasing the drone, and there is a p probability of selecting the gradient descent method as the action of chasing the drone. On this basis, the drone's action output is superimposed with noise. The method of superimposing noise on the output of the drone's action is OU noise, which is expressed as:

[0108] dzt =θ(μ-z t )dt+ρdW t

[0109] Where: μ is the mean; θ is the rate parameter of mean reversion; ρ is the volatility parameter; W t The generated OU noise will gradually decrease over time, which can meet the demand for exploring new strategies in the process of interaction between the UAV and the environment.

[0110] The action options for escaping the drone are as follows: Figure 4 When there are no pursuing drones within the detection range of the escaping drone, the escaping drone performs random motion. When a pursuing drone appears within the detection range, the escaping drone flies in the set escape direction toward the area with weak pursuit power, that is, the direction of the bisector of the maximum angle θ.

[0111] The information of each step of chasing the drone (s, a, r, s') is saved for network training, where s is the state information of the drone at this moment, a is the action taken by the drone at this moment, r is the reward value of the drone at this moment, and s' is the state information of the drone at the next moment after taking the action.

[0112] S5: After initialization, the initial pursuit game model of the UAVs of both parties is trained. The interaction data obtained during training is used to continuously update the neural network of the UAV. After the training is completed, the final model of the many-to-one UAV pursuit game is obtained.

[0113] The ME-DDPG algorithm provided by the present invention is an improved DDPG algorithm. The algorithm flow chart is as follows: Figure 5 shown.

[0114] The ME-DDPG algorithm includes four neural networks: the "actor_eval" policy estimation network and the "actor_target" policy realization network, used to select actions; and the "critic_eval" value estimation network and the "critic_target" value realization network, used to evaluate action selections. During training, the parameters of the eval network are updated in real time with each training step. The parameters of the target network use a soft update strategy, tracking the parameters of the eval network every 100 rounds and assisting the eval network's training process.

[0115] In the ME-DDPG algorithm's neural network, the actor network takes as input the state information of the drone being pursued, with the input dimension being s_dim (20) in the state space. Its output is the decision action to pursue the drone, with the output dimension being a_dim (2) in the action space. The critic network takes as input the current state and behavior of the drone being pursued, with the dimension being s_dim + a_dim (22). Its output is a judgment on the value of the action taken by the actor network at the current moment, with the dimension being 1. The hidden layers of all four neural networks are fully connected, with four layers.

[0116] After N episode After rounds of game training, the final models of the drone many-to-one pursuit game are obtained.

[0117] The present invention conducts a simulation experiment on the final model of the drone many-to-one pursuit game obtained by the above method:

[0118] In a 500*500 simulation environment, three pursuit drones and one escaping drone are randomly generated. The speed and direction of the drones are randomly generated within their respective value ranges.

[0119] Parameter settings: The actor network learning rate is 0.01, the critic network learning rate is 0.001, the reward discount coefficient is 0.95, and the networks are trained using the Adam optimizer. The size of the storage buffer for storing drone information is 10,000, and 32 sets of information data are randomly extracted from the storage buffer each time for network training. Set the number of training rounds N episode The number of training rounds is 3000, and the maximum number of iterations of the drone in a single training round is 800. The probability p of the pursuit drone using the gradient descent method for action selection is 0.1.

[0120] During the training process, some parameter values ​​of the actor_eval network change as follows Figure 6 As shown in the figure, we can see that in the early stage of training, the actor_eval network parameters are unstable and change significantly. As the number of training rounds increases, the actor_eval network parameters gradually stabilize and the network model tends to converge.

[0121] The probability p of the pursuit drone taking the gradient descent method to select an action cannot be given theoretically, and a suitable value must be obtained in experiments. At the same time, considering that in the initial stage of training, the pursuit drone needs to explore the environment as much as possible, the value of p should not be too large. In this regard, the present invention compares the simulation experiment of 3 pursuit drones chasing 1 escaping drone when the P value is 0.2, 0.1, 0.05, 0.01 and 0. The cumulative reward value of the pursuit drone under different P values ​​changes as shown below. Figure 7 shown.

[0122] When P is 0, the pursuit drone chooses actions entirely based on the output of the actor network, resulting in the lowest cumulative reward. As P increases, the probability of the pursuit drone choosing actions via gradient descent gradually increases, the cumulative reward also increases, and the convergence speed gradually accelerates. However, when P = 0.2, although the reward value reaches a peak at 800 training rounds, it fluctuates significantly as the number of training rounds increases, resulting in poor convergence in subsequent training. Therefore, P = 0.1 is the most appropriate probability parameter overall.

[0123] S6: The final model of the multi-to-one drone pursuit game can output the strategy of chasing drones in real time. After completing the training with the above parameters, in order to test the effect of the model, the pursuit model is used to control the pursuit drones in real time to conduct simulation experiments. The simulation trajectory of the pursuit drone is shown in the figure below. Figure 8 shown.

[0124] The network models under different training rounds were used to conduct simulation experiments on the 3-to-1 drone pursuit mission. The results showed that after 1700 rounds of training, the pursuit drone had a probability of more than 83% to successfully capture the escaping drone. The mission success rate curve is shown in the figure below. Figure 9 shown.

[0125] Finally, it should be noted that, unless mutually inconsistent, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples. The foregoing description is merely an embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. For those skilled in the art, various modifications and variations of the embodiments of this specification are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of this specification shall be included within the scope of the claims of the embodiments of this specification.

Claims

1. A multi-to-one UAV hunting game method based on ME-DDPG, characterized by The steps include: S1: Construct a multi-to-one virtual pursuit game scenario for drones; S2: Determine the nonholonomic motion constraints and environmental range constraints of the pursuing and escaping drones in the pursuit game scenario; S3: Construct reward functions for both the pursuing and escaping drones. The reward functions evaluate the execution of a certain action by both drones in a specific state and output a reward value. S4: The ME-DDPG algorithm is introduced. The ME-DDPG algorithm is an improvement on the DDPG algorithm. By adding the gradient descent method as the training action selection for the pursuit drone, and combining the non-holonomic motion constraint condition, the environmental range constraint condition, and the reward function, the initial pursuit game model of the pursuing and fleeing drones is constructed respectively. S5: Train the initial pursuit game model for both the pursuing and escaping drones, obtain interaction data during the training process, and use this interaction data to continuously update the drones' neural network. After training is complete, the final model for the many-to-one pursuit game is obtained. Input the state information of the pursuing and escaping drones into the model's neural network in real time to determine the action choices of the pursuing drone. This allows multiple pursuing drones to complete the pursuit of the escaping drone with a high success rate. In step S5, the process of training the initial pursuit game model of the pursuing and escaping drones includes: At the beginning of each game, the positions, speeds, and directions of multiple chasing drones and one escaping drone are randomly initialized. The action selection for the pursuit drone is calculated using the gradient descent method on the output of the actor neural network, and noise is superimposed. When the pursuit drone is not within the detection range of the escaping drone, the escaping drone performs random movements. When the pursuit drone appears within the detection range of the escaping drone, the escaping drone will fly towards the area with weak pursuit force and perform maneuvering flight. The information of each step of the drone hunting (s, a, r, s') is saved for network training, where s is the state information of the drone at this moment, a is the action taken by the drone at this moment, r is the reward value of the drone at this moment, and s' is the state information of the drone at the next moment after taking the action; After N training rounds episode After rounds of game training, the final pursuit game model of the chasing and escaping agents is obtained; The action of chasing the drone is to use the output of the actor neural network to calculate using the gradient descent method, and then perform noise superposition. The noise superposition method is OU noise, and the expression is: dz t =θ(μ-z t )dt+ρdW t Among them, z t is the state variable, t represents time; μ is the mean; θ is the rate parameter of mean reversion; ρ is the volatility parameter; W t The OU noise generated by the Wiener process will gradually decrease over time, which can meet the demand for exploring new strategies during the interaction between the UAV and the environment. S6: The final model of the many-to-one drone hunting game can output the strategy for hunting drones in real time.

2. The ME-DDPG-based many-to-one UAV hunting game method according to claim 1, characterized in that: The nonholonomic motion constraints for both the pursuing drone and the escaping drone in the pursuit game scenario are the same, and are expressed as follows: Among them, Q t is the matrix of the drone's information at time t; x t and y t is the position coordinate of the UAV at time t; v t is the speed of the UAV at time t; α t is the azimuth of the UAV at time t; and is the linear acceleration and angular acceleration of the UAV at time t; △t is the simulation step length; Q t+1 is the matrix of the drone's information at time t+1; x t+1 and y t+1 is the position coordinate of the UAV at time t+1; v t+1 is the speed of the UAV at time t+1; α t+1 is the azimuth of the UAV at time t+1; x p ,y p To hunt down the drone's location; x e ,y e is the location of the escaping drone; v p ,α p To chase the drone speed; v e ,α e Speed ​​for escaping drones.

3. The ME-DDPG-based many-to-one UAV hunting game method according to claim 1, characterized in that: The expression of the environmental range constraint condition is: In the above formula: [x min ,x max ] and [y min ,y max ] is the boundary of the virtual environment, x min ,x max is the minimum and maximum value of the horizontal coordinate of the virtual environment; y min ,y max is the minimum and maximum value of the vertical coordinate of the virtual environment; x min ,x max ,y min ,y max A rectangular virtual environment is formed, and both the pursuit drone and the escaping drone are generated and move in the virtual environment; x p ,y p To hunt down the drone's location; x e ,y e is the position of the escaping drone. When the pursuit drone or the escaping drone flies out of any environment boundary, the pursuit mission is considered to have failed. When the distance between the pursuit drone and the escaping drone is less than the safe distance d of the escaping drone, the pursuit mission is considered to have failed. E When the capture was successful.

4. The ME-DDPG-based many-to-one UAV hunting game method according to claim 1, characterized in that: The reward function for chasing the drone includes the ending reward, immediate reward and collision reward, and the expression is: Among them, r i is the reward function for chasing drone i; For the ending reward of hunting down drone i; is the immediate reward function for chasing drone i; is the collision reward function of drone i; End Reward The expression is: Among them, d fight The safe distance at which the escaping drone will not be captured; when the distance between the pursuing drone and the t Less than the safety distance d fight When the pursuit drone successfully captures the escaping drone, a reward of 60 is given to the pursuit drone, and this round ends. When the pursuit drone flies out of the scene boundary, the pursuit mission is considered to have failed, and the round ends based on a reward of -30 for the pursuit drone. When the escaping drone flies out of the scene boundary, the pursuit mission is considered to have failed, and a reward of -30 is given to the pursuit drone, and this round ends. Instant Rewards The expression is: Among them, γ and β are reward weight coefficients, both ranging from 0 to 1; dis is the distance between the pursuit drone i and the escaping drone at the current moment; dis_ is the distance between the pursuit drone i and the escaping drone at the next moment; φ is the angle between the velocity direction of the pursuit drone i and the position line between the pursuit drone i and the escaping drone; Collision reward function The expression is: Among them, d ij is the distance between the chasing drone i and the chasing drone j; d danger To hunt down dangerous distances between drone individuals; d safe The safe distance between individual drones for pursuit; β danger is the reward weight coefficient, ranging from 0 to 1. When the distance between the two pursuing drones is less than the danger distance, a collision is considered to have occurred, and the reward value is -20 based on the drone. When the distance between the two pursuing drones is between the danger distance and the safe distance, the drone is given an appropriate negative reward value according to the formula. When the distance between the two pursuing drones is greater than the safe distance, the collision reward value is 0.

5. The ME-DDPG-based many-to-one UAV hunting game method according to claim 1, characterized in that: The update process of the ME-DDPG algorithm is: The ME-DDPG algorithm contains four neural networks, including two types of neural networks, namely actor network and critic network. The actor network outputs the action selection of the drone, and the critic network evaluates the output of the actor network; the actor network and the critic network each contain two types of networks, namely eval network and target network. The eval network is a real-time updated network and outputs a real-time value. The update of the target network lags behind the eval network and outputs an evaluation value. The four neural networks are actor_eval policy reality network, actor_target policy estimation network, critic_eval value reality network, and critic_target value estimation network. The "actor_eval" policy estimation network and "actor_target" policy reality network are used to select actions; the "critic_eval" value estimation network and "critic_target" value reality network; during the training process, the parameters of the actor_eval strategy reality network and the critic_eval value reality network are updated in real time with each step of training, and the parameters of the actor_target strategy estimation network and the critic_target value estimation network use a soft update strategy, following the parameters of the eval neural network every 100 rounds and assisting the training process of the eval neural network.

6. The ME-DDPG-based many-to-one UAV hunting game method according to claim 1, characterized in that: The state space and action space of the drone correspond to the input and output of the neural network respectively. The state space of the drone, with dimension s_dim of 20, includes the state information of the pursuing drone i itself, the state information of the escaping drone, and the state information of the three drones adjacent to the pursuing drone i. The expression is: S=[x i ,and i ,v i ,α i ,x e ,and e ,v e ,α e ,x j ,and j ,v j ,α j ,x k ,and k ,v k ,α k ,x l ,and l ,v l ,α l ] Among them, S is the state space of the UAV; x i ,y i ,v i ,α i The position, speed and direction of the hunting drone i; x e ,y e ,v e ,α e is the position, speed and direction of the escaping drone; x j ,y j ,v j ,α j ,x k ,y k ,v k ,α k ,x l ,y l ,v l ,α l The position, speed, and direction of the three drones adjacent to the drone i are needed to hunt down the drone; The action space of the drone has a dimension a_dim of 2, including the linear acceleration and angular acceleration of the pursuit drone, and the expression is: A=[δ v ,d a ] Among them, A is the UAV action space; δ v is the linear acceleration of the UAV; δ a Angular acceleration for hunting drones.

7. The ME-DDPG-based many-to-one UAV hunting game method according to claim 1, characterized in that: The N episode The value is greater than 1000.

8. The ME-DDPG-based many-to-one UAV hunting game method according to claim 5, characterized in that: In the neural network of the ME-DDPG algorithm, the actor network inputs the state information of the pursuit drone, and the input dimension is the dimension s_dim of the state space, that is, 20; the output is the decision action of the pursuit drone, and the output dimension is the dimension a_dim of the action space, that is, 2. The critic network inputs the current state information and behavior information of the pursuit drone, and the dimension is s_dim+a_dim, that is, 22; the output is the value judgment of the action taken by the actor network at the current moment, and the dimension is 1. The hidden layers of the four neural networks are all 4-layer fully connected neural networks.

Images