A one-to-one pursuit-and-escape game method for drones based on M2GPI

Abstract

The present invention provides a one-on-one drone pursuit and escape game method based on M2GPI. The method constructs a one-on-one virtual drone pursuit and escape game scenario, determines non-holonomic motion constraints and environmental range constraints, constructs reward functions for the pursuing drone and the escaping drone, respectively, introduces the M2GPI algorithm, constructs initial pursuit and escape game models for both the pursuing and escaping drones, trains the initial pursuit and escape game models for both the pursuing and escaping drones, and uses the final model of the one-on-one drone pursuit and escape game to output a strategy for pursuing the drone in real time. The M2GPI algorithm of the present invention improves and develops on the basis of the Minimax Q algorithm. By incorporating neural networks and generalized policy iteration methods, the convergence speed of the model is increased. The trained pursuit and escape game model has good transferability and can be directly applied to one-on-one drone pursuit and escape tasks in various scenarios and situations, and has a high drone pursuit success rate.

Description

Technical Field

[0001] The present invention relates to the fields of drone pursuit and escape game and reinforcement learning, and in particular to a one-on-one drone pursuit and escape game method. Background Art

[0002] One-on-one drone pursuit missions, one of the most classic scenarios in the field, form the foundation for studying complex multi-drone coordinated pursuit missions. As a typical differential game problem, traditional game theory approaches face modeling difficulties and poor transferability of agent interactions in environments with a large number of agents, a large state space, and complex real-time decision-making. In contrast, reinforcement learning (RL) has garnered significant attention in the field of complex systems and multi-agent systems due to its excellent learning performance. Agents modeled using RL can interact with the environment and other agents to learn from their experiences and improve their behavior.

[0003] However, in the multi-agent domain, reinforcement learning lacks targeted strategies and consideration of the impact of the adversarial nature of the game, resulting in poor convergence and rationality. Therefore, a new approach has been proposed to effectively combine game theory with reinforcement learning algorithms for the drone pursuit problem. By replacing the optimal solution with the equilibrium solution in game theory to obtain an effective strategy, this approach improves convergence efficiency while also making the strategy more rational. Furthermore, the reinforcement learning method's characteristic of agents improving their strategies through interaction with the environment, without requiring a model of the environment, addresses the modeling challenges inherent in game algorithms.

[0004] To solve the zero-sum game problem of multi-agent agents, the Minimax Q algorithm introduced game theory ideas into reinforcement learning. It extended the Q-learning algorithm from single-agent Markov decision processes (MDPs) to two-player zero-sum Markov games (TZMG), learning Nash equilibrium points from training data. It was the first multi-agent reinforcement learning algorithm successfully applied to two-player zero-sum games, but the algorithm was inefficient in training and updating.

[0005] Therefore, how to provide a one-on-one pursuit and escape game method for drones based on game reinforcement learning that takes into account kinematic constraints and can obtain a more real-time and intelligent pursuit strategy is an urgent problem that technicians in this field need to solve. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention provides a one-on-one drone pursuit and escape game method based on M2GPI. The present invention is an improved Minimax Q network learning algorithm based on generalized policy iteration and fitted Q function (M2GPI). This algorithm introduces a neural network to fit the Q function, replacing the Q value table in the Minimax Q algorithm, making the algorithm applicable to large-scale data problems. Secondly, generalized policy iteration (GPI) provides a double-loop iterative method to solve zero-sum games, using generalized policy iteration to solve the Nash equilibrium solution of both drones at each moment. This design optimizes the network training process, and the success rate of drone pursuit missions has also been verified in simulation experiments.

[0007] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0008] S1: Construct a one-on-one virtual pursuit and escape game scenario for drones;

[0009] S2: Determine the nonholonomic motion constraints and environmental range constraints for the pursuing drone and the escaping drone in the pursuit-escape game scenario;

[0010] S3: Construct reward functions for both the pursuing and escaping drones. The reward functions evaluate the execution of a certain action by both drones in a specific state and output a reward value.

[0011] S4: Introducing the M2GPI algorithm, and combining it with non-holonomic motion constraints, environmental range constraints, and reward functions, we construct the initial pursuit-escape game models for both the pursuing and fleeing drones.

[0012] S5: Train the initial pursuit-escape game model for both drones, obtain interaction data during the training process, and use this interaction data to continuously update the drone's real Q network, the player's Q1 network, and the opponent's Q2 network. After training is complete, the final model of the drone's one-on-one pursuit-escape game is obtained.

[0013] S6: The final model of the one-on-one pursuit-and-escape game of drones can output the strategy for chasing drones in real time;

[0014] After the training is completed, the final drone one-on-one pursuit and escape game model is obtained. The state information of the pursuit drone is input into the model's neural network in real time to obtain the action selection of the pursuit drone. The pursuit drone is controlled to complete the pursuit of the escaping drone with a high mission success rate.

[0015] In step S2, the nonholonomic motion constraints of the pursuing drone and the escaping drone in the pursuit-escape game scenario are consistent, and the expression is as follows:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] In the above formula: Q t is the matrix of the drone's information at time t; x t and y t is the position coordinate of the UAV at time t; v t is the speed of the UAV at time t; α t is the azimuth of the UAV at time t; and is the linear acceleration and angular acceleration of the UAV at time t; △t is the simulation step length; Q t+1 is the matrix of the drone's information at time t+1; x t+1 and y t+1 is the position coordinate of the UAV at time t+1; v t+1 is the speed of the UAV at time t+1; α t+1 is the azimuth of the UAV at time t+1; x p ,y p To hunt down the drone's location; x e ,y e is the location of the escaping drone; v p ,α p To chase the UAV speed and direction; v e ,α e The speed and direction of the escaping drone.

[0024] In step S2, the expression of the environmental range constraint condition is:

[0025]

[0026] Among them, x min ,x maxis the minimum and maximum value of the horizontal coordinate of the virtual environment; y min ,y max is the minimum and maximum value of the vertical coordinate of the virtual environment; x min ,x max ,y min ,y max The four points form a rectangular virtual environment, and both the pursuit drone and the escaping drone are generated and move in the virtual environment; p ,y p To hunt down the drone's location; x e ,y e is the position of the escaping drone. When the pursuit drone or the escaping drone flies out of any environment boundary, the pursuit mission is considered to have failed. When the distance between the pursuit drone and the escaping drone is less than the safe distance d of the escaping drone, E When the pursuit was successful.

[0027] The reward function includes a reward function for chasing a drone and a reward function for escaping a drone, wherein the reward function for chasing a drone is expressed as:

[0028]

[0029] Among them, d t is the distance between the pursuit drone and the escaping drone at time t, d E is the safe distance of the escaping drone. When the distance between the pursuing and escaping drones is less than or equal to the safe distance, the pursuing drone is considered to have successfully captured the escaping drone, and a reward value of 100 is given to the pursuing drone, and this round ends. When the pursuing drone exceeds the scene boundary during the pursuit process, a reward value of -100 is given to the pursuing drone, and this round ends. In other cases, the pursuing drone is given an instant reward for each step of the simulation. dis is the distance between the pursuing and escaping drones at time t-1, dis_ is the distance between the pursuing and escaping drones at time t, and β is the reward weight ranging from 0 to 1.

[0030] The reward function of the escaping drone is:

[0031]

[0032] Among them, when the distance between the chasing drones is d t Less than the safety distance d E When the escape drone exceeds the scene boundary, the escape drone is considered to be captured, and a reward value of -100 is given to the escape drone, and this round ends. When the escape drone exceeds the scene boundary, a reward value of 100 is given to the escape drone, and this round ends. In other cases, the escape drone is given an instant reward for each step of the simulation.

[0033] The process of training the initial pursuit-escape game model of the UAVs of both pursuit and escape parties includes:

[0034] At the beginning of each game, the position, speed, and direction of the chasing drone and the escaping drone are randomly initialized;

[0035] The action selection for chasing drones and escaping drones adopts the ε-greedy strategy;

[0036] The information of each step of the pursuit drone (s, a, o, r, s') is saved for network training, where s is the drone state information at this moment, a is the action taken by the drone at this moment, o is the observation information of the drone at this moment, r is the reward value of the drone at this moment, and s' is the state information of the drone at the next moment after taking the action;

[0037] After N episode After rounds of game training, the final pursuit and escape game models of the pursuit and escape agents are obtained, N episode The value is 3000.

[0038] The ε-greedy strategy used by the chasing drone is:

[0039] The pursuit drone randomly selects an action with a probability of ε and selects the action with the largest Q value with a probability of 1-ε. The probability ε ranges from 0 to 1.

[0040] The escaping drone randomly selects an action with probability ε and selects the action with the smallest Q value with probability 1-ε.

[0041] The update process of the M2GPI algorithm is:

[0042] S4.1: Use a deep neural network to simulate the Q-value table in Q-learning. Leveraging the computational simulation capabilities of neural networks, the algorithm model can converge stably in a continuous and complex pursuit-and-escape environment. Construct three fully connected neural networks with the same structure: the real Q network, the Q1 network for the chasing drone player, and the Q2 network for the escaping drone opponent. The parameters of the real Q network are ω, and the parameters are updated in real time. The parameters of the Q1 network are ω1, and the parameters of the real Q network ω are transferred to ω1 for update every 100 steps. The parameters of the Q2 network are ω2, and the parameters of the real Q network ω are transferred to ω1 for update every 1000 steps.

[0043] S4.2: Initialize the replay memory unit buffer, and initialize the environment and state. Use the generalized policy iteration method to improve the strategy when the value function has not converged, reducing the amount of computation. At the same time, there is also an exploration and improvement process to speed up the convergence speed of the pursuit and escape model.

[0044] S4.3: The action selection for the pursuit drone and the escape drone adopts the ε-greedy strategy. That is, the pursuit drone randomly selects an action with a probability of ε and selects the action with the largest Q value with a probability of 1-ε; the escape drone randomly selects an action with a probability of ε and selects the action with the smallest Q value with a probability of 1-ε.

[0045] S4.4: Get the environmental feedback reward and the next moment state, store the sample in the buffer, and save the information of each step of the pursuit drone (s, a, o, r, s') to the buffer for network training. s The state information of the drone at this moment, a is the action taken by the drone at this moment, o is the observation information of the drone at this moment, r is the reward value of the drone at this moment, and s' is the state information of the drone at the next moment after taking the action;

[0046] S4.5: Take a batch of samples (s, a, o, r, s') from the recall replay unit and perform network training updates based on the loss function. The parameters of the actual Q network are updated in real time during each round of training.

[0047] S4.6: When the number of training rounds of the real Q network reaches the number of rounds of Q1 network update, the Q1 network is updated. Otherwise, the Q1 network parameters are not updated and the previous parameters are used to output the results.

[0048] S4.7: When the number of training rounds of the real Q network reaches the number of rounds of Q2 network update, the Q2 network parameters are updated. Otherwise, the Q2 network parameters are not updated and the previous parameters are used to output the results.

[0049] S4.8: After the neural network parameters are updated, it is determined whether the current simulation round has ended. If not, the latest information (s, a, o, r, s') of the pursuit drone is saved to the buffer, and the pursuit drone selects an action for the next simulation. If the current round has ended, the training is complete.

[0050] S4.9: After this round of training is completed, determine whether the entire training process is completed. If not, return to the environment initialization step and start a new round of simulation training. If the entire training process is completed, the training is completed.

[0051] The three neural networks, the real Q network, the drone chasing player Q1 network and the drone escaping opponent Q2 network, all have 6 layers, and the input is the state s=(x p ,y p ,v p ,α p ,x e ,y e ,v e,α e ), the dimension is 8; the output is action a, the dimension is 1, and the selection of action a is shown in Table 1:

[0052] Table 1 UAV action space

[0053]

[0054] Table 1 shows the action space of the drone. The loss function L(ω) of the M2GPI algorithm is:

[0055]

[0056] Where: Q(s,a,o,ω) is the Q value under the real Q network; γ is the reward discount coefficient, π(a′|s′,ω1) is the action strategy selected by the chasing drone player under the Q1 network; Q(s′,a′,o′,ω2) is the Q value of the escaping drone opponent under the Q2 network, s is the current state of the drone, a is the action selected by the drone, o is the observation value of the drone, ω is the real Q network parameter of the drone, Q(s,a,o,ω) is the current Q value of the chasing drone under the real Q network; π(a′|s′,ω1) is the current state of the chasing drone The action strategy selected in the Q1 network of the chasing drone player is evaluated under the current situation, where a' is the action selected by the drone in the Q1 network, s' is the current state of the drone in the Q1 network, and ω1 is the network parameter of the chasing drone player Q1; Q(s',a',o',ω2) is the evaluated Q value of the chasing drone in the Q2 network of the escaping drone under the current situation, where s' is the current state of the drone in the Q2 network, a' is the action selected by the drone in the Q2 network, o' is the observation value of the drone in the Q2 network, and ω' is the network parameter of the escaping drone opponent Q2.

[0057] The present invention provides a one-on-one drone pursuit and escape game method based on M2GPI, which takes into account the nonholonomic constraints, velocity constraints, and acceleration constraints in drone kinematics. The M2GPI algorithm provided by the present invention improves upon the Minimax Q algorithm by incorporating neural networks and generalized policy iteration (GPI) to increase the model's convergence rate. The trained pursuit and escape game model exhibits good transferability and can be directly applied to one-on-one drone pursuit and escape missions in various scenarios and situations, achieving a high drone capture success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flowchart of the M2GPI-based drone one-on-one pursuit and escape game method provided by the present invention.

[0059] Figure 2A schematic diagram of a one-on-one drone pursuit and escape game simulation scenario provided by the present invention.

[0060] Figure 3 This is a schematic diagram of the motion modeling of the pursuit drone provided by the present invention.

[0061] Figure 4 This is a schematic diagram of the situation of the pursuit drone provided by the present invention.

[0062] Figure 5 This is a random initial situation diagram for a one-on-one pursuit-and-escape game of drones provided by the present invention.

[0063] Figure 6 This is a flow chart of the M2GPI algorithm provided by the present invention.

[0064] Figure 7 This is a curve diagram of the changes in some parameters of the "real Q network" provided by the present invention during the training process.

[0065] Figure 8 A trajectory diagram of a pursuit drone controlled by the trained model provided by the present invention to pursue a fleeing drone; Figure 8 (a) is the trajectory diagram of situation 1, Figure 8 (b) is the trajectory diagram of situation 2. Figure 8 (c) is the trajectory diagram of situation three.

[0066] Figure 9 This is a comparison chart of the mission success rates of the M2GPI algorithm and the Minimax Q algorithm provided by the present invention in a one-on-one pursuit and escape game of drones. DETAILED DESCRIPTION

[0067] The present invention will be further described below with reference to the accompanying drawings and examples.

[0068] like Figure 1 The embodiment of the present invention discloses a one-to-one pursuit-and-escape game method for drones based on M2GPI, comprising the following steps:

[0069] S1: Construct a one-on-one virtual pursuit and escape game scenario for drones;

[0070] S2: Determine the nonholonomic motion constraints and environmental range constraints for the pursuing drone and the escaping drone in the pursuit-escape game scenario;

[0071] S3: Construct reward functions for both the pursuing and escaping drones. The reward functions evaluate the execution of a certain action by both drones in a specific state and output a reward value.

[0072] S4: Introducing the M2GPI algorithm, and combining the nonholonomic motion constraint, the environmental range constraint, and the reward function, constructing initial pursuit-escape game models for both the pursuing and fleeing drones;

[0073] S5: Train the initial pursuit-escape game model for both drones, obtain interaction data during the training process, and use this interaction data to continuously update the drone's real Q network, the player's Q1 network, and the opponent's Q2 network. After training is complete, the final model of the drone's one-on-one pursuit-escape game is obtained.

[0074] S6: The final model of the one-on-one pursuit-and-escape game of drones can output the strategy for chasing drones in real time.

[0075] The above steps are described in detail below.

[0076] S1: Construct a one-on-one virtual pursuit and escape game scenario for drones. Figure 2 As shown, the blue drone is the pursuit drone e, and the red drone is the escape drone p, d E is the safe distance of the escaping drone. When the distance between the chasing drone and the escaping drone is less than d E At that time, it was believed that the escaping drone had been successfully pursued.

[0077] S2: Apply nonholonomic motion constraints and environmental range constraints to the pursuit drone and the escaping drone. The motion modeling of the pursuit drone is as follows: Figure 3 As shown, the motion of the drone is described by the position coordinates x, y, velocity magnitude v and velocity direction α. ​​The motion expression is as follows:

[0078]

[0079]

[0080]

[0081]

[0082] In the above formula: Q t is the matrix of the drone's information at time t; x t and y t is the position coordinate of the UAV at time t; v t is the speed of the UAV at time t; α t is the azimuth of the UAV at time t; and is the linear acceleration and angular acceleration of the UAV at time t; △t is the simulation step length; Q t+1 is the matrix of the drone's information at time t+1; x t+1 and yt+1 is the position coordinate of the UAV at time t+1; v t+1 is the speed of the UAV at time t+1; α t+1 is the azimuth of the UAV at time t+1; x p ,y p To hunt down the drone's location; x e ,y e is the location of the escaping drone; v p ,α p To chase the drone speed; v e ,α e Speed ​​for escaping drones.

[0083] The situation of chasing drones Figure 4 As shown in Figure 2, based on the UAV motion constraints and environmental range constraints, the acceleration, velocity, and position of the pursuit UAV and the escaping UAV are restricted as follows:

[0084]

[0085]

[0086]

[0087]

[0088] In the above formula, the speed of the pursuit drone is between [1, 2.5], the speed of the escaping drone is between [1.2, 2.2], the acceleration range of the pursuit drone line is [-1, 1], and the angular acceleration range is [x min ,x max ] and [y min ,y max ] is the boundary of the virtual environment. When the pursuit drone or the escaping drone flies out of the environment boundary, the pursuit mission is considered to have failed. When the distance between the pursuit drone and the escaping drone is less than a certain range, the pursuit is considered to have been successful.

[0089] In the pursuit-escape game, assuming that the state information of both drones is observable, the state of each drone can be determined by its position, speed, and direction. Therefore, the state input dimension of the neural network is 8, and the state space of the drone is:

[0090] S=(x p ,y p ,v p ,α p ,x e ,y e ,v e ,α e )

[0091] The motion control of the pursuit drone is controlled by linear acceleration and angular acceleration, which respectively change the speed and direction of the drone. The linear acceleration and angular acceleration are output as an action pair, that is, the output dimension of the neural network is 1, and the drone's motion space is:

[0092]

[0093] Linear acceleration Can take 3 discrete values: -1, 0, 1; angular acceleration It can take 3 discrete values: Therefore, in each state, the pursuit drone can choose from 9 action pairs.

[0094] S3: Construct the reward function for chasing drones and escaping drones. The expression of the reward function for chasing drones is:

[0095]

[0096] Among them, d t is the distance between the pursuit drone and the escaping drone at time t, d E is the safe distance of the escaping drone. When the distance between the pursuing drones is less than the safe distance, the pursuing drone is considered to have successfully captured the escaping drone, and a reward value of 100 is given to the pursuing drone, and this round ends. When the pursuing drone exceeds the scene boundary during the pursuit process, a reward value of -100 is given to the pursuing drone, and this round ends. In other cases, the pursuing drone is given an instant reward for each step of the simulation. dis is the distance between the pursuing drones at time t-1, dis_ is the distance between the pursuing drones at time t, and β is the reward weight.

[0097] The reward function for escaping the drone is:

[0098]

[0099] Among them, when the distance between the chasing drones is d t Less than the safety distance d E When the escape drone exceeds the scene boundary, the escape drone is considered to be captured, and a reward value of -100 is given to the escape drone, and this round ends. When the escape drone exceeds the scene boundary, a reward value of 100 is given to the escape drone, and this round ends. In other cases, the escape drone is given an instant reward for each step of the simulation.

[0100] S4: Introduce the M2GPI algorithm to construct the initial pursuit-escape game model of the drones of both parties.

[0101] like Figure 6As shown, at the beginning of each game, the position, speed, and direction of the chasing drone and the escaping drone are randomly initialized;

[0102] The M2GPI algorithm provided by the present invention is an improved Minimax Q algorithm, and the algorithm flow chart is as follows: Figure 6 shown

[0103] S4.1: Use a deep neural network to simulate the Q-value table in Q-learning. Leveraging the computational simulation capabilities of neural networks, the algorithm model can converge stably in a continuous and complex pursuit-and-escape environment. Construct three fully connected neural networks with the same structure: the real Q network, the Q1 network for the chasing drone player, and the Q2 network for the escaping drone opponent. The parameters of the real Q network are ω, and the parameters are updated in real time. The parameters of the Q1 network are ω1, and the parameters of the real Q network ω are transferred to ω1 for update every 100 steps. The parameters of the Q2 network are ω2, and the parameters of the real Q network ω are transferred to ω1 for update every 1000 steps.

[0104] S4.2: Initialize the replay memory unit buffer, and initialize the environment and state. Use the generalized policy iteration method to improve the strategy when the value function has not converged, reducing the amount of computation. At the same time, there is also an exploration and improvement process to speed up the convergence speed of the pursuit and escape model.

[0105] S4.3: The action selection for the pursuit drone and the escape drone adopts the ε-greedy strategy. That is, the pursuit drone randomly selects an action with a probability of ε and selects the action with the largest Q value with a probability of 1-ε; the escape drone randomly selects an action with a probability of ε and selects the action with the smallest Q value with a probability of 1-ε.

[0106] S4.4: Get the environmental feedback reward and the next moment state, store the sample in the buffer, and save the information (s, a, o, r, s') of each step of the pursuit drone to the buffer for network training, where s is the drone state information at this moment, a is the action taken by the drone at this moment, o is the observation information of the drone at this moment, r is the drone reward value at this moment, and s' is the state information of the drone at the next moment after taking the action;

[0107] S4.5: Take a batch of samples (s, a, o, r, s') from the recall replay unit and perform network training updates based on the loss function. The parameters of the actual Q network are updated in real time during each round of training.

[0108] S4.6: When the number of training rounds of the real Q network reaches the number of rounds of Q1 network update, the Q1 network is updated. Otherwise, the Q1 network parameters are not updated and the previous parameters are used to output the results.

[0109] S4.7: When the number of training rounds of the real Q network reaches the number of rounds of Q2 network update, the Q2 network parameters are updated. Otherwise, the Q2 network parameters are not updated and the previous parameters are used to output the results.

[0110] S4.8: After the neural network parameters are updated, it is determined whether the current simulation round has ended. If not, the latest information (s, a, o, r, s') of the pursuit drone is saved to the buffer, and the pursuit drone selects an action for the next simulation. If the current round has ended, the training is complete.

[0111] S4.9: After this round of training is completed, determine whether the entire training process is completed. If not, return to the environment initialization step and start a new round of simulation training. If the entire training process is completed, the training is completed.

[0112] The final loss function of the improved M2GPI algorithm is:

[0113]

[0114] Where: r is the reward value at this moment, γ is the reward discount coefficient, ranging from 0 to 1, min o′ To take action o', take the minimum value of escaping the drone, corresponding to min o′ Q(s′,a′,o′,ω2) is the Q value of minimizing the escape drone opponent Q2 network, Q(s,a,o,ω) is the Q value of the real Q network; π(a′|s′,ω1) is the action strategy selected by the chasing drone player Q1 network; after N episode After rounds of game training, the final models of the drone one-on-one pursuit and escape game are obtained.

[0115] S5: After initialization, the initial pursuit-escape game model of the UAVs of both pursuit and escape is trained. The interaction data obtained during training is used to continuously update the neural network of the pursuit UAV. After the training is completed, the final model of the one-on-one pursuit-escape game of UAVs is obtained.

[0116] The present invention conducts a simulation experiment on the final model of the drone one-on-one pursuit and escape game obtained by the above method:

[0117] In a 500*500 simulated environment scene, a pursuit drone and an escaping drone are randomly generated. The speed and direction of the drones are randomly generated within their respective value ranges.

[0118] Parameter settings: The learning rate is set to 0.0009, the reward discount coefficient is 0.99, the size of the storage buffer for storing drone information is 10000, and 32 sets of information data are randomly extracted from the storage buffer each time for network training. The greed coefficient ε-greedy is initially 1 and gradually decreases to 0.001 as the number of training rounds increases. Set the number of training rounds N episode 3000 rounds.

[0119] During the training process, some parameter values ​​of the real Q network change as follows Figure 7 As shown in the figure, we can see that in the early stage of training, the true Q network parameters are unstable and change significantly. As the number of training rounds increases, the true Q network parameters gradually stabilize and the network model tends to converge.

[0120] S6: The final model of the one-on-one pursuit-and-escape game can output a real-time strategy for chasing drones. After training, to test the effectiveness of the model, a simulation experiment was conducted using the pursuit-and-escape model to control the chasing drone in real time. After 1,000 rounds of experiments, the results showed that the chasing drone had a probability of over 80% of successfully capturing the escaping drone. The simulation trajectory of the chasing drone is shown in the figure below. Figure 8 shown.

[0121] In order to verify the superiority of the M2GPI algorithm, the M2GPI algorithm and the Minimax Q algorithm were used to train the drone one-on-one pursuit and escape game, and the network models of the two at different iteration rounds were retained. The final comparison chart of the success rate of capturing the escaping drone is shown in the figure below. Figure 9 As shown in the figure, it is clear that the M2GPI algorithm converges after about 1200 rounds of training, while the Minimax Q algorithm completes network convergence after about 1800 rounds of training. At the same time, the task success rate of the M2GPI algorithm after model convergence is slightly higher than that of the Minimax Q algorithm, proving that the M2GPI algorithm has a faster convergence speed and better task completion effect.

Claims

1. A one-to-one pursuit-and-escape game method for drones based on M2GPI, characterized by The steps include: S1: Construct a one-on-one virtual pursuit and escape game scenario for drones; S2: Determine the nonholonomic motion constraints and environmental range constraints for the pursuing drone and the escaping drone in the pursuit-escape game scenario; S3: Construct reward functions for both the pursuing and escaping drones. The reward functions evaluate the execution of a certain action by both drones in a specific state and output a reward value. S4: Introducing the M2GPI algorithm, and combining it with non-holonomic motion constraints, environmental range constraints, and reward functions, we construct the initial pursuit-escape game models for both the pursuing and fleeing drones. The update process of the M2GPI algorithm is: S4.1: Use a deep neural network to simulate the Q-value table in Q-learning. Leveraging the computational simulation capabilities of neural networks, the algorithm model can converge stably in a continuous and complex pursuit-and-escape environment. Construct three fully connected neural networks with the same structure: the real Q network, the Q1 network for the chasing drone player, and the Q2 network for the escaping drone opponent. The parameters of the real Q network are ω, and the parameters are updated in real time. The parameters of the Q1 network are ω1, and the parameters of the real Q network ω are transferred to ω1 for update every 100 steps. The parameters of the Q2 network are ω2, and the parameters of the real Q network ω are transferred to ω1 for update every 1000 steps. S4.2: Initialize the replay memory unit buffer, and initialize the environment and state. Use the generalized policy iteration method to improve the strategy when the value function has not converged, reducing the amount of computation. At the same time, there is also an exploration and improvement process to speed up the convergence speed of the pursuit and escape model. S4.3: The action selection for the pursuit drone and the escape drone adopts the ε-greedy strategy. That is, the pursuit drone randomly selects an action with a probability of ε and selects the action with the largest Q value with a probability of 1-ε; the escape drone randomly selects an action with a probability of ε and selects the action with the smallest Q value with a probability of 1-ε. S4.4: Get the environmental feedback reward and the next moment state, store the sample in the buffer, and save the information (s, a, o, r, s') of each step of the pursuit drone to the buffer for network training, where s is the drone state information at this moment, a is the action taken by the drone at this moment, o is the observation information of the drone at this moment, r is the drone reward value at this moment, and s' is the state information of the drone at the next moment after taking the action; S4.5: Take a batch of samples (s, a, o, r, s') from the recall replay unit and perform network training updates based on the loss function. The parameters of the actual Q network are updated in real time during each round of training. S4.6: When the number of training rounds of the real Q network reaches the number of rounds of Q1 network update, the Q1 network parameters are updated. Otherwise, the Q1 network parameters are not updated and the previous parameters are used to output the results. S4.7: When the number of training rounds of the real Q network reaches the number of rounds of Q2 network update, the Q2 network parameters are updated. Otherwise, the Q2 network parameters are not updated and the previous parameters are used to output the results. S4.8: After the neural network parameters are updated, determine whether the current simulation round has ended. If not, the action selection for chasing the drone and escaping the drone uses the ε-greedy strategy to select behaviors and interact with the environment. If the current round has ended, the training is complete. S4.9: After this round of training is completed, determine whether the entire training process is completed. If not, return to the environment initialization step and start a new round of simulation training. If the entire training process is completed, the training is completed; S5: Train the initial pursuit-escape game model for both drones, obtain interaction data during the training process, and use this interaction data to continuously update the drone's real Q network, the player's Q1 network, and the opponent's Q2 network. After training is complete, the final model of the drone's one-on-one pursuit-escape game is obtained. S6: The final model of the one-on-one pursuit-and-escape game of drones can output the strategy for chasing drones in real time; After the training is completed, the final drone one-on-one pursuit and escape game model is obtained. The state information of the pursuit drone is input into the model's neural network in real time to obtain the action selection of the pursuit drone. The pursuit drone is controlled to complete the pursuit of the escaping drone with a high mission success rate.

2. The M2GPI-based one-on-one drone pursuit and escape game method according to claim 1 is characterized by: In step S2, the nonholonomic motion constraints of the pursuing drone and the escaping drone in the pursuit-escape game scenario are consistent, and are expressed as follows: In the above formula: Q t is the matrix of the drone's information at time t; x t and y t is the position coordinate of the UAV at time t; v t is the speed of the UAV at time t; α t is the azimuth of the UAV at time t; and is the linear acceleration and angular acceleration of the UAV at time t; △t is the simulation step length; Q t+1 is the matrix of the drone's information at time t+1; x t+1 and y t+1 is the position coordinate of the UAV at time t+1; v t+1 is the speed of the UAV at time t+1; α t+1 is the azimuth of the UAV at time t+1; x p ,y p To hunt down the drone's location; x e ,y e is the location of the escaping drone; v p ,α p To chase the UAV speed and direction; v e ,α e The speed and direction of the escaping drone.

3. The one-to-one pursuit-and-escape game method for drones based on M2GPI according to claim 1 is characterized by: In step S2, the expression of the environmental range constraint condition is: Among them, x min ,x max is the minimum and maximum value of the horizontal coordinate of the virtual environment; y min ,y max is the minimum and maximum value of the vertical coordinate of the virtual environment; x min ,x max ,y min ,y max The four points form a rectangular virtual environment, and both the pursuit drone and the escaping drone are generated and move in the virtual environment; p ,y p To hunt down the drone's location; x e ,y e is the position of the escaping drone. When the pursuit drone or the escaping drone flies out of any environment boundary, the pursuit mission is considered to have failed. When the distance between the pursuit drone and the escaping drone is less than the safe distance d of the escaping drone, E When the capture was successful.

4. The one-to-one pursuit-and-escape game method for drones based on M2GPI according to claim 1 is characterized in that: The reward function includes a reward function for chasing a drone and a reward function for escaping a drone, wherein the reward function for chasing a drone is expressed as: Among them, d t is the distance between the pursuit drone and the escaping drone at time t, d E is the safe distance of the escaping drone. When the distance between the pursuing and escaping drones is less than or equal to the safe distance, the pursuing drone is considered to have successfully captured the escaping drone, and a reward value of 100 is given to the pursuing drone, and this round ends. When the pursuing drone exceeds the scene boundary during the pursuit process, a reward value of -100 is given to the pursuing drone, and this round ends. In other cases, the pursuing drone is given an instant reward for each step of the simulation. dis is the distance between the pursuing and escaping drones at time t-1, dis_ is the distance between the pursuing and escaping drones at time t, and β is the reward weight ranging from 0 to 1.

5. The one-to-one pursuit-and-escape game method for drones based on M2GPI according to claim 1 is characterized in that: The reward function for escaping the drone is: Among them, dis is the distance between the chasing drones at time t-1, dis_ is the distance between the chasing drones at time t, and β is the reward weight ranging from 0 to 1. t Less than the safety distance d E When the escape drone exceeds the scene boundary, the escape drone is considered to be captured, and a reward value of -100 is given to the escape drone, and this round ends. When the escape drone exceeds the scene boundary, a reward value of 100 is given to the escape drone, and this round ends. In other cases, the escape drone is given an instant reward for each step of the simulation.

6. The one-to-one pursuit-and-escape game method for drones based on M2GPI according to claim 1, characterized in that: The process of training the initial pursuit-escape game model of the UAVs of both pursuit and escape parties includes: At the beginning of each game, the position, speed, and direction of the chasing drone and the escaping drone are randomly initialized; The action selection for chasing drones and escaping drones adopts the ε-greedy strategy; The information of each step of the pursuit drone (s, a, o, r, s') is saved for network training, where s is the drone state information at this moment, a is the action taken by the drone at this moment, o is the observation information of the drone at this moment, r is the reward value of the drone at this moment, and s' is the state information of the drone at the next moment after taking the action; After N episode After rounds of game training, the final pursuit and escape game models of the pursuit and escape agents are obtained, N episode The value is 3000; The ε-greedy strategy used by the chasing drone is: The pursuit drone randomly selects an action with a probability of ε, and selects the action with the largest Q value with a probability of 1-ε. The probability ε ranges from 0 to 1. The escaping drone randomly selects an action with probability ε and selects the action with the smallest Q value with probability 1-ε.

7. The one-to-one pursuit-and-escape game method for drones based on M2GPI according to claim 1, characterized in that: The three neural networks, the real Q network, the drone chasing player Q1 network and the drone escaping opponent Q2 network, all have 6 layers, and the input is the state s=(x p ,y p ,v p ,α p ,x e ,y e ,v e ,α e ), the dimension is 8; the output is action a, the dimension is 1, and the action space of the drone is: Linear acceleration Takes 3 discrete values: -1, 0, 1; angular acceleration Takes values ​​from 3 discrete values:

0. Therefore, in each state, the pursuit drone chooses 9 action pairs.

8. The M2GPI-based one-on-one drone pursuit and escape game method according to claim 7 is characterized by: The loss function L(ω) of the M2GPI algorithm is: Where: Q(s,a,o,ω) is the Q value under the real Q network; γ is the reward discount coefficient, π(a′|s′,ω1) is the action strategy selected by the chasing drone player under the Q1 network; Q(s′,a′,o′,ω2) is the Q value of the escaping drone opponent under the Q2 network, s is the current state of the drone, a is the action selected by the drone, o is the observation value of the drone, ω is the real Q network parameter of the drone, Q(s,a,o,ω) is the current Q value of the chasing drone under the real Q network; π(a′|s′,ω1) is the current state of the chasing drone The action strategy selected in the Q1 network of the chasing drone player is evaluated under the current situation, where a' is the action selected by the drone in the Q1 network, s' is the current state of the drone in the Q1 network, and ω1 is the network parameter of the chasing drone player Q1; Q(s',a',o',ω2) is the evaluated Q value of the chasing drone in the Q2 network of the escaping drone under the current situation, where s' is the current state of the drone in the Q2 network, a' is the action selected by the drone in the Q2 network, o' is the observation value of the drone in the Q2 network, and ω' is the network parameter of the escaping drone opponent Q2.

Images