Multi-uav cooperative pursuit-evasion path planning method

Abstract

The application discloses a path planning method for multi-unmanned aerial vehicle cooperative pursuit and evasion. The method comprises the following steps: converting the cooperative pursuit and evasion task into a local observable Markov decision process according to a relative observation state; constructing a centralized critic network based on a graph attention mechanism according to an attribute space vector and a reward function; training a preset multi-agent reinforcement learning network based on the centralized critic network according to the attribute space vector and the reward function; and determining a control instruction of an arbitrary pursuer unmanned aerial vehicle according to the converged multi-agent reinforcement learning network after training. The centralized critic network is used to enable the value network to adaptively focus on the most important neighbor agent information, alleviate the non-stationarity of the multi-agent environment, improve the cooperative decision level of the unmanned aerial vehicle cluster, and guide the pursuers to uniformly disperse and gradually approach from a global perspective by using multiple reward functions, so that the escape loophole caused by the clustering pursuit of multiple pursuer unmanned aerial vehicles is avoided.

Description

Technical Field

[0001] This application relates to a path planning method for multi-UAV collaborative pursuit and escape, belonging to the field of UAV swarm control. Background Technology

[0002] The application of drone swarms is becoming increasingly widespread in fields such as battlefield reconnaissance, border patrol, counter-terrorism and stability maintenance, and emergency search and rescue. In particular, the execution of missions in which multiple drones work together to pursue non-cooperative or even highly adversarial escape targets usually requires maintaining a high degree of flexibility and real-time response capability when facing complex, unknown, and dynamically changing formal environments.

[0003] Current technologies generally employ cooperative pursuit methods based on multi-agent reinforcement learning, leveraging model-free learning capabilities and adaptability to address the aforementioned problems. However, in the application of these methods, multiple agents simultaneously perform policy updates, leading to a highly unstable environment for each agent. Centralized or fully distributed algorithms lack explicit modeling of the complex interactions between agents. In adversarial environments, the escape target's strategy evolves rapidly, causing the pursuing drone's predetermined strategy to fail, easily triggering gradient oscillations or "catastrophic forgetting." Furthermore, in environments with complex obstacle conditions, the pursuing drone is highly susceptible to losing observation information of the escape target, resulting in tracking interruption. Summary of the Invention

[0004] This application discloses a path planning method, device, and storage medium for multi-UAV collaborative pursuit and escape.

[0005] The path planning method for multi-UAV cooperative pursuit and escape in the embodiments of this application includes: Based on the relative observation states of the drone swarm and the escaping drone, the cooperative pursuit task is transformed into a locally observable Markov decision process to define the attribute space vector and reward function of any agent. Each agent corresponds to a pursuer drone or an escaping drone in the drone swarm. Based on the attribute space vector and the reward function, a centralized commentator network based on a graph attention mechanism is constructed. Based on the centralized commentator network, according to the attribute space vector and the reward function, the pursuer drones in the drone swarm perform adversarial games with the updated escape strategy or the historical escape strategy with a preset probability, so as to train the preset multi-agent reinforcement learning network. Based on the converged multi-agent reinforcement learning network after training, control commands are determined for any of the pursuer drones in the drone swarm to complete the collaborative pursuit mission.

[0006] In some implementations, the step of converting the cooperative pursuit task into a locally observable Markov decision process based on the relative observation states of the drone swarm and the escaping drones, to define the attribute space vector and reward function of any agent, includes: Obtain the relative observation status of any pursuer drone and the escaping target in the drone cluster; Based on the relative observation state, a continuous observation space vector is constructed for any of the intelligent agents, wherein the continuous observation space vector includes the self-motion state of the intelligent agent, the adversarial situation state between different intelligent agents, and the local environmental perception state of any intelligent agent in response to obstacles in the environment. Based on the relative observation state, a continuous action space vector is constructed for any agent, and the attribute space vector includes the continuous observation space vector and the continuous action space vector.

[0007] In some implementations, the step of converting the cooperative pursuit task into a locally observable Markov decision process based on the relative observation states of the drone swarm and the escaping drones, to define the attribute space vector and reward function of any agent, further includes: Based on the relative observation state, the reward function is constructed, wherein the reward function includes a global cooperative reward function, an individual tracking reward function, a pursuit posture reward function, a constraint regularization negative reward function, and an escape distance reward function.

[0008] In some implementations, the constraint regularization negative reward function is:

[0009] in Let the constraint regularization negative reward function be... Indicates radar ray parameters, The safe distance between the pursuer drone and the obstacle. The minimum collision distance between the pursuer drone and the obstacle.

[0010] In some implementations, the escape distance reward function is configured to encourage the escaping drone to increase its overall spatial distance from the drone swarm, and the escape distance reward function is:

[0011] in The escape distance reward function is... As for the escape distance reward weight, The number of the pursuer drones. In order to be in The location of the escaping drone as stated at the time. In order to be in Time of the first The location of the pursuer drone mentioned above.

[0012] In some implementations, constructing a centralized commentator network based on a graph attention mechanism according to the attribute space vector and the reward function includes: Based on the continuous observation space vectors corresponding to all the agents, an initial centralized commentator network is constructed based on a preset graph attention neural network basic model, a preset shared weight matrix, a preset weight vector, and a normalization function. Each of the pursuer drones or each of the escape drones in the drone swarm corresponds to a node in the initial centralized commentator network. The initial centralized commentator network includes node features of all the nodes after attention-weighted aggregation. Based on the node features and the continuous action space vectors corresponding to all the agents, global state-action value parameters are determined to construct the centralized critic network.

[0013] In some implementations, training a pre-defined multi-agent reinforcement learning network based on the centralized critic network, according to the attribute space vector and the reward function, includes: The centralized critic network is loaded to evaluate the training process performed by the multi-agent reinforcement learning network; Based on the policy network and value evaluation function of the multiple agents, multiple loss functions are constructed; Using multiple loss functions as optimization objectives and the reward function as the reward value for task completion or failure, the pursuer drones in the drone swarm engage in adversarial game with the updated escape strategy or the historical escape strategy at a preset probability, thereby training the multi-agent reinforcement learning network.

[0014] In some implementations, constructing multiple loss functions based on the policy network of multiple agents and the value evaluation function includes: Within the maximum entropy framework, define the optimization objective loss function for any of the aforementioned agents. Based on the policy coupling of multiple agents, a soft Bellman equation is constructed. Based on the dual-value network structure and the soft Bellman equation, the first loss function corresponding to any agent is determined. Based on KL divergence minimization, the second loss function of the policy network corresponding to any agent in the centralized commentator network is determined; Determine the third loss function corresponding to the entropy network of the agent.

[0015] The electronic device in this application includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the path planning method for multi-UAV collaborative pursuit and escape in the above-described embodiments is implemented.

[0016] The computer-readable storage medium in this application embodiment stores a computer program that, when executed by one or more processors, implements the multi-UAV collaborative pursuit path planning method described above.

[0017] The beneficial effects of this application are as follows: This application utilizes a centralized commentator network based on a graph attention mechanism to represent a multi-agent system as a graph topology, enabling the value network to adaptively focus on the most important neighbor agent information in the current situation, effectively mitigating the non-stationarity of the multi-agent environment and greatly improving the collaborative decision-making level of the drone swarm. In addition, by using multiple reward functions, the pursuers are guided to be evenly dispersed in space and gradually approach from a global perspective, effectively avoiding the escape loophole caused by multiple pursuer drones "clustering" to pursue, and significantly improving the success rate of encirclement against escaping drones. Attached Figure Description

[0018] Figure 1 This is one of the flowcharts illustrating the path planning method for multi-UAV collaborative pursuit in the embodiments of this application; Figure 2 This is the second flowchart illustrating the path planning method for multi-UAV collaborative pursuit in the embodiments of this application; Figure 3 This is the third flowchart illustrating the path planning method for multi-UAV collaborative pursuit in the embodiments of this application; Figure 4 This is the fourth flowchart illustrating the path planning method for multi-UAV collaborative pursuit in the embodiments of this application; Figure 5 This is one of the application scenarios of the path planning method for multi-UAV collaborative pursuit and escape in the embodiments of this application; Figure 6 This is the second schematic diagram of the application scenario of the path planning method for multi-drone collaborative pursuit in the embodiments of this application. Detailed Implementation

[0019] Please see Figure 1 The multi-UAV collaborative pursuit path planning method in this application is used for path planning in collaborative pursuit tasks of UAV swarms. The method specifically includes: Step 01: Based on the relative observation states of the drone swarm and the escaping drones, the cooperative pursuit task is transformed into a locally observable Markov decision process to define the attribute space vector and reward function for any agent. Each agent corresponds to either a pursuer drone or an escape drone in the drone swarm.

[0020] Specifically, the multi-UAV cooperative pursuit path planning method in this application is used to perform path planning for a cooperative pursuit task of a UAV swarm. The cooperative pursuit task refers to the task of all or some of the pursuer UAVs in the UAV swarm cooperating to pursue the escaping UAV. The aforementioned cooperative pursuit means that while each pursuer UAV is chasing the escaping UAV, it avoids positional interference between adjacent pursuer UAVs and searches for the escaping UAV in the environment. Considering the pursuit process of each pursuer UAV against the escaping UAV and the positional relationship between the pursuer UAVs, the aforementioned cooperative pursuit task is first converted into a locally observable Markov decision process (POMDP). This process is an offline pre-training step for training a multi-agent reinforcement learning network. The multi-agent reinforcement learning network is a neural network used to generate control commands corresponding to each pursuer UAV. Each agent in this network corresponds to a UAV, which can be a pursuer UAV in the UAV swarm or an escaping UAV.

[0021] In some implementations, please refer to Figure 2 Step 01 specifically includes: Step 011: Obtain the relative observation status of any pursuer drone and the escaping target in the drone swarm; Step 012: Based on the relative observation state, construct the continuous observation space vector corresponding to any intelligent agent, where the continuous observation space vector includes the agent's own motion state, the adversarial situation state between different intelligent agents, and the local environmental perception state of any intelligent agent in response to obstacles in the environment. Step 013: Based on the relative observation state, construct the continuous action space vector corresponding to any agent. The attribute space vector includes the continuous observation space vector and the continuous action space vector.

[0022] Specifically, regarding the determination process of the attribute space vector, for example, the attribute space vector generally includes two parts: a continuous observation space vector and a continuous action space vector. The continuous observation space vector structure consistently describes the shared continuous observation space of each pursuer and escape drone in the drone swarm, providing decision data for the subsequent training process of multi-agent reinforcement learning networks, and supporting online decision-making in the application of multi-agent reinforcement learning networks. Specifically, let the current time be... , No. The continuous observation space vector of an agent is represented as: Therefore, the aforementioned continuous observation space vector can be expressed in the form shown in Formula 1.

[0023] ... Formula 1 in express Time of the first The motion state corresponding to each intelligent agent. express The state of adversarial relations between different intelligent agents at any given moment. for Time of the first An intelligent agent perceives the local environmental state of obstacles within the environment.

[0024] Regarding one's own movement status It is mainly used to describe the drones corresponding to the intelligent agent in The controllable kinematic state at time t is defined as shown in Equation 2.

[0025] ... Formula 2 in express Time of the first The normalized position of the drone corresponding to each agent in the global plane coordinate system. for Time of the first Each agent corresponds to the normalized yaw angle of the drone. for Time of the first The normalized velocity of each agent, the aforementioned position, yaw angle, and velocity can all be obtained or determined based on the relative observation state of the UAV.

[0026] Regarding the state of confrontation It mainly includes the relative observation relationship between the pursuer drone and the escape drone, assuming the index of the escape drone is... Record its most recently visible location as The confrontational situation (i.e.) Time of the first The relative observations of the pursuer drone to the escaping drone for each agent are shown in Equation 3.

[0027] ... Formula 3 in Represents relative distance term The normalized form, where the relative distance term , for Time of the first The position of the pursuer drone corresponding to each agent in the global plane coordinate system. relative distance term It indicates The Euclidean distance and relative distance term between the pursuer drone and the escape drone at any given time. and its normalized form The relationship between them is: ,in This indicates the maximum detection range of the Chaser drone.

[0028] This indicates the relative azimuth term. The normalized form of the relative azimuth term The azimuth angle used to represent the heading of the escaping drone relative to the pursuer drone; relative azimuth angle term. The definition is shown in Formula 4.

[0029] ... Formula 4 in The function represents normalizing the angle to... The normalization function within the interval, The function is used to calculate the relative position in the global coordinate system. The most recent valid visible position of the escaping drone is... , express Time of the first The yaw angle of the pursuer drone corresponding to each agent. Relative azimuth term. This reflects the direct needs of the pursuer's drone in making turning and maneuvering decisions. The aforementioned relative azimuth angle term... and its normalized form The relationship between them is: .

[0030] also, This is a binary visibility indicator variable. Its purpose is to distinguish between real-time tracking and memory-based search decision modes. For any pursuer drone, when the escaping drone is within the pursuer drone's field of vision, A value of 1 indicates that the current decision-making mode is real-time tracking. Apart from the cases mentioned above, A value of 1 indicates that the current decision mode is based on memory-based search.

[0031] For local environmental perception state Its main function is to act as an explicit obstacle constraint model. For each of the aforementioned pursuer and escape drones, a multi-ray distance sensor is installed. Therefore, for each drone corresponding to an agent, let the set of rays from its multi-ray distance sensor be... for The elements included in this set represent the azimuth angle of any ray in the multi-ray distance sensor in the global coordinate system. This represents the total number of rays emitted by the multi-ray distance sensor. Then, for the _th _ ... A ray is defined by its ray observation, as shown in Equation 5.

[0032] ... Formula 5 in for Time of the first The first agent of the intelligent agent The ray observations corresponding to each ray, This indicates the first ray in a multi-ray distance sensor. A ray, in Time of the first The agents move along the azimuth angle as The distance in the direction of the nearest obstacle. To determine the maximum ray detection range, based on the aforementioned ray observations, the local environmental perception state... This can be represented as shown in Formula 6.

[0033] ... Formula 6 The construction of the continuous observation space vector is now complete.

[0034] For a continuous action space vector, for any intelligent agent and its corresponding drone, it is in Continuous action space vector at time step As shown in Formula 7.

[0035] ... Formula 7 in express Time of the first Each intelligent agent corresponds to a longitudinal acceleration command for the drone. express Time of the first Each agent corresponds to a yaw rate command for the UAV. Both the longitudinal acceleration command and the yaw rate command mentioned above can be obtained or determined based on the UAV's relative observation state.

[0036] The construction of the continuous dynamic space vector is now complete.

[0037] In some implementations, step 01 further includes: Based on the relative observation state, a reward function is constructed, which includes a global collaborative reward function, an individual tracking reward function, a pursuit posture reward function, a constraint regularization negative reward function, and an escape distance reward function.

[0038] Specifically, based on the above implementation method, in addition to the continuously observed spatial vector and the continuously dynamic spatial vector, the locally observable Markov decision process also includes the construction process of multiple reward functions. These reward functions are applicable to both the pursuer UAV and the escaping UAV. For example, the reward functions include at least a global cooperative reward function, an individual tracking reward function, a pursuit attitude reward function, a constraint regularization negative reward function, and an escape distance reward function. The reward functions will be explained below.

[0039] For the global collaborative reward function Its definition is shown in Formula 8.

[0040] ... Formula 8 in To normalize the enclosing quality parameters, This is the preset theoretical maximum angular deviation. express Time of the first The azimuth angle of the pursuer drone relative to the escaping drone for each intelligent agent. For the number of pursuer drones, express The angular interval between adjacent agents and their corresponding pursuer drones is considered. Ideally, the cooperative state of multiple pursuer drones should enable all... The value tends to a constant, that is... .

[0041] For the individual tracking reward function, which is used to encourage the pursuer drone to form an effective tracking of the escaping drone, assuming the first... If the pursuer drone corresponding to an agent is currently in an effective tracking state, an individual tracking reward value is granted and distance differential reward is activated; conversely, if an effective tracking state is not currently established, a slight individual tracking penalty value is granted, forcing the pursuer drone to perform a re-search on the escaping drone, thus avoiding the introduction of misleading gradients during phases where the target is invisible or ineffective pursuit. The aforementioned individual tracking reward value and individual tracking penalty value are the function values ​​of the individual tracking reward function under different conditions. The specific details are shown in Formula 9.

[0042] ... Formula 9 in This is the temperature coefficient.

[0043] The pursuit attitude reward function is constructed based on the angle between the current heading of the pursuer drone and the direction of the escaping drone, and is used to encourage the pursuer drone to maintain an effective pursuit attitude towards the escaping drone. Pursuit Attitude Reward Function Specifically, as shown in Formula 10.

[0044] … Formula 10 in This is the reward value coefficient. The angle between the current heading of the pursuer drone and the direction of the escape drone.

[0045] The main function of the constrained regularized negative reward function is to ensure the overall physical feasibility of the control process. It suppresses the pursuit drone from colliding with obstacles in the environment by introducing a quadratic nonlinear penalty term. The constrained regularized negative reward function is shown in Equation 11.

[0046] ... Formula 11 in To constrain the regularized negative reward function, The radar ray parameters, for example, refer to the specific index directions corresponding to the rays generated by the multi-ray range sensor. To maintain a safe distance between the pursuer drone and obstacles, This represents the minimum collision distance between the pursuer drone and the obstacle.

[0047] The main function of the escape distance reward function is to encourage escaping drones to actively increase the overall spatial distance between themselves and the drone swarm. The escape distance reward function is shown in Equation 12.

[0048] ... Formula 12 in The escape distance reward function is... As for the escape distance reward weight, For the number of pursuer drones, In order to be in The drone is constantly escaping its location. In order to be in Time of the first The location of the Chaser drone.

[0049] At this point, the transformation of the locally observable Markov decision process is complete.

[0050] Please continue reading. Figure 1 The path planning method for multi-UAV collaborative pursuit and escape in the embodiments of this application further includes: Step 02: Construct a centralized commentator network based on the attribute space vector and the reward function.

[0051] Specifically, based on the above implementation method, the next step is to use the continuous observation space vector, continuous action space vector and multiple reward functions as environmental parameters to train the preset graph neural network model, so as to construct a centralized commentator network based on graph attention mechanism. This centralized commentator network is used to design the policy network when training the preset multi-agent reinforcement learning network, thereby conducting self-game heterogeneous cooperation-adversarial training, so that the pursuer drone can engage in adversarial games with the latest escape strategy or the historical escape strategy with a certain probability.

[0052] Furthermore, in some implementations, please refer to Figure 3 Step 02 specifically includes: Step 021: Based on the continuous observation space vectors corresponding to all agents, and using a pre-defined graph attention neural network basic model, a pre-defined shared weight matrix, a pre-defined weight vector, and a normalization function, construct an initial centralized commentator network. Each pursuer drone or each escape drone in the drone swarm corresponds to a node in the initial centralized commentator network, which includes the node features of all nodes after attention-weighted aggregation. Step 022: Determine the global state-action value parameters based on node features and the continuous action space vectors corresponding to all agents to construct a centralized critic network.

[0053] Specifically, based on the above implementation method, the training model of the graph neural network model is initialized, and the training process of the centralized critic network is realized by using the continuous observation space vector, the continuous action space vector and multiple reward functions as environmental parameters.

[0054] First, let the time step corresponding to each moment be . The system formed by the aforementioned multiple intelligent agents can be represented as a graph structure. ,in This represents a set of nodes, which includes all the aforementioned agents, with each node corresponding to one agent. A set of edges representing a node, used to describe the interaction relationships between agents.

[0055] The graph attention neural network model (GAT network) described above is a neural network model designed for graph structures. The network input consists of node features. , ,in The total number of nodes. Given the number of features for each node, the network output is the number of node features. , ,in The number of shuchu features for each node. To ensure that the input features can be transformed into higher-level features, a pre-defined shared weight matrix is ​​used for linear transformation of each node's features, followed by a self-attention mechanism calculation for the nodes, as shown in Equation 13.

[0056] ... Formula 13 in This represents a pre-defined trainable weight vector. The attention coefficient represents the neighboring nodes. For nodes The importance of The above is the shared weight matrix. To enhance the comparability between different nodes, the attention coefficients of all neighboring nodes j of node i are normalized using the softmax function, as shown in Equation 14.

[0057] ... Formula 14 in The attention coefficient mentioned above The corresponding normalized attention coefficient, Represents a node The set of all neighboring nodes. Next, based on the normalized attention coefficients mentioned above, the output features of each node can be determined according to the input features, as shown in Equation 15.

[0058] ... Formula 15 in and Represents any two adjacent nodes. The nonlinear activation function can be exemplified by the LeakyReLU function. This attention-weighted feature aggregation process essentially implements an implicit credit allocation mechanism: agents more critical to the current decision have higher weights in the value function representation, while the influence of less relevant agents is naturally suppressed. Compared to methods that explicitly decompose global rewards, this approach improves the discriminativeness and stability of value evaluation without additional constraints. At this point, the initial centralized critic network based on the GAT network is complete.

[0059] Next, to integrate the outputs of the GAT network to form the final centralized critic network, for example, the joint observations and joint actions are first preprocessed using a two-layer fully connected network (FC network) MLP, with the ReLU activation function applied between the two layers. The processed data will then be concatenated as the network input, as shown in Equation 16.

[0060] ... Formula 16 in For any node The concatenated vector is obtained by concatenating the continuous observation space vector and the continuous action space vector of the corresponding intelligent agent.

[0061] Furthermore, the concatenated vector is further concatenated with the network's output features, and the resulting vector is input into a new MLP using ReLU as the activation function. The final layer directly outputs the global state-action value. Specifically, as shown in Formula 17. At this point, the centralized commentator network is complete, and its output is the global state-action value described above. .

[0062] ... Formula 17 Please continue reading. Figure 1 The path planning method for multi-UAV collaborative pursuit and escape in the embodiments of this application further includes: Step 03: Based on the centralized commentator network, according to the attribute space vector and the reward function, the pursuer drones in the drone swarm perform adversarial games with the updated escape strategy or the historical escape strategy with preset probabilities, so as to train the preset multi-agent reinforcement learning network.

[0063] Specifically, based on the above implementation method, and after the centralized commentator network is constructed, the next step is to use the continuous observation space vector, continuous action space vector, various reward functions, and centralized commentator network in the above implementation method as data foundation. The multi-agent reinforcement learning network, including the above-mentioned agents, is trained by having each pursuer drone in the drone swarm play adversarial games with the updated escape strategy or the historical escape strategy until the output of the multi-agent reinforcement learning network converges.

[0064] In some implementations, please refer to Figure 4 Step 03 further includes: Step 031: Load a centralized critic network to evaluate the training process of the multi-agent reinforcement learning network; Step 032: Construct multiple loss functions based on the policy network of multiple agents and the value evaluation function; Step 033: Using multiple loss functions as optimization objectives and a reward function as the reward value for task completion or failure, the pursuer drones in the drone swarm are made to perform adversarial games with the updated escape strategy or the historical escape strategy with preset probabilities, so as to train the multi-agent reinforcement learning network.

[0065] Specifically, for the training process of a multi-agent reinforcement learning network, firstly, the centralized critic network constructed in the above embodiments is loaded to act as the critic network for the training process of the multi-agent reinforcement learning network. Next, based on the preset reinforcement learning network, multiple loss functions are constructed, and the specific training process is executed with the above loss functions as the optimization objective.

[0066] In some embodiments, step 032 further includes: Within the maximum entropy framework, we define an optimization objective loss function for any agent. Based on the policy coupling of multiple agents, a soft Bellman equation is constructed. Based on the dual-value network structure and the soft Bellman equation, the first loss function for any agent is determined. Based on minimizing KL divergence, the second loss function of the policy network corresponding to any agent in a centralized commentator network is determined. Determine the third loss function corresponding to the entropy network of the agent.

[0067] For example, the various loss functions mentioned above are as follows: First, within the maximum entropy framework, for any intelligent agent... The corresponding optimization objective function is shown in Equation 18.

[0068] ... Formula 18 in For the first The optimization objective function corresponding to each agent. In a multi-agent reinforcement learning network, the first... The policy network corresponding to each agent. for Discount factor of time, for Time of the first The sum of all the above reward function values ​​for each agent, For the first Temperature coefficient corresponding to each agent The entropy represents the policy, specifically the policy. In the observation space vector The degree of randomness is defined as shown in Formula 19.

[0069] ... Formula 19 Furthermore, unlike single-agent reinforcement learning, in the multi-agent reinforcement learning network described above, the policies of each agent are coupled with each other. The centralized critic network described above is used to evaluate the observation vector and action vector, and the resulting soft Bellman equation is shown in Equation 20.

[0070] … Formula 20 in The value function representing the joint observation vector and action vector is specifically the output of the aforementioned centralized critic network. This is the state value function in the soft Bellman equation. The global observation space vector is formed by concatenating all continuous observation space vectors at the current moment. The global action space vector is formed by concatenating all continuous action space vectors. The global observation space vector is formed by concatenating all consecutive observation space vectors at the next moment. Indicates the discount factor. This represents the sum of all reward function values. The state value function is... It is rewritten in the form shown in Formula 21.

[0071] ... Formula 21 Based on the aforementioned soft Bellman equation, the multi-agent reinforcement learning network generally employs a dual-value network structure, that is, it uses two action value functions and respectively constructs corresponding target value networks, as shown in Equation 20. Generally, two groups are used. During training, in each iteration, a target value network with a smaller value function value is selected to reduce overestimation bias. Based on the soft Bellman equation and the dual-value network structure described above, a target value network is constructed for the first... The first loss function of each agent is shown in Equation 22.

[0072] ... Formula 22 in For the first loss function mentioned above, This is the target value used to calculate the loss when the network is updated.

[0073] Furthermore, for policy networks The corresponding second loss function is obtained based on the KL divergence and optimized with the goal of minimizing its value. The specific second loss function is shown in Equation 23.

[0074] ... Formula 23 in This is the second loss function mentioned above.

[0075] Furthermore, by taking into account both value functions, low variance gradient estimation can be achieved for the second loss function mentioned above, as shown in Equation 24.

[0076] ... Formula 24 in It refers to the minimum value of the function among two sets of value functions.

[0077] Furthermore, entropy networks possess the ability to automatically adjust their temperature coefficients, thereby maintaining a balance between exploration and exploitation. (Agent) The corresponding entropy network is The third loss function at time t is shown in Equation 25.

[0078] ... Formula 25 in For the third loss function mentioned above, The target entropy. When the policy entropy is lower than the target value, the third loss function... The optimization direction is to increase the temperature coefficient. This further increases the importance of the policy term in the process of minimizing the loss function. When the policy entropy is higher than the target value, the third loss function... The optimization direction is to reduce the temperature coefficient. This allows the system to maintain a high degree of exploration in the early stages of training and automatically reduce randomness during the convergence phase.

[0079] Next, taking the loss functions in the above implementation as optimization objectives and the reward functions as reward values ​​for task completion or failure, the historical policy pool of the escapee is dynamically maintained, so that the pursuer drones in the drone swarm can perform adversarial games with the updated escape policy or the historical escape policy with preset probabilities, in order to train the multi-agent reinforcement learning network.

[0080] For example, within the aforementioned multi-agent reinforcement learning network framework, a historical opponent pool training mechanism based on self-play is introduced. By randomly sampling adversarially between the current policy and historical policies, the speed of policy evolution and training stability are effectively balanced. During training, the escape policy is considered the "environment side" in the adversarial game, and its changes have a decisive impact on the learning stability of the pursuer drone. To avoid the pursuer drone overfitting to only a single, rapidly evolving escape policy, this invention employs a periodic snapshot mechanism to archive escape policies. Specifically, after a fixed number of training rounds, the network parameters corresponding to the escaped drone in the current multi-agent reinforcement learning network are saved as a historical policy and added to the historical opponent pool. This historical pool gradually expands during the training process, forming a set of escape policies covering different stages and ability levels, thereby providing the pursuer drone with diverse and representative adversarial samples.

[0081] At the start of each training round, the escape strategy faced by the Chaser drone is not always the latest strategy, but rather sampled probabilistically between the current main strategy and the historical opponent pool strategies. Specifically, the Chaser drone selects an escape strategy with a certain probability. To counter the escape strategy currently being learned, ensuring the strategy continues to evolve and remains adaptable to the best current opponent; The probability is used to randomly select an escape strategy from the historical opponent pool as the opponent strategy, in order to enhance the robustness of the strategy against different adversarial styles and ability levels. When a strategy from the historical opponent pool is used for adversarial training, the network parameters corresponding to that escape strategy are frozen and only participate in the interaction as part of the environmental dynamics, without participating in gradient updates. When the game mode is restored, the latest escape strategy is switched back and joint training continues until the output of the multi-agent reinforcement learning network converges to a preset range, thus completing the training process.

[0082] Please continue reading. Figure 1 The path planning method for multi-UAV cooperative pursuit and escape in the embodiments of this application further includes: Step 04: Based on the converged multi-agent reinforcement learning network after training, determine the control commands for any pursuer drone in the drone swarm to complete the collaborative pursuit mission.

[0083] Specifically, based on the above implementation method, a multi-agent reinforcement learning network that converges after training is used as input. The local real-time observation data of each pursuer drone in the drone swarm regarding the environment and the escaping drone are used as input. Forward reasoning and calculation are performed using the multi-agent reinforcement learning network to obtain control commands for each pursuer drone. This controls the flight attitude and direction of each pursuer drone in space, thereby enabling all pursuer drones in the entire drone swarm to cooperate with each other to complete the cooperative pursuit and escape mission in the above implementation method.

[0084] The following examples illustrate the path planning method for multi-UAV collaborative pursuit in the above embodiments.

[0085] Example: Please refer to Figure 5 In this embodiment, a digital simulation method is used, and a 100-pixel × 100-pixel square area is set as the simulation test scene. Rectangular obstacles (gray rectangular blocks in the figure) are randomly distributed in the area. The drone swarm is set to include 3 pursuer drones and one adversarial escape drone.

[0086] Figure 6 This paper demonstrates the changing trends of the average reward and success rate of the agent corresponding to the pursuer drone in the cooperative pursuit task described above, as the number of training iterations increases. As training progresses, the agent corresponding to the pursuer drone gradually learns effective cooperative pursuit strategies, causing the system's average reward to rise rapidly, while the task success rate curve also shows a corresponding continuous upward trend. This indicates that in this stage, the agent corresponding to the pursuer drone gradually learns effective cooperative pursuit strategies and can complete the task more efficiently. Although the reward curve fluctuates to some extent in the mid-training stage, it maintains a stable upward trend overall. It is evident that the introduction of an adversarial training mechanism and the design of constructing a historical opponent strategy pool effectively alleviates the non-stationarity and pursuer strategy forgetting problems caused by continuous policy changes in multi-agent reinforcement learning, improving the stability and robustness of policy learning. This shows that the self-game mechanism of introducing a historical opponent pool can maintain policy diversity while avoiding drastic oscillations during the training process. In the later stages of training, the average reward curve gradually stabilizes, the reward fluctuation amplitude decreases significantly, and the capture success rate stabilizes at a high level, indicating that the policy network parameters have basically converged. Overall analysis shows that the reward function and the success rate curve exhibit a relatively consistent trend, indicating that the reward function design can effectively reflect the task objective and provide a stable learning signal for strategy optimization. The proposed multi-agent reinforcement learning algorithm based on the self-game mechanism can achieve stable convergence within a limited number of training rounds and gradually learn an effective multi-agent cooperative pursuit strategy, laying a good strategic foundation for subsequent pursuit and escape missions in complex environments.

[0087] Please refer to the previous document. Figure 5 , Figure 5 The trajectories of both the pursuer and the fleeing drone are shown in the path planning method of multi-drone cooperative pursuit based on the above-described embodiments, demonstrating their ability to counter-escapism. The trajectories reveal that, through adversarial training, both sides have learned complex and targeted movement patterns. When the pursuer drone approaches, the fleeing drone moves towards obstacles, using these obstacles to block the pursuer's line of sight and thus disrupting the pursuer's direct pursuit path. The fleeing drone moves along the edge of obstacles or even around them, making it difficult for the pursuer drone to form a direct approach route. These behavioral patterns indicate that, during adversarial training, the fleeing drone gradually learns to utilize environmental structures to increase its escape space, thereby prolonging the time it is locked on.

[0088] The electronic device in this application includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the path planning method for multi-UAV collaborative pursuit and escape in the above-described embodiments is implemented.

[0089] The computer-readable storage medium in the embodiments of this application stores a computer program that, when executed by one or more processors, implements the path planning method for multi-UAV collaborative pursuit described above.

[0090] Thus, this application utilizes a centralized commentator network based on a graph attention mechanism to represent the multi-agent system as a graph topology, enabling the value network to adaptively focus on the most important neighboring agent information in the current situation. This effectively alleviates the non-stationarity of the multi-agent environment and greatly improves the collaborative decision-making level of the drone swarm. In addition, by using multiple reward functions, the pursuers are guided to disperse evenly in space and gradually approach from a global perspective, effectively avoiding escape vulnerabilities caused by multiple pursuer drones "clustering" together, and significantly improving the success rate of encirclement against escaping drones.

[0091] The above description is merely a preferred embodiment of this application and is not intended to limit this application in any way. Although this application has disclosed the preferred embodiment as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the technical solution of this application, based on the technical essence of this application and within the spirit and principles of this application, shall still fall within the protection scope of the technical solution of this application.

Claims

1. A path planning method for multi-UAV collaborative pursuit, characterized in that, The method is used for path planning in collaborative pursuit missions involving drone swarms, and the method includes: Based on the relative observation states of the drone swarm and the escaping drone, the cooperative pursuit task is transformed into a locally observable Markov decision process to define the attribute space vector and reward function of any agent, wherein each agent corresponds to a pursuer drone or an escaping drone in the drone swarm. Based on the attribute space vector and the reward function, a centralized commentator network based on a graph attention mechanism is constructed. Based on the centralized commentator network, according to the attribute space vector and the reward function, the pursuer drones in the drone swarm perform adversarial games with the updated escape strategy or the historical escape strategy with a preset probability, so as to train the preset multi-agent reinforcement learning network. Based on the converged multi-agent reinforcement learning network after training, control commands are determined for any of the pursuer drones in the drone swarm to complete the collaborative pursuit mission.

2. The method according to claim 1, characterized in that, The step of converting the cooperative pursuit task into a locally observable Markov decision process based on the relative observation states of the drone swarm and the escaping drones, to define the attribute space vector and reward function of any agent, includes: Obtain the relative observation status of any pursuer drone and the escaping target in the drone cluster; Based on the relative observation state, a continuous observation space vector is constructed for any of the intelligent agents, wherein the continuous observation space vector includes the self-motion state of the intelligent agent, the adversarial situation state between different intelligent agents, and the local environmental perception state of any intelligent agent in response to obstacles in the environment. Based on the relative observation state, a continuous action space vector is constructed for any agent, and the attribute space vector includes the continuous observation space vector and the continuous action space vector.

3. The method according to claim 2, characterized in that, The step of converting the cooperative pursuit task into a locally observable Markov decision process based on the relative observation states of the drone swarm and the escaping drones, to define the attribute space vector and reward function of any agent, further includes: Based on the relative observation state, the reward function is constructed, wherein the reward function includes a global cooperative reward function, an individual tracking reward function, a pursuit posture reward function, a constraint regularization negative reward function, and an escape distance reward function.

4. The method according to claim 3, characterized in that, The constrained regularization negative reward function is: in Let the constraint regularization negative reward function be... Indicates radar ray parameters, The safe distance between the pursuer drone and the obstacle. The minimum collision distance between the pursuer drone and the obstacle.

5. The method according to claim 3, characterized in that, The escape distance reward function is configured to encourage the escaping drone to increase its overall spatial distance from the drone swarm. The escape distance reward function is: in The escape distance reward function is... As for the escape distance reward weight, The number of the pursuer drones. In order to be in The location of the escaping drone at that moment. In order to be in Time of the first The location of the pursuer drone mentioned above.

6. The method according to claim 2, characterized in that, The step of constructing a centralized commentator network based on a graph attention mechanism according to the attribute space vector and the reward function includes: Based on the continuous observation space vectors corresponding to all the agents, an initial centralized commentator network is constructed based on a preset graph attention neural network basic model, a preset shared weight matrix, a preset weight vector, and a normalization function. Each of the pursuer drones or each of the escape drones in the drone swarm corresponds to a node in the initial centralized commentator network. The initial centralized commentator network includes node features of all the nodes after attention-weighted aggregation. Based on the node features and the continuous action space vectors corresponding to all the agents, global state-action value parameters are determined to construct the centralized critic network.

7. The method according to claim 1, characterized in that, The training of a pre-defined multi-agent reinforcement learning network based on the centralized critic network, according to the attribute space vector and the reward function, includes: The centralized critic network is loaded to evaluate the training process performed by the multi-agent reinforcement learning network; Based on the policy network and value evaluation function of the multiple agents, multiple loss functions are constructed; Using multiple loss functions as optimization objectives and the reward function as the reward value for task completion or failure, the pursuer drones in the drone swarm engage in adversarial game with the updated escape strategy or the historical escape strategy at a preset probability, thereby training the multi-agent reinforcement learning network.

8. The method according to claim 6, characterized in that, The process involves constructing multiple loss functions based on the policy network of multiple agents and the value evaluation function, including: Within the maximum entropy framework, define the optimization objective loss function for any of the aforementioned agents. Based on the policy coupling of multiple agents, a soft Bellman equation is constructed. Based on the dual-value network structure and the soft Bellman equation, the first loss function corresponding to any agent is determined. Based on KL divergence minimization, the second loss function of the policy network corresponding to any agent in the centralized commentator network is determined; Determine the third loss function corresponding to the entropy network of the agent.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the path planning method for multi-UAV collaborative pursuit as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the path planning method for multi-UAV collaborative pursuit as described in any one of claims 1-8.

Images