Deep reinforcement learning multi-unmanned aerial vehicle cooperative path planning method of HCA-MAPPO

Abstract

The invention discloses a deep reinforcement learning multi-unmanned aerial vehicle cooperative path planning method of an HCA-MAPPO, and relates to multi-agent reinforcement learning and unmanned aerial vehicle cluster control. From a multi-unmanned aerial vehicle cluster collaborative path planning method, a simulation environment comprising a three-degree-of-freedom unmanned aerial vehicle kinematic model and a dual-coordinate system management system is established, and accurate simulation of a complex geographic scene is realized; a deep reinforcement learning method based on hierarchical cross attention HCA and multi-agent near-end strategy optimization MAPPO is provided; the method comprises the following steps: introducing an HCA executor network for performing priority processing on multi-source heterogeneous information; an internal curiosity module based on collaborative consistency gating is designed, so that the effective exploration efficiency in a sparse reward environment is improved; the invention provides a tangent guidance mixed reward shaping method based on conflict perception, and solves the problem of local optimum in path planning. And the situation awareness capability, the training convergence efficiency and the path planning success rate of the unmanned aerial vehicle cluster are obviously improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent reinforcement learning and unmanned aerial vehicle swarm control, in particular to a deep reinforcement learning multi-unmanned aerial vehicle cooperative path planning method based on HCA-MAPPO. BACKGROUND

[0002] Multi-unmanned aerial vehicle swarm cooperative path planning is a research hotspot in the field of multi-agent systems. In recent years, multi-agent reinforcement learning (MARL) has become the mainstream technology route in this field due to its potential in handling complex decision-making problems. Among them, the multi-agent proximal policy optimization (MAPPO) algorithm is widely used due to its efficiency and stability in cooperative tasks. As Yu, C. et al. explained in their paper, MAPPO uses a centralized critic network to evaluate the global state to guide the learning of each independent, decentralized actor network through the structure of "centralized training, decentralized execution" (CTDE), effectively alleviating the non-stationarity problem in the multi-agent environment.

[0003] Although MAPPO provides a solid framework for multi-agent cooperation, existing technical solutions based on this framework still have the following shortcomings when applied to complex unmanned aerial vehicle swarm cooperative path planning tasks:

[0004] First, there are limitations in the perception information processing mechanism of the agent. To process the multi-source heterogeneous information faced by the UAV and solve the information aggregation problem in multi-agent reinforcement learning, existing technologies have begun to integrate attention mechanisms into the algorithm framework. In the latest research, Wu, J. et al. (Wu J, Li D, Yu Y, et al. An Attention Mechanism and Adaptive Accuracy Triple-Dependent MADDPG Formation Control Method for Hybrid UAVs, IEEE Transactions on Intelligent Transportation Systems, 2024, 25(9): 11648-11663) introduced the attention mechanism into the centralized critic network (Critic) of the multi-agent algorithm to achieve selective attention to the information of other agents, thereby improving the efficiency of formation control. However, such methods usually adopt a "flattened" attention mode, i.e., the information of all other agents is processed and weighted at the same level, and the main purpose is to solve the information aggregation problem of large-scale agents. These methods do not achieve strict and prioritized sequential processing of different types of entities (such as leaders, obstacles, and ordinary teammates) according to the intrinsic logic of decision-making (e.g., safety over task). Existing technologies generally lack a perception mechanism that can deeply couple entity relationship modeling and decision-making priority, resulting in insufficient efficiency and robustness of decision-making in critical situations that require multi-objective trade-offs.

[0005] Second, at the training level, existing exploration enhancement mechanisms fail to adequately address the inherent conflict between individual exploration and collective goals, particularly the reward sparsity problem prevalent in complex collaborative tasks. To resolve this issue, curiosity-driven intrinsic rewards have been introduced to incentivize exploration. For example, Ningombam, DD, YooB, Kim HW, et al. CuMARL: curiosity-based learning in multiagent reinforcement learning, IEEE Access, 2022, 10: 87254-87265, proposed a curiosity mechanism based on conditional mutual information (CMI), which generates intrinsic rewards by quantifying the "knowledge differences" among agents to promote information-level exploration. However, these methods typically employ a general curiosity mechanism, where any state that can bring new knowledge or information generates exploration incentives. While this mechanism can promote exploration, it may incentivize individual agents to engage in ineffective explorations that are not beneficial to the overall task goal of the cluster (e.g., leaving the formation to satisfy curiosity), failing to effectively resolve the contradiction between individual exploration and collective goals. Existing technologies generally lack a lightweight, decentralized solution that directly couples individual exploration with dynamic collaborative goals at the agent level.

[0006] Third, regarding the guidance mechanism for path planning, existing reinforcement learning-based methods are still generally plagued by the "local minimum" problem of the traditional artificial potential field method proposed by Khatib, O. in his classic paper. This problem manifests as follows: when the agent, obstacle, and target point are approximately aligned in a straight line, the implicit attraction and repulsion from the reward function cancel each other out, leading to deadlock. Solving this problem is currently a hot topic in this field. For example, recent research such as Liu, H., et al. (Liu H, Long X, Li Y, et al. Adaptive multi-UAV cooperativepath planning based on novel rotation artificial potential fields, Knowledge-Based Systems, 2025, 317: 113429) proposed a novel Rotational Artificial Potential Field (RAPF), which guides the UAV out of trouble by introducing an additional rotational force and uses multi-agent reinforcement learning to optimize the parameters of the force field. However, such methods typically rely on modifying the force field structure, i.e., continuously applying a rotational component. Existing technologies generally lack a more adaptive mechanism that can proactively and explicitly identify conflicting "locally optimal" states first, and then dynamically switch the entire guidance logic to make more decisive bypass actions. Summary of the Invention

[0007] The purpose of this invention is to address the problems of flattened multi-source heterogeneous information processing, ineffective exploration under sparse rewards, local optimum traps in path planning, and low collaborative efficiency in existing technologies. It provides a deep reinforcement learning multi-UAV collaborative path planning method, HCA-MAPPO, which simultaneously improves the situational awareness capability, training convergence efficiency, and path planning success rate of UAV swarms.

[0008] A deep reinforcement learning-based multi-UAV cooperative path planning method for HCA-MAPPO includes the following steps:

[0009] 1) Establish a multi-UAV simulation environment: Starting from the multi-UAV swarm collaborative path planning method, establish a simulation environment that includes a three-degree-of-freedom UAV kinematic model and a dual-coordinate system management system to achieve accurate simulation of complex geographical scenes;

[0010] 2) Design a hierarchical cross-attention (HCA) mechanism to introduce the actor network to realize priority perception and processing of multi-source heterogeneous information; divide the observed information into its own state, leader entity state, environmental obstacle entity state, and other follower entity states, and process multi-source heterogeneous information through a sequential three-layer cross-attention mechanism of intention perception layer, threat perception layer, and collaborative perception layer to output the control actions of the UAV.

[0011] 3) A decentralized collaborative control mechanism based on a weighted consensus algorithm calculates the dynamic consistency of each follower's target point in real time, including target position, velocity, and acceleration, as observation input and reward benchmark;

[0012] 4) Construct an intrinsic curiosity module based on cooperative consistency gating to guide the agent to explore effectively. Generate original novelty signals by distilling RND through random networks and adjust the intrinsic reward intensity through a Gaussian cooperative gating function.

[0013] 5) Design a tangent-guided hybrid reward function based on conflict perception. The conflict state is determined by calculating the angle between the attractive and repulsive force vectors. When the angle is greater than or equal to a preset threshold, the function switches to tangent-guided mode to solve the "local optimum" problem in path planning.

[0014] 6) The MAPPO paradigm update strategy is adopted. The critic network learns the global value function by minimizing the temporal difference mean square error; the executor network updates by maximizing the pruning agent objective function and adding an entropy regularization term; a phased training mode of leader training first and follower training later is adopted, and the performance is verified based on a multi-process parallel simulation platform.

[0015] In step 1), a first-order inertial system is used to implement discrete state updates. The UAV's state at discrete time steps is described by the following first-order inertial system. The actual state change process within:

[0016]

[0017] in, , , These are the time constants for the speed, climb angle, and heading angle responses, respectively. These parameters can be adjusted according to the maneuverability of different UAVs; in particular, for heading angles with periodicity... The update uses normalization processing; the angle error used for updating... Instead of directly subtracting the instruction value from the current value, the original difference is normalized and mapped to... Within a certain range, the shortest angle difference between the two is obtained; this method ensures that the drone always turns to a new heading along the most efficient path, effectively avoiding angle crossings. The problem of invalid long-distance steering caused by the boundary; in the updated... After determining the kinematic state at each moment, these states are decomposed into velocity vectors of the UAV in three-dimensional space through triangular projection in a standard aerospace coordinate system. ,in:

[0018]

[0019] The position of the UAV at the next moment can be calculated by performing Euler forward integration on the velocity vector. This completes one iteration of the simulation step.

[0020]

[0021] In step 1), the dual-coordinate system management system uses a preset datum origin in the geodetic coordinate system. The system uses a three-dimensional Cartesian coordinate system with the origin at 0.5. Its X-axis points due east, its Y-axis points due north, and its Z-axis is perpendicular to the local ground plane and points upwards. At the start of each simulation round, the system performs a planar approximation transformation based on the WGS-84 Earth model to adjust the geodetic coordinates of any entity. Convert to local northeast celestial coordinates First, convert all angle units to radians, then perform the following calculations:

[0022]

[0023] in, The average radius of the Earth in the WGS-84 model is given. Throughout the simulation, all state updates and calculations are performed within this Euclidean space, which greatly simplifies physical calculations and provides more easily processed, linear input features for neural networks. When the simulation ends or visualization output is required, the system then accurately restores the recorded local ENU coordinate trajectory to geodetic coordinates through the following inverse transformation. :

[0024]

[0025] This dual-coordinate system management method ensures the computational efficiency and stability of the simulation process.

[0026] In step 2), the hierarchical cross-attention (HCA) agent network is based on the hierarchical cross-attention (HCA) mechanism. To facilitate this mechanism, the agent's raw observation information is first processed in a structured manner; a single follower agent... Observations It is divided into four parts: its own state Leader entity status Physical state of environmental obstacles and the status of other follower entities ;in Represents the number of corresponding entities. Represents its feature dimension;

[0027] The HCA enforcer network performs prioritized data preprocessing on the aforementioned structured information through a sequential three-layer cross-attention mechanism. First, it maps various types of original state information to the high-dimensional feature space of the attention mechanism through independent linear embedding layers with non-linear activation functions.

[0028]

[0029] in, and These are the learnable weight matrix and bias vector for the corresponding embedding layer. This serves as the initial query vector; subsequently, the network computes according to the following hierarchical logic:

[0030] Layer 1 (Intent Awareness Layer): The agent observes the leader's state to clarify the task intent; it utilizes the initial query vector. For the leader entity Perform multi-head cross-attention calculation to generate leader context vector. This calculation is performed by It consists of several parallel attention heads, each head Independently compute a weighted context; for the first Size, its attention weight The calculation is as follows:

[0031]

[0032] in, Is the query vector and the first Scaling dot product similarity score among leader entities:

[0033]

[0034] express The first of the matrix Line; based on this weight, the first The output for each head is:

[0035]

[0036] in, It is the first The learnable projection matrix corresponds to each head; the outputs of all heads are concatenated and then passed through a linear projection to obtain the final context vector:

[0037]

[0038] The second layer (threat perception layer): After clarifying the task intent, the agent shifts its focus to environmental threats; the leader context vector generated in the first layer is then used... With the initial query vector The vectors are concatenated and then passed through an update module (a linear layer with non-linear activation) to generate an enhanced query vector. :

[0039]

[0040] This enhanced query vector For environmental obstacle entities Perform a second multi-head cross-attention calculation to generate obstacle context vectors. Its calculation process is exactly the same as the first layer, only the input is replaced with... ;

[0041] The third layer (cooperative perception layer): After fully perceiving the task intent and environmental threats, the agent ultimately focuses on the cooperative needs within the cluster; again, it uses the obstacle context vector generated in the second layer. Enhanced query vectors from the preceding sequence The vectors are then concatenated and generated through a separate update module to produce the final query vector. :

[0042]

[0043] The final query vector For other follower entities Perform a third multi-head cross-attention calculation to generate the cluster context vector. Its calculation process is exactly the same as the first layer, only the input is replaced with... ;

[0044] Final decision: The network embeds the initial query vector Leader context vector Obstacle context vector and cluster context vector The final feature fusion is performed; this fusion is achieved by concatenating the four vectors:

[0045]

[0046] The fused high-dimensional feature vector Input to a policy network In the end, the control actions of the drone are output. .

[0047] In step 3), the core driving force of the decentralized collaborative control mechanism based on the weighted consensus algorithm originates from a configurable communication topology adjacency matrix. ,in This represents the number of all drones in the cluster (including leader and followers); the matrix defines the information interaction topology within the cluster, where matrix elements... Indicates drone Capable of sensing drones The state information (position and velocity) is given, and if it is not given, it is 0; by designing the asymmetry of this matrix, complex command and coordination relationships can be easily constructed.

[0048] At each time step For any follower agent in the cluster The system first uses the matrix Determine the set of all its neighbors. Subsequently, a weighted consensus algorithm is used to calculate the agent's... The complete and consistent target state at the current moment, including the target location. Target speed Even target acceleration This calculation is a weighted average of its own state and the states of all its neighbors:

[0049]

[0050] in, and These represent the position and velocity vectors, respectively. It is an intelligent agent Its own weighting coefficient, and It is with the neighbor Configurable weighting coefficients related to the type (leader or follower); This is the total weight, used for normalization; furthermore, to enable the agent to better predict the target's dynamics, the target's acceleration is estimated by performing a backward difference on the target's velocity over time.

[0051]

[0052] This dynamically calculated consistency target state As part of the structured observation information, it is input into the HCA agent network, enabling its decision-making process to fully perceive its dynamic and personalized task objectives. At the same time, it becomes the core benchmark for reward function design, and the agent's behavior will be rewarded or punished based on how to track this target point most efficiently and safely. Finally, at the cluster level, through each agent continuously pursuing its own dynamically changing consistent objectives, the entire cluster can emerge with the desired, flexible, and adaptive cooperative formation and path planning behavior in a completely decentralized manner.

[0053] In step 4), the specific steps for constructing the intrinsic curiosity module based on cooperative consistency gating, generating the original novelty signal through random network distillation (RND), and adjusting the intrinsic reward intensity through a Gaussian cooperative gating function can be as follows:

[0054] Through a fixed, random target network and a learnable prediction network The novelty of the quantification state between prediction errors; for agents New situation encountered Its original novelty signal It is defined as the square of the Euclidean distance between the output vectors of two networks:

[0055]

[0056] in, These are the learnable parameters of the prediction network;

[0057] A collaborative consensus gating mechanism is proposed to regulate and filter the original novelty information. This mechanism applies to each follower agent. Introduce a cooperative gating function The value of this function depends on the degree of consistency between the agent's current behavior and the cluster's collaborative goal, which is determined by the agent's current position. Its dynamic "consistent movement of target points" Euclidean distance between To quantize; gate function Designed to be distance-dependent A function that increases and monotonically decreases, specifically in the form of a Gaussian function:

[0058]

[0059] in, ; It is a feature follower radius used to control the decay rate. It is a positive hyperparameter; when the agent closely follows ( Approaching 0, When the value approaches 1, the agent is allowed to receive the full novelty reward; when the agent deviates significantly from its cooperative goal ( Much greater than 0 The value approaches 0), thereby suppressing the invalid novelty reward generated by deviation from the route;

[0060] The effective intrinsic reward obtained by the agent after gating regulation Defined as the product of the original novel signal and the cooperative gating function: By introducing a collaborative gating mechanism, the exploratory behavior of agents is effectively coupled with the macro-level collaborative goals of the cluster; this ultimately provides an effective intrinsic reward. The statistics are normalized at runtime before being used for policy updates.

[0061] In step 5), the design of the conflict-aware tangential-guided hybrid reward function involves designing multi-dimensional reward functions with clear objectives and independent mutualities for agents (leaders and followers) at different levels. These include: arrival reward R1, proximity penalty R2, collision penalty R3, conflict-aware intelligent guidance R4, distance change reward R5, smoothing reward R6, time-coordination reward R7, leader-to-leader exclusion reward R8, and falling behind penalty R9. R1 to R6 are reward functions shared by both leader and follower drones, R7 and R8 are reward functions unique to the leader, and R9 is a reward function unique to the follower. The total reward for the leader is... Total reward for followers ;in and It is a constant between 0 and 1;

[0062] (1) Reaching reward item R1

[0063] Reaching reward item R1 is achieved through phased achievement rewards. and end mission rewards constitute; For the sum of a series of discrete reward terms, when the drone Location First crossing of the preset distance threshold Triggered at time: ,in For indicator functions; Then, when the drone first enters the target radius During this time, a huge positive reward is given. ;

[0064] (2) Proximity penalty term R2:

[0065] When the distance between the drone and environmental obstacles or the ground is less than a preset safety threshold At that time, a nonlinear repulsive field penalty is applied; for any obstacle The penalty for approaching drones is as follows:

[0066] if

[0067] in The horizontal distance from the drone to the obstacle surface; for collision avoidance in the ground direction, the penalty is:

[0068] if

[0069] in This refers to the drone's altitude above the ground. For positive numbers, penalty term The sum of the two penalties above is:

[0070]

[0071] (3) Collision penalty R3:

[0072] When a drone collides with an obstacle or the ground, that is or This results in a large negative reward:

[0073]

[0074] in, Take a large positive number;

[0075] (4) Conflict-aware intelligent guidance item R4:

[0076] The core idea behind this reward is to vectorize the drone's current velocity. This is coupled with a dynamically calculated "safety guidance vector" capable of switching modes based on the situation. Alignment is performed in the direction between them, and the degree of alignment is quantified by the cosine similarity between them:

[0077]

[0078] "Safety Guidance Vector" The dynamic generation mechanism involves the system first performing conflict detection at each time step; specifically, the system calculates the points pointing to the global target point. The attractive vector And the total repulsive force vector generated by all hazards in the environment (obstacles, ground, etc.) Subsequently, the angle between these two key vectors was calculated. This allows for precise quantification of the degree of conflict faced in the current decision-making process; when the angle approaches 180 degrees, it indicates that the directions of attraction and repulsion are almost completely opposite, meaning that the drone has fallen into a deadlock state.

[0079]

[0080] Based on the results of the aforementioned conflict perception, the guidance mechanism will dynamically switch between the two modes; when the calculated angle... Less than a preset conflict threshold When the system determines that the decision-making conflict is not severe, the standard potential field guidance mode is adopted, and the guidance vector is directly formed by the superposition of attractive and repulsive forces. When the included angle When the value is greater than or equal to this threshold, the system determines that the drone has fallen into a "local optimum" and automatically switches to tangential guidance mode; in this mode, the system will calculate a vector related to the total repulsive force vector. Orthogonal "tangent vectors" And select with attraction vector The included angle is acute; the final guiding vector is redefined as a weighted combination of the attraction and the tangent vector. ;

[0081] (5) Distance Change Bonus Item R5:

[0082] Distance change reward is a type of reward that provides dense feedback based on the change in distance between the agent and the target point over consecutive time steps. ,in For the target location, The current time step is the location of the drone. The location of the drone at the previous time step; It is a positive number;

[0083] (6) Smoothing the reward item R6:

[0084] To encourage smooth control, the reward / penalty is applied to the current action. With recent average movement of step Deviation between: ,in , It is a positive number;

[0085] (7) Time-based synergy reward R7 and inter-leader exclusion reward R8:

[0086]

[0087] in This indicates the time-coordination reward, representing whether leaders L0 and L1 arrive at the target point synchronously; This indicates that leaders exclude certain reward items; and It is a positive number;

[0088] (8) Penalty for falling behind (R9):

[0089] When the distance between the follower and its consistency target point Exceeding a preset "maximum following radius" At that time, apply a penalty term proportional to the distance:

[0090] if

[0091] in , It is a positive number.

[0092] In step 6), the MAPPO paradigm update strategy is adopted: the critic network learns the global value function by minimizing the temporal difference mean square error; the executor network updates by maximizing the pruning agent objective function and adding an entropy regularization term.

[0093] The underlying policy optimization and network update follow the basic paradigm of Multi-Agent Proximal Policy Optimization (MAPPO); after collecting enough interaction data, the policy is learned by optimizing two networks: the Actor and the Critic.

[0094] Critics Network Update: Shared Critics Network The accurate global state value function is learned by minimizing the temporal difference error; its loss function is... Using mean square error form:

[0095]

[0096] in, The target value is calculated using the Generalized Advantage Estimation (GAE) method;

[0097] Enforcer Network Update: Each Individual Enforcer Network By maximizing a pruned surrogate objective function The strategy is updated to improve sample utilization efficiency while ensuring update stability; its objective function is defined as follows:

[0098]

[0099] in, , is the probability ratio of the new strategy to the old strategy. It is the advantage function calculated by the critic network. This involves pruning hyperparameters; to encourage exploration, a regularization term for policy entropy is usually added to the final loss function.

[0100] This invention starts with a collaborative path planning method for multi-UAV swarms, establishing a simulation environment that includes a three-degree-of-freedom UAV kinematic model and a dual-coordinate system management system, achieving accurate simulation of complex geographical scenes. Addressing the shortcomings of existing deep reinforcement learning methods in environmental perception, exploration efficiency, and path planning decision-making, this invention proposes a deep reinforcement learning method based on hierarchical cross-attention (HCA) and multi-agent proximal policy optimization (MAPPO). This method introduces an HCA actor network for prioritizing multi-source heterogeneous information. An intrinsic curiosity module based on collaborative consistency gating is designed to improve effective exploration efficiency in sparse reward environments. Furthermore, a conflict-aware "tangent-guided" hybrid reward shaping method is pioneered to solve the "local optimum" problem in path planning. Finally, simulation experiments of collaborative path planning for multi-UAV swarms were completed in a Python environment, verifying that compared with traditional reinforcement learning methods, it can significantly improve the situational awareness capability, training convergence efficiency, and path planning success rate of UAV swarms.

[0101] Compared with the prior art, the advantages of the present invention are:

[0102] First, compared to the flat information processing mechanisms commonly used in existing technologies, this invention proposes an agent network based on hierarchical cross-attention (HCA), which can achieve priority perception and processing of multi-source heterogeneous environmental information, and has stronger situational understanding and decision-making capabilities. Existing technologies usually input observation information from different entities such as obstacles, teammates, and leaders into multilayer perceptron networks without discrimination. The network needs to implicitly and inefficiently learn the importance of different information from massive amounts of data. The HCA network designed in this invention, however, imitates the internal logic of agent decision-making through a sequential processing flow: first, focus on the leader to clarify the intention, then focus on obstacles to avoid threats, and finally focus on teammates to achieve cooperation. This structured sequential attention mechanism enables the agent to dynamically allocate attention to different types of environmental information, improving the agent's situational awareness capabilities in complex dynamic environments. Its performance and efficiency are superior to traditional flat attention or fully connected network structures.

[0103] Second, this invention designs an intrinsic curiosity module based on cooperative consistency gating, which solves the problem of ineffective exploration that may occur in existing general exploration mechanisms in multi-agent cooperative tasks, significantly improving the training efficiency of the algorithm and the effectiveness of the final strategy. Existing technologies, such as random network distillation (RND), which are directly applied, can promote exploration, but may incentivize individual agents to break away from the group to satisfy their own curiosity, engaging in explorations that are not beneficial or even harmful to the overall task objective of the cluster. The cooperative gating mechanism proposed in this invention effectively couples exploration behavior with the macro-level cooperative objective of the cluster by dynamically linking the intensity of intrinsic rewards with the degree to which agents follow the cluster's "consistency objective." This mechanism encourages agents to explore only when they maintain a good cooperative state, effectively avoiding destructive exploration behavior, greatly improving the effectiveness of exploration, and accelerating the learning of meaningful cooperative strategies.

[0104] Thirdly, addressing the "local optimum" trap prevalent in existing path planning methods, stemming from traditional artificial potential field methods, this invention proposes a conflict-aware "tangential guidance" hybrid reward shaping method. This enables the agent to make more decisive and efficient obstacle avoidance maneuvers. Existing reward-guided reinforcement learning methods often cause the agent to hesitate when the drone and obstacle are in a head-on confrontation due to the mutual cancellation of guidance signals. This invention introduces a conflict-aware module at the reward function level, enabling real-time monitoring of the angle between the attractive and repulsive force vectors. When the system is determined to be trapped in a "local optimum" dilemma, the guidance mechanism automatically switches from the standard potential field superposition mode to the novel "tangential guidance" mode, providing the agent with a clear and effective lateral obstacle avoidance direction. This method effectively solves the local optimum problem in traditional guidance methods, enabling the agent to learn more robust and efficient obstacle avoidance strategies, significantly improving task success rate and path planning reliability. Attached Figure Description

[0105] Figure 1 Complete flowchart of this invention;

[0106] Figure 2 HCA-MAPPO algorithm framework diagram;

[0107] Figure 3 Hierarchical Cross-Attention (HCA) mechanism structure diagram;

[0108] Figure 4 Schematic diagram of normalization matrix and consensus target point calculation; the left figure shows the normalization matrix solution process, and the right figure shows the consensus algorithm target point calculation process;

[0109] Figure 5 The intrinsic curiosity module structure diagram of collaborative consistency gating;

[0110] Figure 6 Leader-Follower Path Planning Trajectory Map;

[0111] Figure 7 Comparison of convergence of reward functions for different algorithms;

[0112] Figure 8 A graph showing the change in distance from the leader to the target point;

[0113] Figure 9 Distance variation curves among members within the formation. The top graph represents formation one, and the bottom graph represents formation two. Detailed Implementation

[0114] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments will be used to further illustrate this invention in conjunction with the accompanying drawings.

[0115] This invention provides a collaborative path planning method for multi-UAV swarms based on HCA-MAPPO deep reinforcement learning. This method aims to systematically address the shortcomings of existing technologies in environmental perception, exploration efficiency, and path planning decision-making, thereby improving the collaborative capabilities and mission success rate of UAV swarms in complex environments. First, this invention establishes a simulation environment including a three-degree-of-freedom UAV kinematic model and a dual-coordinate system management system. Then, it proposes a collaborative path planning algorithm based on HCA-MAPPO, which follows a reinforcement learning framework of "centralized training and decentralized execution," and designs the swarm's state space, action space, and a multi-dimensional reward function oriented towards hierarchical tasks. In terms of algorithm structure, this invention introduces a hierarchical cross-attention (HCA) mechanism into the agent network to achieve priority perception and processing of multi-source heterogeneous information. Simultaneously, to address the problem of insufficient exploration efficiency in sparse reward environments, this invention designs an intrinsic curiosity module based on collaborative consistency gating to guide the agent in effective exploration. Finally, to solve the "local optimum" problem in path planning, a conflict-aware "tangential guidance" hybrid reward shaping method is proposed. The overall process of this invention is as follows: Figure 1 As shown.

[0116] The purpose of this invention is to provide an effective intelligent collaborative path planning method for multi-UAV swarms. Under this method, the UAV swarm can learn efficient collaborative strategies, achieve intelligent obstacle avoidance and adaptive formation in complex environments, and ultimately complete collaborative tasks in complex obstacle environments.

[0117] Step 1: Set up the simulation environment

[0118] 1. Establish a kinematic model for the unmanned aerial vehicle (UAV).

[0119] This invention employs a three-degree-of-freedom (3-DOF) point mass kinematics model to describe the motion of a single unmanned aerial vehicle (UAV). This model abstracts the UAV as a point mass; in this model, the UAV's motion state is determined by its velocity magnitude *f*, its angle of climb, and its position. (Defines the flight path direction of the UAV in the vertical plane) and heading angle (The flight direction of the drone in the horizontal plane is defined) and jointly determined; the actions output by the intelligent agent serve as command values. This invention describes the UAV in discrete time steps using the following first-order inertial system. The actual state change process within:

[0120]

[0121] in, , , These are the time constants for the speed, climb angle, and heading angle responses, respectively. These parameters can be adjusted according to the maneuverability of different UAVs; in particular, for heading angles with periodicity... For the update, this invention employs normalization processing; the angle error used for updating... Instead of directly subtracting the instruction value from the current value, the original difference is normalized and mapped to... Within a certain range, the shortest angle difference between the two is obtained; this method ensures that the drone always turns to a new heading along the most efficient path, effectively avoiding deviations due to angle crossings. The problem of invalid long-distance steering caused by the boundary; in the updated... After determining the kinematic state at each moment, these states are decomposed into velocity vectors of the UAV in three-dimensional space through triangular projection in a standard aerospace coordinate system. ,in:

[0122]

[0123] Finally, by performing Euler forward integration on the velocity vector, the position of the UAV at the next moment is calculated. This completes one iteration of the simulation step.

[0124]

[0125] 2. Conversion between the geodetic coordinate system and the local northeast-sky coordinate system

[0126] This invention employs a dual-coordinate system management system; this system consists of a geodetic coordinate system (LLA) and a local east-north-up (ENU) coordinate system; all core kinematic simulations, state observation construction, and reinforcement learning algorithm iterations are performed within the local east-north-up coordinate system; this coordinate system is based on a preset datum origin in the geodetic coordinate system. The system uses a three-dimensional Cartesian coordinate system with the origin at 0.5. Its X-axis points due east, its Y-axis points due north, and its Z-axis is perpendicular to the local ground plane and points upwards. At the start of each simulation round, the system performs a planar approximation transformation based on the WGS-84 Earth model to adjust the geodetic coordinates of any entity. Convert to local northeast celestial coordinates First, convert all angle units to radians, then perform the following calculations:

[0127]

[0128] in, Let be the average radius of the Earth in the WGS-84 model. Throughout the simulation, all state updates and calculations are performed within this Euclidean space, which greatly simplifies physical calculations and provides more easily processed, linear input features for neural networks. When the simulation ends or visualization output is required, the system then accurately restores the recorded local ENU coordinate trajectory to geodetic coordinates through the following inverse transformation. :

[0129]

[0130] This dual-coordinate system management method ensures the computational efficiency and stability of the simulation process.

[0131] Step 2: Propose a multi-UAV swarm cooperative path planning algorithm based on HCA-MAPPO.

[0132] 1. Networks and Structured Observation Space Based on Hierarchical Cross-Attention (HCA)

[0133] Figure 2 The framework diagram of the multi-UAV cooperative path planning based on deep reinforcement learning based on HCA-MAPPO of the present invention is given. Figure 3 A diagram of the hierarchical cross-attention mechanism (HCA) is provided.

[0134] To achieve intelligent perception and decision-making regarding multi-source heterogeneous environmental information, this invention designs a novel actor network, the core of which is a hierarchical cross-attention (HCA) mechanism. To complement this mechanism, the raw observation information of the agents is first processed in a structured manner. A single follower agent... Observations It is divided into four parts: its own state Leader entity status Physical state of environmental obstacles and the status of other follower entities ;in Represents the number of corresponding entities. This represents its characteristic dimension.

[0135] The HCA enforcer network performs prioritized data preprocessing on the aforementioned structured information through a sequential three-layer cross-attention mechanism. First, various types of raw state information are mapped to the high-dimensional feature space of the attention mechanism through independent linear embedding layers with non-linear activation functions:

[0136]

[0137] in, and These are the learnable weight matrix and bias vector for the corresponding embedding layer. This is the initial query vector. Subsequently, the network computes according to the following layered logic:

[0138] Layer 1 (Intent Awareness Layer): The agent first focuses on the leader's state to clarify the task intent. This is achieved using the initial query vector. For the leader entity Perform multi-head cross-attention calculation to generate leader context vector. This calculation is performed by It consists of several parallel attention heads, each head Compile a weighted context independently. For the first... Size, its attention weight The calculation is as follows:

[0139]

[0140] in, Is the query vector and the first Scaling dot product similarity score among leader entities:

[0141]

[0142] express The first of the matrix Okay. Based on this weight, the first... The output for each head is:

[0143]

[0144] in, It is the first Each head has a corresponding learnable projection matrix. The outputs of all heads are concatenated and then subjected to a linear projection to obtain the final context vector:

[0145]

[0146] The second layer (threat perception layer): After clarifying the task intent, the agent then focuses on environmental threats. The leader context vector generated in the first layer is then... With the initial query vector The vectors are concatenated and then passed through an update module (a linear layer with non-linear activation) to generate an enhanced query vector. :

[0147]

[0148] This enhanced query vector For environmental obstacle entities Perform a second multi-head cross-attention calculation to generate obstacle context vectors. Its calculation process is exactly the same as the first layer, only the input is replaced with... .

[0149] The third layer (cooperative perception layer): After fully perceiving the task intent and environmental threats, the agent ultimately focuses on the cooperative needs within the cluster. Again, the obstacle context vector generated in the second layer... Enhanced query vectors from the preceding sequence The vectors are then concatenated and generated through a separate update module to produce the final query vector. :

[0150]

[0151] The final query vector For other follower entities Perform a third multi-head cross-attention calculation to generate the cluster context vector. Its calculation process is exactly the same as the first layer, only the input is replaced with... .

[0152] Final decision: The network embeds the initial query vector Leader context vector Obstacle context vector and cluster context vector The final feature fusion is then performed. This fusion is achieved by concatenating the four vectors:

[0153]

[0154] The fused high-dimensional feature vector Input to a policy network In the end, the control actions of the drone are output. .

[0155] 2. Cooperative Control Based on Weighted Consensus Algorithm

[0156] To achieve complex distributed collaborative tasks with multiple starting points and synchronized endpoints, and to enable multi-UAV swarms to break free from the constraints of fixed formations and form adaptive formation behavior, this invention proposes a decentralized collaborative control mechanism based on a weighted consensus algorithm. The purpose of this mechanism is to dynamically and in real-time generate a personalized, time-varying "consistent moving target point" for each independent follower agent in the swarm. This virtual target point not only provides a clear local navigation reference but also macroscopically couples the behavior of the entire swarm together, achieving distributed collaboration.

[0157] The core driving force of this mechanism stems from a configurable communication topology adjacency matrix. ,in This represents the total number of drones in the cluster (including leader and followers). The matrix defines the information interaction topology within the cluster, where matrix elements... Indicates drone Capable of sensing drones The state information (position and velocity) is given, and vice versa. By designing the matrix asymmetrically, complex command and coordination relationships can be easily constructed. For example, by setting only some followers to be able to perceive the first leader, while other followers can only perceive the second leader, the division of squads into clusters can be achieved from a topological perspective.

[0158] At each time step For any follower agent in the cluster The system first uses the matrix Determine the set of all its neighbors. Subsequently, a weighted consensus algorithm was used to calculate the agent's... The complete and consistent target state at the current moment, including the target location. Target speed Even target acceleration This calculation is a weighted average of its own state and the states of all its neighbors:

[0159]

[0160] in, and These represent the position and velocity vectors, respectively. It is an intelligent agent Its own weighting coefficient, and It is with the neighbor Configurable weighting coefficients related to the type (leader or follower); This is the total weight, used for normalization. Furthermore, to enable the agent to better predict the target's dynamics, the target's acceleration is estimated by performing a backward difference on the target's velocity over time.

[0161]

[0162] This dynamically calculated consistency target state It plays a crucial role in this invention: on the one hand, as part of the structured observation information, it is input into the HCA agent network, enabling its decision-making process to fully perceive its dynamic and personalized task objectives; on the other hand, it becomes the core benchmark for reward function design, with agent behavior revolving around how to most efficiently and safely track this target point for rewards and penalties. Ultimately, at the cluster level, through each agent continuously pursuing its own dynamically changing consistent objective, the entire cluster can emerge with the desired, flexible, and adaptive cooperative formation and path planning behavior in a completely decentralized manner.

[0163] Figure 4 A schematic diagram illustrating the calculation of the normalized matrix and the consensus target point; the left diagram shows the process of solving the normalized matrix, and the right diagram shows the process of calculating the target point using the consensus algorithm.

[0164] 3. Intrinsic Curiosity Module Based on Cooperative Consistency Gating

[0165] To address the reward sparsity problem commonly encountered in complex tasks such as long-range and multi-target collaboration of UAV swarms, and to overcome the drawbacks of "ineffective exploration" that may result from traditional intrinsic curiosity mechanisms in collaborative tasks, this invention designs and implements a novel intrinsic curiosity module based on collaborative consistency gating within the HCA-MAPPO framework. Figure 5 The intrinsic curiosity module structure diagram of the collaborative consistency gating is given.

[0166] This module is still based on the principle of Random Network Distillation (RND), which uses a fixed, random target network. and a learnable prediction network The novelty of a state is quantified by the prediction error between the two states. For an agent... New situation encountered Its original novelty signal It is defined as the square of the Euclidean distance between the output vectors of two networks:

[0167]

[0168] in, These are the learnable parameters of the prediction network. However, this invention recognizes that in multi-agent cooperative tasks, not all novel states are beneficial. An agent that deviates significantly from the formation and abandons the cooperative task to satisfy its own curiosity is detrimental to the entire cluster. To address this issue, this invention proposes a cooperative consistency gating mechanism to regulate and filter the original novelty information.

[0169] This mechanism applies to each follower agent. Introduce a cooperative gating function The value of this function depends on the degree of consistency between the agent's current behavior and the cluster's collaborative goal. In this invention, this degree of consistency is determined by the agent's current position. Its dynamic "consistent movement of target points" Euclidean distance between To quantize. Gating function. Designed to be a distance-dependent A function that increases and monotonically decreases, specifically in the form of a Gaussian function:

[0170]

[0171] in, . It is a feature follower radius used to control the decay rate. It is a positive hyperparameter. When the agent closely follows ( Approaching 0, When the value approaches 1, the agent is allowed to receive the full novelty reward; when the agent deviates significantly from its cooperative goal ( Much greater than 0 The value approaches 0), thereby suppressing the invalid novelty reward generated by deviation from the route.

[0172] Ultimately, the effective intrinsic reward obtained by the agent after gating regulation. Defined as the product of the original novel signal and the cooperative gating function:

[0173]

[0174] By introducing this cooperative gating mechanism, this invention effectively couples the agent's exploratory behavior with the cluster's macro-level cooperative goals. It incentivizes agents to explore while maintaining good cooperative relationships, thereby significantly improving the effectiveness of exploration, accelerating the learning of cooperative strategies, and avoiding ineffective exploratory behavior. To further enhance training stability, this final effective intrinsic reward... It will also undergo runtime statistical normalization before being used for policy updates.

[0175] 4. Reward Function Design

[0176] In reinforcement learning, the agent needs to interact with the environment in real time, receiving rewards or penalties in return to determine whether a step was good or bad, and learn from this experience so that the agent can make the best possible actions in the future to obtain the maximum reward. To effectively guide the agent in learning desired complex cooperative strategies, this invention designs goal-oriented, independent, multi-dimensional reward functions for agents at different levels (leader and follower). R1 ​​to R6 are reward functions shared by both the leader and follower drones, R7 and R8 are reward functions unique to the leader, and R9 is a reward function unique to the follower. (Leader's total reward...) Total reward for followers .in and It is a constant between 0 and 1. Furthermore, the leader's goal point remains fixed throughout the process, while the followers' goals change, determined by the consensus algorithm described above. The following are the specific reward functions for each item:

[0177] (1) Reaching reward item R1:

[0178] The reward consists of phased achievement rewards. and end mission rewards constitute. For the sum of a series of discrete reward terms, when the drone Location First crossing of the preset distance threshold Triggered at time: ,in This is an indicator function. Then, when the drone first enters the target radius During this time, a huge positive reward is given. .

[0179] (2) Proximity penalty term R2:

[0180] To ensure flight safety, when the distance between the drone and environmental obstacles or the ground is less than a preset safety threshold... At that time, a nonlinear repulsive field penalty is applied. For any obstacle... The penalty for approaching drones is as follows:

[0181] if

[0182] in This represents the horizontal distance from the drone to the obstacle surface. For collision avoidance in the ground direction, the penalty is:

[0183] if

[0184] in This refers to the drone's altitude above the ground. For positive numbers, penalty term The sum of the two penalties above is:

[0185]

[0186] (3) Collision penalty R3:

[0187] When a drone collides with an obstacle or the ground, that is or This results in a large negative reward:

[0188]

[0189] in, Choose a large positive number.

[0190] (4) Conflict-aware intelligent guidance item R4:

[0191] To address the "local optima" problem inherent in traditional potential field methods, this invention designs an intelligent guidance reward based on conflict perception. The core idea of ​​this reward is to use the current velocity vector of the drone... This is coupled with a dynamically calculated "safety guidance vector" capable of switching modes based on the situation. Alignment is performed in the direction between them, and the degree of alignment is quantified by the cosine similarity between them: The key innovation of this invention lies in the "safety guidance vector". The system employs a dynamic generation mechanism. At each time step, the system first performs conflict detection. Specifically, the system calculates the points pointing to the global target point. The attractive vector And the total repulsive force vector generated by all hazards in the environment (obstacles, ground, etc.). Then, the angle between these two key vectors was calculated. This allows for precise quantification of the degree of conflict faced in the current decision-making process. When the angle approaches 180 degrees, it indicates that the directions of attraction and repulsion are almost completely opposite, meaning the drone is in a deadlock state.

[0192]

[0193] Based on the results of the conflict perception described above, the guidance mechanism will dynamically switch between the two modes. When the calculated angle... Less than a preset conflict threshold (For example, at 135 degrees), the system determines that the decision conflict is not serious. At this time, the standard potential field guidance mode is adopted, and the guidance vector is directly formed by the superposition of attractive and repulsive forces. However, when the included angle When the value is greater than or equal to this threshold, the system determines that the drone has fallen into a "local optimum" and automatically switches to tangential guidance mode. In this mode, the system calculates a vector related to the total repulsive force vector. Orthogonal "tangent vectors" And select with attraction vector The included angle is acute. The final guiding vector is redefined as a weighted combination of the attraction and the tangent vector. This mode switching allows the reward function to proactively guide the agent to learn decisive and efficient lateral detour strategies, thereby effectively solving the problem of getting trapped in local optima.

[0194] (5) Distance Change Bonus Item R5:

[0195] Distance change reward is a type of reward that provides dense feedback based on the change in distance between the agent and the target point over consecutive time steps. ;in For the target location, The current time step is the location of the drone. This shows the drone's location at the previous time step. It is a positive number.

[0196] (6) Smoothing the reward item R6:

[0197] To encourage smooth control, the reward / penalty is applied to the current action. With recent average movement of step Deviation between: ;in , It is a positive number.

[0198] (7) Time-coordination rewards R7 and R8:

[0199]

[0200] in This indicates the time-coordination reward, representing whether leaders L0 and L1 arrive at the target point synchronously. This indicates that leaders are mutually exclusive about reward items. and It is a positive number.

[0201] (8) Penalty for falling behind (R9):

[0202] When the distance between the follower and its consistency target point Exceeding a preset "maximum following radius" At that time, apply a penalty term proportional to the distance:

[0203] if

[0204] in , positive numbers

[0205] 5. MAPPO Update Paradigm

[0206] The HCA-MAPPO algorithm proposed in this invention follows the basic paradigm of Multi-Agent Proximal Policy Optimization (MAPPO) in its underlying policy optimization and network update. After collecting sufficient interaction data, the algorithm learns the policy by optimizing two networks: the Actor and the Critic.

[0207] Critics Network Update: Shared Critics Network The accurate global state value function is learned by minimizing the temporal difference error. Its loss function... Using mean square error form: ;in, The target value is calculated using the Generalized Advantage Estimation (GAE) method.

[0208] Enforcer Network Update: Each Individual Enforcer Network By maximizing a pruned surrogate objective function This is used to update the strategy, aiming to improve sample utilization efficiency while ensuring update stability. The objective function is defined as: ;in, , is the probability ratio of the new strategy to the old strategy. It is the advantage function calculated by the critic network. This refers to the pruning hyperparameters. To encourage exploration, a regularization term for the policy entropy is usually added to the final loss function.

[0209] Step 3: Conduct simulation experiments using an algorithm written in Python.

[0210] 1. Experimental Environment Setup

[0211] In one specific embodiment of the present invention, the simulation experimental platform is built based on the Python 3.9 programming language and utilizes the open-source PyTorch deep learning framework to build and train all neural network models (including the executor network and the critic network) in the HCA-MAPPO algorithm. The simulation environment is constructed in accordance with the industry-standard Gymnasium interface, ensuring modular decoupling between the algorithm and the environment. To significantly improve training efficiency, the simulation process employs multi-process parallel computing, achieving parallel acquisition of large-scale data by running multiple independent simulation environment instances in parallel on multiple CPU cores.

[0212] 2. Training Process

[0213] The training process of this invention strictly follows a hierarchical and phased model. First, a leader training phase is conducted. In this phase, through course learning, the leader agent first learns basic strategies to reach the target point in an unobstructed environment, and then learns cooperative path planning and obstacle avoidance strategies in a complex environment with numerous static obstacles. Second, after the leader model converges, its optimal flight trajectory is exported as a data file. Finally, the follower training phase begins. In this phase, the simulation environment loads the previously generated leader trajectory, and 10 follower agents, driven by the HCA-MAPPO algorithm, learn how to autonomously form two sub-formations based on a set communication topology and closely and safely follow their respective leaders' trajectories. Throughout the training process, the TensorBoard tool is used to monitor key indicators such as the cluster's average round reward, task success rate, and collision rate in real time to determine the model's convergence status.

[0214] 3. The parameter model settings are shown in Table 1 below:

[0215] Table 1 Parameter Model Settings

[0216]

[0217] In the simulation experiment, it is first necessary to set the number of leader drones, the number of follower drones, the starting point and target point of the leader drones, and the number, position, and radius of obstacles. Initialize the drone parameters (speed range, heading angle variation range, climb angle variation range, time constants of speed, climb angle, and heading angle response, etc.).

[0218] This experiment used 2 leader drones and 10 follower drones. Each leader drone led 5 follower drones, forming two swarms with different starting points but the same target point. The 5 followers were evenly distributed within 250 meters of their respective leaders. The leader starting points were set at (120.008°E, 60°N, 800m) and (120°E, 60.005°N, 800m), with the target point at (120.05°E, 60°N, 1500m). The distance between the two starting points and the target point was approximately 3.5km. Table 2 below shows the drone parameters:

[0219] Table 2. Parameter information of UAVs

[0220]

[0221] Figure 6 This is a leader-follower path planning trajectory diagram. This invention designs 2 leader drones and 10 follower drones. Each leader drone leads 5 follower drones, forming two groups that start from different starting points (green spheres) and simultaneously reach the target convergence point (blue sphere). Throughout the swarm collaboration process, each group's follower drones follow their leader drone in real time, with a tracking range of 400 meters. If the distance to the leader exceeds 400 meters, the drone is considered to have fallen behind. The colored trajectories in the diagram represent the follower path planning trajectories; the four smaller diagrams represent different perspectives: the top left is a top-down view; the top right is a 3D view from the starting point; the bottom left is a side view trajectory diagram; and the bottom right is a view of one of the starting points.

[0222] Figure 7 This is a reward function graph. This invention uses a learning-by-course approach. In the graph, the left side of the dashed line represents the first stage of learning (no obstacles), and the right side represents the second stage (with obstacles). Two comparative ablation experiments were also designed. The red curve represents the reward function curve of this algorithm (HCA-MAPPO with an improved curiosity module), the blue curve represents the reward function curve using MAPPO and the improved intrinsic curiosity module (without the HCA framework), and the green curve represents the reward function curve using MAPPO (without the HCA framework and the improved intrinsic curiosity module).

[0223] Figure 8 This is a graph showing the distance changes between leader L0 and leader L1 from the target point. Figure 8 As can be seen, throughout the entire process, the two leaders were almost equidistant from the target point. Furthermore, their arrival times differed by 7 steps (equivalent to 0.7 seconds).

[0224] Figure 9These are distance variation curves between members within each team, with the top graph representing team one and the bottom graph representing team two. The invention sets the drone safety radius to 25 meters; when the distance between two drones exceeds 50 meters, a collision will not occur. As the curves show, the distances between members within each team are all greater than 50 meters. Clearly, no collisions occurred between the drones.

[0225] This invention proposes a deep reinforcement learning method, HCA-MAPPO, which integrates hierarchical cross-attention (HCA) and multi-agent proximal policy optimization (MAPPO). The core contributions of this method are: First, addressing the problem of processing multi-source heterogeneous information, an innovative HCA-MAPPO agent architecture is proposed, using a hierarchical cross-attention mechanism to achieve priority perception and processing of heterogeneous environmental information. Second, to improve the agent's exploration ability and prevent ineffective exploration due to unclear collaborative goals, an intrinsic curiosity module based on collaborative consistency gating is designed and applied, significantly improving the agent's effective learning efficiency in sparse reward environments. Third, addressing the "local optimum" problem in path planning, a conflict-aware "tangential guidance" hybrid reward shaping method is proposed, enabling UAVs to dynamically switch guidance modes and make decisive and efficient detour decisions in critical obstacle avoidance scenarios. This invention systematically solves key technical problems in multi-UAV swarm path planning from three aspects: hierarchical cross-attention mechanism, collaborative consistency gating curiosity module, and conflict-aware tangential guidance reward function setting, demonstrating high practical value and technological innovation.

[0226] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A deep reinforcement learning-based multi-UAV cooperative path planning method based on HCA-MAPPO, characterized in that, Includes the following steps: 1) Starting from the multi-UAV swarm collaborative path planning method, establish a simulation environment that includes a three-degree-of-freedom UAV kinematic model and a dual-coordinate system management system; 2) Design a hierarchical cross-attention (HCA) mechanism to introduce an actor network to achieve priority perception and processing of multi-source heterogeneous information; 3) A decentralized collaborative control mechanism based on a weighted consensus algorithm calculates the dynamic consistency of each follower's target point in real time, including target position, velocity, and acceleration, as observation input and reward benchmark; 4) Construct an intrinsic curiosity module based on cooperative consistency gating to guide the agent to explore effectively. Generate original novelty signals by distilling RND through random networks and adjust the intrinsic reward intensity through a Gaussian cooperative gating function. 5) Design a tangential-guided hybrid reward function based on conflict perception. The conflict state is determined by calculating the angle between the attractive and repulsive force vectors. When the angle is greater than or equal to a preset threshold, the function switches to tangential-guided mode. 6) The MAPPO paradigm update strategy is adopted. The critic network learns the global value function by minimizing the temporal difference mean square error; the executor network updates by maximizing the pruning agent objective function and adding an entropy regularization term; a phased training mode of leader training first and follower training later is adopted, and the performance is verified based on a multi-process parallel simulation platform.

2. The deep reinforcement learning multi-UAV cooperative path planning method of HCA-MAPPO according to claim 1, characterized in that, In step 1), a simulation environment is established that includes a three-degree-of-freedom UAV kinematic model and a dual-coordinate system management system. First, a first-order inertial system is used to implement discrete state updates. The UAV's state at discrete time steps is described by the following first-order inertial system. The actual state change process within: in, , , These are the time constants for the speed, climb angle, and heading angle responses, respectively; for a periodic heading angle... The update uses normalization processing; the angle error used for updating... Through standardized operations, mapped to Within the interval, obtain the shortest angle difference between the two; in the updated interval... After determining the kinematic state at each moment, these states are decomposed into velocity vectors of the UAV in three-dimensional space through triangular projection in a standard aerospace coordinate system. ,in: The position of the UAV at the next moment can be calculated by performing Euler forward integration on the velocity vector. This completes one iteration of the simulation step. The dual-coordinate system management system uses a preset reference origin in the geodetic coordinate system. The system uses a three-dimensional Cartesian coordinate system with the origin at , where the X-axis points due east, the Y-axis points due north, and the Z-axis is perpendicular to the local ground plane and points upwards. At the beginning of each simulation round, the system performs a planar approximation transformation based on the WGS-84 Earth model to adjust the geodetic coordinates of any entity. Convert to local northeast celestial coordinates Convert all angle units to radians and perform the following calculations: in, Here, represents the average radius of the Earth in the WGS-84 model. Throughout the simulation, all state updates and calculations are performed within this Euclidean space. When the simulation ends or visualization is required, the system then accurately reconstructs the recorded local ENU coordinate trajectory into geodetic coordinates through the following inverse transformation. : This dual-coordinate system management method ensures the computational efficiency and stability of the simulation process.

3. The deep reinforcement learning multi-UAV cooperative path planning method of HCA-MAPPO according to claim 1, characterized in that, In step 2), the hierarchical cross-attention (HCA) mechanism is introduced into the agent network, and its core is the hierarchical cross-attention mechanism; to cooperate with this mechanism, the agent's original observation information is first processed in a structured manner. Single follower agent Observations Divided into four parts: its own state Leader entity status Physical state of environmental obstacles and the status of other follower entities ;in Represents the number of corresponding entities. represent Its feature dimensions; The HCA enforcer network performs prioritized data preprocessing on the aforementioned structured information through a sequential three-layer cross-attention mechanism. First, it maps various types of raw state information to the high-dimensional feature space of the attention mechanism through independent linear embedding layers with non-linear activation functions. in, and These are the learnable weight matrix and bias vector for the corresponding embedding layer. This serves as the initial query vector; subsequently, the network computes according to the following hierarchical logic: The first layer is the intent-aware layer: the agent observes the leader's state to clarify the task intent; it utilizes the initial query vector. For the leader entity Perform multi-head cross-attention calculation to generate leader context vector. This calculation is performed by It consists of several parallel attention heads, each head Independently compute a weighted context; for the first Size, its attention weight The calculation is as follows: in, Is the query vector and the first Scaling dot product similarity score among leader entities: express The first of the matrix Line; based on this weight, the first The output for each head is: in, It is the first The learnable projection matrix corresponds to each head; the outputs of all heads are concatenated and then passed through a linear projection to obtain the final context vector: The second layer is the threat perception layer: after clarifying the task intent, the agent then focuses on environmental threats; the leader context vector generated in the first layer is then used... With the initial query vector The vectors are concatenated and an enhanced query vector is generated through an update module. : This enhanced query vector For environmental obstacle entities Perform a second multi-head cross-attention calculation to generate obstacle context vectors. Its calculation process is exactly the same as the first layer, only the input is replaced with... ; The third layer is the collaborative perception layer: after fully perceiving the task intent and environmental threats, the agent ultimately focuses on the collaborative needs within the cluster; furthermore, the obstacle context vector generated in the second layer is used... Enhanced query vectors from the preceding sequence The vectors are then concatenated and generated through a separate update module to produce the final query vector. : The final query vector For other follower entities Perform a third multi-head cross-attention calculation to generate the cluster context vector. Its calculation process is exactly the same as the first layer, only the input is replaced with... ; Final decision: The network embeds the initial query vector Leader context vector Obstacle context vector and cluster context vector The final feature fusion is performed; this fusion is achieved by concatenating the four vectors: The fused high-dimensional feature vector Input to a policy network In the end, the control actions of the drone are output. .

4. The deep reinforcement learning multi-UAV cooperative path planning method of HCA-MAPPO according to claim 1, characterized in that, In step 3), the core driving force of the decentralized collaborative control mechanism based on the weighted consensus algorithm originates from a configurable communication topology adjacency matrix. ,in This represents the total number of drones in the cluster, including the leader and followers; the matrix defines the information interaction topology within the cluster, where the matrix elements... Indicates drone Capable of sensing drones The state information is 0 if it is true and 0 otherwise; through the asymmetric design of this matrix, complex command and coordination relationships are constructed. At each time step For any follower agent in the cluster The system first uses the matrix Determine the set of all its neighbors. Subsequently, a weighted consensus algorithm is used to calculate the agent's... The complete and consistent target state at the current moment, including the target location. Target speed Even target acceleration This calculation is a weighted average of its own state and the states of all its neighbors: in, and These represent the position and velocity vectors, respectively. It is an intelligent agent Its own weighting coefficient, and It is with the neighbor Type-dependent, configurable weighting coefficients; This is the total weight, used for normalization; furthermore, to enable the agent to better predict the target's dynamics, the target's acceleration is estimated by performing a backward difference on the target's velocity over time. This dynamically calculated consistency target state As part of the structured observation information, it is input into the HCA agent network, enabling its decision-making process to fully perceive its dynamic and personalized task objectives. At the same time, it becomes the core benchmark for reward function design, with the agent's behavior revolving around how to track this target point most efficiently and safely for rewards and penalties. Finally, at the cluster level, through each agent continuously pursuing its own dynamically changing consistent objectives, the entire cluster provides the expected, flexible, and adaptive cooperative formation and path planning behavior in a completely decentralized manner.

5. The deep reinforcement learning multi-UAV cooperative path planning method of HCA-MAPPO according to claim 1, characterized in that, In step 4), the specific steps for constructing the intrinsic curiosity module based on cooperative consistency gating, generating the original novelty signal through random network distillation of RND, and adjusting the intrinsic reward intensity through a Gaussian cooperative gating function are as follows: Through a fixed, random target network and a learnable prediction network The novelty of the quantification state between prediction errors; for agents New situation encountered Its original novelty signal It is defined as the square of the Euclidean distance between the output vectors of two networks: in, These are the learnable parameters of the prediction network; A collaborative consensus gating mechanism is proposed to regulate and filter the original novelty information. This mechanism applies to each follower agent. Introduce a cooperative gating function The value of this function depends on the degree of consistency between the agent's current behavior and the cluster's collaborative goal, which is determined by the agent's current position. Its dynamic "consistent movement of target points" Euclidean distance between To quantize; gate function Designed to be distance-dependent A function that increases and monotonically decreases, specifically in the form of a Gaussian function: in, ; It is a feature follower radius used to control the decay rate. It is a positive hyperparameter; when the agent closely follows, Approaching 0, When the value approaches 1, the agent is allowed to receive the full novelty reward; when the agent deviates significantly from its cooperative goal... Much greater than 0 The value approaches 0, thus suppressing the invalid novelty reward generated by deviation from the route; The effective intrinsic reward obtained by the agent after gating regulation Defined as the product of the original novel signal and the cooperative gating function: By introducing a collaborative gating mechanism, the exploratory behavior of agents is effectively coupled with the macro-level collaborative goals of the cluster; this ultimately provides an effective intrinsic reward. The statistics are normalized at runtime before being used for policy updates.

6. The deep reinforcement learning multi-UAV cooperative path planning method of HCA-MAPPO according to claim 1, characterized in that, In step 5), the conflict-aware tangent-guided hybrid reward function includes: The reward functions are as follows: arrival reward R1, proximity penalty R2, collision penalty R3, conflict-aware intelligent guidance R4, distance change reward R5, smoothing reward R6, time coordination reward R7, leader-to-leader exclusion reward R8, and falling behind penalty R9. R1 to R6 are reward functions shared by both leader and follower drones, R7 and R8 are reward functions unique to the leader, and R9 is a reward function unique to the follower. The total reward for the leader is... Total reward for followers ;in and It is a constant between 0 and 1; (1) Reaching reward item R1 Reaching reward item R1 is achieved through phased achievement rewards. and end mission rewards constitute; For the sum of a series of discrete reward terms, when the drone Location First crossing of the preset distance threshold Triggered at time: ,in For indicator functions; Then, when the drone first enters the target radius During this time, a huge positive reward is given. ; (2) Proximity penalty term R2: When the distance between the drone and environmental obstacles or the ground is less than a preset safety threshold At that time, a nonlinear repulsive field penalty is applied; for any obstacle The penalty for approaching drones is as follows: ,if in The horizontal distance from the drone to the obstacle surface; for collision avoidance in the ground direction, the penalty is: ,if in This refers to the drone's altitude above the ground. For positive numbers, penalty term The sum of the two penalties above is: (3) Collision penalty R3: When a drone collides with an obstacle or the ground, that is or This results in a large negative reward: in, Take a large positive number; (4) Conflict-aware intelligent guidance item R4: The drone's current velocity vector This is coupled with a dynamically calculated "safety guidance vector" capable of switching modes based on the situation. Alignment is performed in the direction between them, and the degree of alignment is quantified by the cosine similarity between them: "Security Boot Vector" The dynamic generation mechanism involves the system first performing conflict detection at each time step; specifically, the system calculates the points pointing to the global target point. The attractive vector and the total repulsive force vector generated by all hazards in the environment. By calculating the angle between these two key vectors This allows for precise quantification of the degree of conflict faced in the current decision-making process; when the angle approaches 180 degrees, it indicates that the directions of attraction and repulsion are almost completely opposite, meaning that the drone has fallen into a deadlock state. Based on the results of the aforementioned conflict perception, the guidance mechanism will dynamically switch between the two modes; when the calculated angle... Less than a preset conflict threshold When the system determines that the decision-making conflict is not severe, the standard potential field guidance mode is adopted, and the guidance vector is directly formed by the superposition of attractive and repulsive forces. When the included angle When the value is greater than or equal to this threshold, the system determines that the drone has fallen into a "local optimum" and automatically switches to tangential guidance mode; in this mode, the system will calculate a vector related to the total repulsive force vector. Orthogonal "tangent vectors" And select with attraction vector The included angle is acute; the final guiding vector is redefined as a weighted combination of the attraction and the tangent vector. ; (5) Distance Change Bonus Item R5: Distance change reward is a type of reward that provides dense feedback based on the change in distance between the agent and the target point over consecutive time steps. ,in For the target location, The current time step is the location of the drone. The location of the drone at the previous time step; It is a positive number; (6) Smoothing the reward item R6: To encourage smooth control, the reward / penalty is applied to the current action. With recent average movement of step Deviation between: ,in , It is a positive number; (7) Time-based synergy reward R7 and inter-leader exclusion reward R8: in This indicates the time-coordination reward, representing whether leaders L0 and L1 arrive at the target point synchronously; This indicates that leaders exclude certain reward items; and It is a positive number; (8) Penalty for falling behind (R9): When the distance between the follower and its consistency target point Exceeding a preset "maximum following radius" At that time, apply a penalty term proportional to the distance: , if in , It is a positive number.

7. The deep reinforcement learning multi-UAV cooperative path planning method of HCA-MAPPO according to claim 1, characterized in that, In step 6), the MAPPO paradigm update strategy is adopted: the critic network learns the global value function by minimizing the temporal difference mean square error; the actor network updates the strategy by maximizing the pruning agent objective function and adding an entropy regularization term; its underlying policy optimization and network update follow the basic paradigm of multi-agent proximal policy optimization MAPPO; after collecting enough interaction data, the algorithm learns the strategy by optimizing the two networks, the actor and the critic. Critics Network Update: Shared Critics Network The accurate global state value function is learned by minimizing the temporal difference error; its loss function is... Using mean square error form: in, The target value is calculated using the generalized advantage estimation method; Enforcer Network Update: Each Individual Enforcer Network By maximizing a pruned surrogate objective function To perform policy updates, the objective function is defined as follows: in, , is the probability ratio of the new strategy to the old strategy. It is the advantage function calculated by the critic network. This involves pruning hyperparameters; to encourage exploration, a regularization term for policy entropy is usually added to the final loss function.

Images