Multi-agent reinforcement learning based method for autonomous cooperative obstacle avoidance of UAV swarm

Abstract

The unmanned aerial vehicle group autonomous cooperative obstacle avoidance method based on multi-agent reinforcement learning belongs to the technical field of unmanned aerial vehicle intelligent control. In order to solve the problem of unmanned aerial vehicle group autonomous safe cooperative obstacle avoidance, the present application comprises the following steps: constructing a three-dimensional extended observation vector of the unmanned aerial vehicle; constructing a MAPPO+GAT-GNN strategy network with distance bias attention, including a MLP encoder module, a distance bias graph attention network module, an Actor strategy head module, a residual action fusion module and a centralized Critic value head module connected in sequence, processing the 108-dimensional three-dimensional extended observation vector of the unmanned aerial vehicle obtained in step S1, and outputting enhanced features; designing a hierarchical multi-objective reward function; using an APF auxiliary residual strategy curriculum to train the strategy network, performing MAPPO strategy optimization, updating the strategy network parameters by using a PPO truncation objective function, and then repeatedly iterating the training until the set number of training times is reached.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) intelligent control technology, specifically relating to a UAV swarm autonomous cooperative obstacle avoidance method based on multi-agent reinforcement learning. Background Technology

[0002] With the rapid development of the low-altitude economy, the application of UAV swarms is becoming increasingly widespread. When multiple UAVs simultaneously perform tasks in three-dimensional space, achieving safe and efficient autonomous cooperative obstacle avoidance in environments with numerous static obstacles is a core technical challenge in the field of low-altitude intelligent agents. Existing methods can be categorized into five types: traditional rule-based methods (APF, ORCA, etc.), multi-agent reinforcement learning based on value function decomposition (QMIX, VDN, etc.), methods based on multi-agent deterministic policy gradients (MADDPG, etc.), methods based on graph neural networks (GNN+MARL), and methods using only basic PPO / MAPPO. Each type of method has certain limitations and cannot simultaneously meet the comprehensive requirements of large-scale swarms, complex three-dimensional obstacle environments, high success rates, and low collision rates. Summary of the Invention

[0003] The problem this invention aims to solve is to achieve autonomous, safe, and collaborative obstacle avoidance for a swarm of unmanned aerial vehicles (UAVs). It proposes a method for autonomous, collaborative obstacle avoidance for UAV swarms based on multi-agent reinforcement learning.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A method for autonomous cooperative obstacle avoidance of unmanned aerial vehicle (UAV) swarms based on multi-agent reinforcement learning includes the following steps:

[0006] S1. Construct the UAV's three-dimensional extended observation vector, including its own state vector, neighbor feature vector, obstacle feature vector, and extended safety perception feature vector;

[0007] S2. Construct a MAPPO+GAT-GNN policy network with distance bias attention, including an MLP encoder module, a distance bias map attention network module, an Actor policy head module, a residual action fusion module, and a centralized Critic value head module connected in sequence. Process the 108-dimensional extended observation vector of the UAV obtained in step S1 and output enhanced features.

[0008] S3. Design a hierarchical multi-objective reward function;

[0009] S4. Based on the hierarchical multi-objective reward function obtained in step S3, the MAPPO+GAT-GNN policy network with distance-biased attention is trained using the APF-assisted residual policy course. The training data comes from online interactive simulation data.

[0010] After completing one training round, perform MAPPO policy optimization, using PPO to truncate the objective function to update the policy network parameters, and then repeat the iterative training until the set number of training rounds is reached.

[0011] Furthermore, the specific implementation method of step S1 includes the following steps:

[0012] S1.1. Construct a 10-dimensional self-state vector, a 56-dimensional neighbor feature vector, and a 30-dimensional obstacle feature vector;

[0013] S1.2. Construct a 12-dimensional extended safety awareness feature vector, including a 3-dimensional unit vector of the nearest obstacle direction, a 3-dimensional unit vector of the target direction, a 1-dimensional normalized distance of the nearest obstacle, a 1-dimensional normalized distance of the nearest neighbor, a 1-dimensional current speed, a 1-dimensional target distance emphasis, a 1-dimensional step progress, and a 1-dimensional team arrival ratio.

[0014] S1.3. Combining steps S1.1 and S1.2, obtain the three-dimensional extended observation vector of UAV i. , It is a real matrix.

[0015] Furthermore, the specific implementation method of step S2 includes the following steps:

[0016] S2.1. Construct an MLP encoder module consisting of three linear layers connected in series. Each linear layer is followed by a normalized LN and a linear rectified activation function ReLU, which maps the three-dimensional extended observation vector oi of UAV i to the hidden state feature vector. ;

[0017] S2.2. Construct a distance-biased graph attention network module. N drones are abstracted as N agents. An interaction graph is constructed from the hidden state feature vectors {h1, h2, ..., hN} of the N agents, with each agent acting as a node in the graph. Neighborhood collaboration is modeled through two layers of graph attention updates. A distance bias term is introduced on top of the standard multi-head attention mechanism. After obtaining the attention coefficients, weighted aggregation and residual connections are used to obtain the output of the m-th layer. After two layers of GAT updates, the final output is an enhanced feature vector. ;

[0018] S2.3. Construct the Actor policy head module to enhance feature vectors. The motion is mapped to a 3D normalized acceleration motion using a three-layer MLP. The first and second layers of the three-layer MLP are ReLU functions, and the third layer is a hyperbolic tangent function, ultimately outputting the RL motion. ;

[0019] S2.4. Construct a residual action fusion module to linearly mix the RL action and the APF-guided action according to the APF weight to form the actual execution action. The expression is:

[0020]

[0021] where, is the final execution action vector of UAV i, is the element-wise truncation function, is the APF weight coefficient, is the three-dimensional guided action calculated in real time by the artificial potential field method based on the current state, which is obtained by normalizing the comprehensive target gravitational force and obstacle repulsive force;

[0022] S2.5. Construct a centralized Critic value head module. After concatenating the enhanced feature vector hi' and the global feature output the scalar value estimate Vi of the current state of UAV i through a three-layer MLP. The centralized Critic value head module calculates Vi during the training phase for advantage function estimation and does not participate in decision-making during the execution phase.

[0023] Furthermore, the specific implementation method of step S3 includes the following steps:

[0024] S3.1. Design the arrival target reward. When UAV i meets the arrival condition ‖Δgi(t)‖≤ragent at time t, give the arrival target reward, , where Δgi(t) is the Euclidean distance between UAV i and the target point gi at time t, and ragent is the collision radius of the UAV, is the arrival target reward of UAV i, which is a one-time trigger reward;

[0025] S3.2. Design the progress reward. The expression is:

[0026]

[0027] where, is the progress reward of UAV i at time step t, is the progress weight coefficient, with a value of 1.5; ‖Δgi(t - 1)‖ is the Euclidean distance between UAV i and the target point at the previous time step t - 1; is the Euclidean distance between UAV i and the target point at time step t;

[0028] S3.3. Design the approaching target exponential acceleration reward, which is triggered only when ‖Δgi(t)‖<dprox, where dprox is the trigger distance threshold. The expression is:

[0029]

[0030] Among them, is the proximity-to-goal acceleration reward of the UAV i at time step t, is the decay scale parameter;

[0031] S3.4. Design obstacle hierarchical penalties, and design three-level hierarchical penalties for the nearest obstacle. The three levels include a danger zone, a warning zone, and a pre-warning zone;

[0032] S3.5. Design soft constraint hierarchical penalties between agents, and design three-level hierarchical penalties for the distance between the UAV i and its nearest neighbor j. The three levels include a collision zone, a warning zone, and a pre-warning zone;

[0033] S3.6. Design a collaborative team reward, and the expression is:

[0034]

[0035] Among them, is the collaborative team reward of the UAV i at time step t, is the collaborative weight coefficient, is the total number of UAVs that have successfully reached their respective target points in the cluster at time t, and N is the total number of UAVs in the cluster;

[0036] S3.7. Design an orientation alignment reward, and the expression is:

[0037]

[0038] Among them, is the orientation alignment reward of the UAV i at time step t, is the orientation weight coefficient, is the angle between the current velocity direction of the UAV i and the target direction;

[0039] S3.8. Construct a boundary soft constraint penalty, which is triggered when the minimum distance d_bound,i(t) of the UAV i from any spatial boundary < d_safe, where d_safe is the boundary safety distance threshold, and the expression is:

[0040]

[0041] Among them, is the boundary penalty of the UAV i at time step t, is the boundary penalty weight coefficient;

[0042] S3.9. Design a hierarchical multi-objective reward function is the linear superposition of the eight components in steps S3.1 and S3.8.

[0043] Furthermore, the specific implementation method of step S4 includes the following steps:

[0044] S4.1. Based on online interactive simulation to collect training data, in each training round, N=20 drones are set to interact in real time with 12 random spherical obstacles in a 3D space of 30×30×15 m. In each step, the oi and ai of drone i are generated. exec R i Vi, the quintuple trajectory of the logarithmic probability of actions logπ(aiexec|oi);

[0045] S4.2. After collecting the trajectory of a complete round, calculate the advantage function using generalized advantage estimation, and adopt the PPO truncation objective function update strategy:

[0046] L_CLIP(θ) = E[min(ρt(θ)·Ât, clip(ρt(θ), 1-ε, 1+ε)·Ât)]

[0047] Where L_CLIP(θ) is the objective function of the PPO pruning strategy, E is the expectation of the mini-batch sample, ρt(θ) is the probability ratio of the new and old strategies, Ât is the advantage estimate after mini-batch normalization, clip is the cutoff function, and ε is the PPO pruning hyperparameter.

[0048] In each policy update cycle, the Actor policy network is updated once, and the centralized Critic value network is updated six times. The gradient norm pruning method is used to suppress parameter update fluctuations, and the maximum gradient norm is set to 0.5.

[0049] The training consisted of 500 simulation rounds, with a maximum number of steps per round set to Tmax = 400 steps, and a 30-second timeout protection was set.

[0050] The beneficial effects of this invention are:

[0051] The present invention describes an autonomous cooperative obstacle avoidance method for UAV swarms based on multi-agent reinforcement learning. To address the shortcomings of existing GNN-MARL methods that do not utilize edge features and whose attention weights cannot distinguish between nearest and far neighbors, the invention proposes for the first time to use the normalized distance between agents, ‖pi-pj‖ / robs, as the negative bias of GAT attention. This allows nearby UAVs to automatically obtain higher attention weights without the need for an additional edge feature encoding network.

[0052] The present invention describes an autonomous cooperative obstacle avoidance method for UAV swarms based on multi-agent reinforcement learning. It designs a 108-dimensional observation space that includes 12-dimensional safety extended features such as obstacle direction unit vector dobs,1, target direction unit vector dg,i, step progress t / Tmax, and team arrival ratio nr / N. This method solves the problems of weak gradients, lack of time awareness, and lack of cooperative awareness when the target is far away, and improves the success rate from ~0% (96 dimensions) to 89.5%.

[0053] The present invention discloses a method for autonomous cooperative obstacle avoidance of UAV swarms based on multi-agent reinforcement learning, which includes potential-based progress reward (weight γs=1.5), exponential approach reward (0.5×exp(-d / 4), triggered when d<8 m), three-level graded penalty for obstacles up to 4 m away, and cooperative team reward of 0.05×nr / N, systematically solving the problem of convergence difficulty of sparse rewards.

[0054] The present invention describes an autonomous cooperative obstacle avoidance method for UAV swarms based on multi-agent reinforcement learning. The APF weight hw linearly decays from 1.0 to 0.75 and then remains fixed. The PPO loss uses log π(aexec|o) to ensure gradient alignment. Experiments show that 75% is the critical optimal threshold (50% minimum only achieves 20.0%, and 0% minimum completely collapses to 0.0%), which increases the success rate from 0% to 89.5%.

[0055] This invention presents a method for autonomous cooperative obstacle avoidance of UAV swarms based on multi-agent reinforcement learning. It overcomes the scalability degradation problem of QMIX / VDN value function decomposition when N>10, achieving stable training at a scale of N=20 through a parameter-sharing MAPPO+GAT structure. It addresses the problem of dimensionality explosion in the centralized Critic input of MADDPG by using global mean pooling CTDECritic, where the Critic input dimension is independent of N. It fills the gap in edge feature utilization in existing GNN-MARL methods by using the inter-agent distance matrix as an attention bias for the first time, achieving adaptive information aggregation based on geometric perception. It solves the convergence failure problem of basic MAPPO in dense 3D scenes with 12 obstacles due to sparse rewards by achieving effective convergence through a 108-dimensional observation design and a hierarchical multi-objective reward function. It overcomes the performance collapse problem caused by the complete withdrawal of APF during course training in existing methods by proposing a residual policy training framework that guarantees at least 75% APF. Attached Figure Description

[0056] Figure 1 This is a flowchart of a method for autonomous cooperative obstacle avoidance of a drone swarm based on multi-agent reinforcement learning, as described in this invention.

[0057] Figure 2This is a schematic diagram of the three-dimensional extended observation structure of the UAV of the present invention;

[0058] Figure 3 This is a schematic diagram of the structure of the MAPPO+GAT-GNN policy network with distance-biased attention according to the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0060] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0061] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 -Appendix Figure 3 Detailed explanation is as follows:

[0062] Example 1:

[0063] A method for autonomous cooperative obstacle avoidance of unmanned aerial vehicle (UAV) swarms based on multi-agent reinforcement learning includes the following steps:

[0064] S1. Construct the UAV's three-dimensional extended observation vector, including its own state vector, neighbor feature vector, obstacle feature vector, and extended safety perception feature vector;

[0065] Furthermore, the specific implementation method of step S1 includes the following steps:

[0066] S1.1. Construct a 10-dimensional self-state vector, a 56-dimensional neighbor feature vector, and a 30-dimensional obstacle feature vector;

[0067] Furthermore, the self-state vector (10-dimensional, oi[0:10]): contains the normalized position of UAV i. (Each axis divided by the spatial dimension W), normalized velocity (Divided by the maximum velocity vmax = 4.0 m / s), the target's relative displacement (gi is the target position of UAV i), and the Euclidean distance to the target. .

[0068] Neighbor feature vector (56-dimensional, oi[10:66]): K nearest neighbors (sorted by distance) within a receptive radius robs=12 m, each neighbor j has a 7-dimensional feature vector: relative position relative speed The survival state is alivej∈{0,1}. Zero-filling occurs when there are fewer than K neighbors. Here, pj and vj represent the position and velocity of neighbor j, respectively (j≠i).

[0069] Obstacle feature vector (30-dimensional, oi[66:96]): The feature vector of the nearest L=6 obstacles within the perception range (sorted by surface distance), each obstacle l has a 5-dimensional feature vector: relative center position (cl is the coordinate of the center of the sphere of obstacle l), radius of obstacle Surface distance .

[0070] S1.2. Construct a 12-dimensional extended safety awareness feature vector, including a 3-dimensional unit vector of the nearest obstacle direction, a 3-dimensional unit vector of the target direction, a 1-dimensional normalized distance of the nearest obstacle, a 1-dimensional normalized distance of the nearest neighbor, a 1-dimensional current speed, a 1-dimensional target distance emphasis, a 1-dimensional step progress, and a 1-dimensional team arrival ratio.

[0071] S1.3. Combining steps S1.1 and S1.2, obtain the three-dimensional extended observation vector of UAV i. , It is a real matrix.

[0072] S2. Construct a MAPPO+GAT-GNN policy network with distance bias attention, including an MLP encoder module, a distance bias map attention network module, an Actor policy head module, a residual action fusion module, and a centralized Critic value head module connected in sequence. Process the 108-dimensional extended observation vector of the UAV obtained in step S1 and output enhanced features.

[0073] A multi-agent collaborative decision-making network based on the centralized training-distributed execution (CTDE) paradigm, MAPPO+GAT-GNN, is constructed. All N=20 UAVs share the same set of network parameters (parameter sharing mechanism). During the execution phase, each UAV makes independent decisions based solely on its own 108-dimensional local observations, without the need for real-time communication.

[0074] Furthermore, the specific implementation method of step S2 includes the following steps:

[0075] S2.1. Construct an MLP encoder module consisting of three linear layers connected in series. Each linear layer is followed by a normalized LN and a linear rectified activation function ReLU, which maps the three-dimensional extended observation vector oi of UAV i to the hidden state feature vector. ;

[0076] Furthermore, the complete calculation process is as follows:

[0077] First layer (input layer → hidden layer):

[0078] ;

[0079] Second layer (hidden layer → hidden layer):

[0080] ;

[0081] Third layer (hidden layer → output):

[0082] ;

[0083] in, is the encoded 128-dimensional hidden state feature vector of UAV i; Let be the 108-dimensional local observation vector of UAV i; These are the feature vectors of the intermediate layer; The first layer weight matrix maps the 108-dimensional observations to the 128-dimensional latent space; These are the weight matrices for the second and third layers, respectively. These are the bias vectors for each layer; LN(·) is the layer normalization function, which normalizes the output of each layer to a mean of 0 and a variance of 1 to stabilize the training process; ReLU(·)=max(0, ·) is the linear rectified activation function. All weight matrices in the three-layer structure are orthogonally initialized (gain is √2), and the bias vectors are initialized to zero.

[0084] S2.2. Construct a distance-biased graph attention network module. N drones are abstracted as N agents. An interaction graph is constructed from the hidden state feature vectors {h1, h2, ..., hN} of the N agents, with each agent acting as a node in the graph. Neighborhood collaboration is modeled through two layers of graph attention updates. A distance bias term is introduced on top of the standard multi-head attention mechanism. After obtaining the attention coefficients, weighted aggregation and residual connections are used to obtain the output of the m-th layer. After two layers of GAT updates, the final output is an enhanced feature vector. ;

[0085] Furthermore, a distance bias term is introduced based on the standard multi-head attention mechanism, automatically assigning higher attention weights to spatially closer neighbors. The calculation of the attention coefficients in the m-th layer (m=1,2) involves two steps:

[0086] S2.2.1. Calculation of raw attention score:

[0087]

[0088] in, The original attention score (scalar) before introducing distance bias; The transpose of the learnable attention vector at layer m is used to calculate the attention score of the concatenated features of the two nodes. Let m be the linear transformation weight matrix of the m-th layer; Let be the output feature of agent i at layer m-1 (the input of layer 1 is hi, i.e., the output of the MLP encoder). Let be the output feature of neighbor j at layer m-1; ‖ is the vector concatenation operation; It is a linear rectifier function with leakage, and the slope in the negative region is 0.2;

[0089] S2.2.2. Introduce distance bias and normalize:

[0090]

[0091] in, Let be the normalized attention coefficient of agent i to neighbor j in the m-th layer; It is a learnable distance bias scalar that controls the strength of the influence of distance on attention weights; Let be the three-dimensional Euclidean distance between agent i and neighbor j. , respectively, are the three-dimensional position coordinates of agent i and neighbor j; The sensing radius is used to normalize the distance; To perform Softmax normalization on all neighbors j, ensuring that the sum of attention coefficients is 1, the distance bias term -β·‖pi-pj‖ / robs is negative (when β>0), so that the closer the neighbors are, the higher the score, thus achieving adaptive information aggregation of geometric perception;

[0092] After obtaining the attention coefficients, the output of the m-th layer is obtained through weighted aggregation and residual connection:

[0093]

[0094] in, Let be the output feature vector of agent i after the GAT update at layer m; K=8, which is the number of multi-head attention heads; Let be the attention coefficient of agent i to neighbor j in the k-th attention head of the m-th layer; It is the linear transformation matrix of the m-th layer and k-th head (mapping 128-dimensional features to 16-dimensional features per head). To concatenate the outputs of K=8 heads (resulting in an 8×16=128 dimensional vector); , is the linear mapping matrix after multi-head splicing; As a residual connection term, the input of this layer is directly added to the output, alleviating the gradient vanishing problem in deep networks. After two layers of GAT updates, the final output enhances the features. It simultaneously includes individual local observation information and neighborhood collaborative interaction information;

[0095] S2.3. Construct the Actor policy head module to enhance feature vectors. The Actor policy head module is mapped to a 3D normalized acceleration action using a three-layer MLP. The network structure is 128→64→64→3. The first and second layers of the three-layer MLP use ReLU functions, and the third layer uses a hyperbolic tangent function. The final output is the RL action. ;

[0096] Furthermore, the first layer of the three-layer MLP (128 dimensions → 64 dimensions):

[0097] ;

[0098] Second layer (64 dimensions → 64 dimensions):

[0099] ;

[0100] Third layer (64-dimensional → 3-dimensional, output layer):

[0101] ;

[0102] in, ∈[-1,1]³, is the reinforcement learning policy action of UAV i, which is a normalized three-dimensional acceleration vector (corresponding to the x, y, and z axes respectively). , is the feature vector of the intermediate layer; , is the first-level weight matrix; , is the second-layer weight matrix; , is the output layer weight matrix; , , are the bias vectors for each layer; Tanh(·) is the hyperbolic tangent activation function, which compresses the output value to the interval [-1,1] to ensure the action meets physical constraints; in addition, the Actor head also has a shared logarithmic standard deviation parameter vector. (and the mean of movement) (Independent), training is done via ai, sample= +σ⊙ε (ε ~ ​​N(0, I), ⊙ is the element-wise product) implements random exploration, and the mean is used directly during evaluation. As a deterministic action. Output layer weight matrix. Orthogonal initialization (gain of 0.01) is used to make the initial policy output close to zero, avoiding physical instability caused by excessively large motion amplitude in the early stage of training.

[0103] S2.4. Construct a residual motion fusion module to fuse RL actions. The APF-guided action is linearly mixed with the APF-weighted action to form the actual action to be executed, expressed as:

[0104]

[0105] in, Let be the final action vector of drone i. This is an element-wise truncation function. These are the APF weighting coefficients. The three-dimensional guiding action is calculated in real time based on the current state using the artificial potential field method, and is obtained by normalizing the combined target attraction and obstacle repulsion.

[0106] Furthermore, key implementation details of the residual action fusion module: UAVs that have reached the target (‖Δgi‖≤ragent) are excluded from the APF repulsion calculation to prevent them from interfering with UAVs still navigating. During PPO training, the policy gradient is calculated based on log π(aiexec|oi) instead of log π(aiRL|oi) to ensure that the gradient direction is consistent with the actual executed action and the resulting reward signal.

[0107] S2.5. Construct a centralized Critic value head module, which integrates the enhanced feature vector hi' with global features. After concatenation, the scalar value estimate Vi of the current state of UAV i is output through a three-layer MLP. The network structure is 256→256→128→1. The centralized Critic value head module calculates Vi for advantage function estimation during the training phase and does not participate in decision-making during the execution phase.

[0108] Furthermore, the Critic value head module adopts the centralized value evaluation mechanism in the CTDE framework, integrating individual enhanced features hi' with the global collaborative representation. The estimated state value Vi is obtained after concatenation. The global collaborative representation h̄ is obtained by mean pooling the features of all N UAVs:

[0109]

[0110] in, , is the global collaborative feature vector, which is obtained by element-wise mean pooling of the enhanced features of N=20 UAVs; , is the enhanced feature vector of drone j after GAT update (j=1,2,...,N); N=20, is the total number of drones. The dimension is fixed at 128, independent of N, thus decoupling the Critic input dimension from the number of agents. Individual features hi' are integrated with global features. After splicing, the scalar state value Vi is output through a three-layer MLP, and the network structure is 256→256→128→1:

[0111] The first layer of the centralized Critic value head module (256 dimensions → 256 dimensions):

[0112] ;

[0113] Second layer (256 dimensions → 128 dimensions):

[0114] ;

[0115] Third layer (128 dimensions → 1 dimension, output value):

[0116] ;

[0117] in, , is the scalar value estimate of the current state of drone i, representing the expected cumulative reward with discounts that can be obtained by executing the current strategy from the current state; It is the individual feature hi' (128 dimensions) and the global feature (128-dimensional) concatenated vectors; || is the vector concatenation operation; , , which are the feature vectors of the intermediate layer; , is the first-level weight matrix; —Second layer weight matrix; — Output layer weight matrix; , which are the bias vectors for each layer. Critic only calculates the value Vi for advantage function estimation during the training phase; it does not participate in decision-making during the execution phase and does not increase deployment computational overhead.

[0118] The entire network has a total of 269,831 trainable parameters, including approximately 33,024 for the MLP encoder, approximately 131,328 for the two-layer GAT, approximately 17,667 for the Actor policy heads, and approximately 87,812 for the Critic value heads. The optimizer configuration and hyperparameters are as follows: the Actor parameters use the Adam optimizer, with an initial learning rate lra = 3 × 10⁻⁶. -4, cosine annealing is scheduled to the minimum learning rate lra,min = 3×10 -5 ; The Critic parameters use an independent Adam optimizer with a learning rate lrc = 9×10 -4 (3 times the Actor learning rate to accelerate the convergence of the value function); The Adam momentum parameters of both optimizers are β1 = 0.9 and β2 = 0.999. Online observation normalization adopts the RunningMeanStd algorithm to maintain the mean and variance , which are saved together with the model weights to ensure that the observation distributions in the training and deployment phases are exactly the same.

[0119] S3. Design a hierarchical multi-objective reward function;

[0120] Further, the specific implementation method of step S3 includes the following steps:

[0121] S3.1. Design the reaching target reward. When the drone i satisfies the reaching condition ‖Δgi(t)‖ ≤ ragent at time t, give the reaching target reward, , where Δgi(t) is the Euclidean distance of the drone i from the target point gi at time t, and ragent is the drone collision radius, is the reaching target reward of the drone i, which is a one-time triggered reward;

[0122] Further, compared with the original method of +20, this reward makes the single reaching reward equivalent to the sum of 25-step progress rewards, ensuring that reaching the target always dominates in the global reward.

[0123] S3.2. Design the progress reward, and the expression is:

[0124]

[0125] Among them, is the progress reward of the drone i at time step t, is the progress weight coefficient, with a value of 1.5; ‖Δgi(t - 1)‖ is the Euclidean distance of the drone i from the target point at the previous time step t - 1; is the Euclidean distance of the drone i from the target point at time step t;

[0126] Further, when the drone approaches the target, rprog,i(t) > 0, and when it moves away from the target, rprog,i(t) < 0, forming a continuous gradient signal to guide the drone to continuously approach the target.

[0127] S3.3. Design the approaching target exponential acceleration reward, which is triggered only when ‖Δgi(t)‖ < dprox, where dprox is the trigger distance threshold, and the expression is:

[0128]

[0129] in, A bonus is awarded to drone i for accelerating its approach to the target at time step t. For attenuation scale parameters;

[0130] Furthermore, when ||Δgi(t)|| = 8 m ≈0.08, when |Δgi(t)|=1 m The reward is approximately 0.41, meaning the reward increases exponentially as the distance decreases, effectively solving the problem of difficult exploration in the final stage.

[0131] S3.4. Design obstacle classification penalties. Design three levels of penalties for the nearest obstacle, including danger zone, warning zone and early warning zone;

[0132] Furthermore, a three-tiered penalty is designed for the nearest obstacle (numbered l=1), with its surface distance defined as dobs,1(t)=‖pi(t)-c1‖-r1:

[0133]

[0134]

[0135]

[0136] in, `dobs,1(t)` is the obstacle penalty (negative value) for drone i at time step `t`; `dobs,1(t)` is the surface distance (m) between drone i and the nearest obstacle (l=1), equal to the center distance ‖pi(t)−c1‖ minus the obstacle radius `r1` (m); `pi(t)` is the 3D position coordinates (m) of drone i at time `t`; `c1` is the coordinates of the center of the nearest obstacle sphere (m); `r1` is the radius (m) of the nearest obstacle sphere; penalty coefficients of 3.0, 0.8, and 0.15 correspond to the three levels of intensity: danger, warning, and alert, respectively, with a smooth gradient transition, supporting drone braking and avoidance 2–3 m in advance. When `dobs,1(t)≤0` (actual collision occurs), an additional collision penalty of −20 is applied, but the agent does not disappear (continues to survive), ensuring that the RL algorithm can learn and correct behavior from subsequent experience after the collision.

[0137] S3.5. Design a layered penalty system for soft constraints between intelligent agents. Design a three-layered penalty system for the distance between drone i and its nearest neighbor j. The three layers include a collision zone, a warning zone, and an early warning zone.

[0138] Furthermore, regarding the distance between drone i and its nearest neighbor j... (t) = ||pi(t) - pj(t)||, design a three-level hierarchical penalty:

[0139]

[0140]

[0141]

[0142] in, It is the inter-agent penalty (negative value) for drone i at time step t. pj(t) is the Euclidean distance (m) between drone i and its nearest neighbor j at time t; pj(t) is the three-dimensional position coordinate (m) of neighbor drone j at time t; the collision threshold is 1.0 m = 2 × ragent, which is equal to the sum of the collision radii of the two drones; the penalty coefficients of 15.0, 2.0, and 0.4 correspond to the three levels of intensity: collision, warning, and alert, respectively; the outer boundary of the warning zone is 6.0 m = 6 × ragent, designed according to Long et al. (ICRA 2018), enabling drones to perceive and avoid neighbors in advance over a larger range. At the time of collision ( (t)<1.0 m), both drone i and its neighbor j are penalized by −15.0.

[0143] S3.6. Design a team collaboration reward, expressed as:

[0144]

[0145] in, Rewards for the collaborative team that uses drones at time step t. For collaboration weighting coefficients, Let t be the total number of drones in the cluster that have successfully reached their respective target points, and N be the total number of drones in the cluster. This reward enables individual drones that are still flying to perceive and motivate the overall progress of the team. When more teammates reach the target, individuals receive higher rewards, thus aligning individual incentives with team goals.

[0146] S3.7. Design orientation alignment reward, the expression is:

[0147]

[0148] in, The orientation alignment reward for drone i at time step t. For orientation weighting coefficients, Let be the angle between the current velocity direction of UAV i and the target direction;

[0149]

[0150] Among them, is the three-dimensional velocity vector (m / s) of the UAV i at time t; = Δgi(t) / ‖Δgi(t)‖, which is the unit direction vector (dimensionless) of the UAV i pointing to the target; ‖·‖ is the vector L2 norm. When the velocity direction is exactly the same as the target direction, cosθi = 1 and the reward is the largest; when they are perpendicular = 0; when departing from the target < 0, which is converted into a penalty to guide the UAV to take a more direct path towards the target.

[0151] S3.8. Construct a boundary soft constraint penalty, which is triggered when the minimum distance dbound,i(t) of the UAV i from any spatial boundary < dsafe, where dsafe is the boundary safety distance threshold, and the expression is:

[0152]

[0153] Among them, is the boundary penalty of the UAV i at time step t, is the boundary penalty weight coefficient;

[0154] Furthermore, = 2.0 m; when dbound,i(t) ≤ 0 (a hard collision occurs when flying out of the boundary), an additional penalty of -0.3 is imposed and elastic rebound processing is performed on the velocity.

[0155] S3.9. Design a hierarchical multi-objective reward function is the linear superposition of the eight components in steps S3.1 and S3.8.

[0156] Furthermore,

[0157]

[0158] The reward function is designed according to five priority levels: arrival reward (+50, strong main signal) > progress + proximity reward (dense guidance) > safety constraint penalty (collision penalty) > soft constraint warning penalty (graded warning) > cooperation + orientation + boundary regulation (fine guidance), ensuring that the network always takes reaching the target as the primary task during the optimization process.

[0159] The sub-items of the hierarchical multi-objective reward function are shown in Table 1:

[0160] Table 1

[0161]

[0162] S4. Based on the hierarchical multi-objective reward function obtained in step S3, the MAPPO+GAT-GNN policy network with distance-biased attention is trained using the APF-assisted residual policy course. The training data comes from online interactive simulation data.

[0163] After completing one training round, perform MAPPO policy optimization, using PPO to truncate the objective function to update the policy network parameters, and then repeat the iterative training until the set number of training rounds is reached.

[0164] Furthermore, the specific implementation method of step S4 includes the following steps:

[0165] S4.1. Based on online interactive simulation to collect training data, in each training round, N=20 drones are set to interact in real time with 12 random spherical obstacles in a 3D space of 30×30×15 m. In each step, the oi and ai of drone i are generated. exec R i Vi, the quintuple trajectory of the logarithmic probability of actions logπ(aiexec|oi);

[0166] Furthermore, S4.1.1. Training scenario initialization: In each round, a random seed is randomly selected, and 12 spherical obstacles (with radii uniformly distributed in [0.8,1.8] m and a spacing ≥3.5×ragent) are randomly placed in the [0,W]³ space. N=20 drones are uniformly initialized in a three-dimensional mesh (with a spacing ≥3.5×ragent to avoid initial collisions), and each drone is randomly assigned a target position gi.

[0167] S4.1.2. APF Guiding Action Calculation: Using the APF formula, the potential field guiding action of each UAV is calculated based on the current observations. The APF action is obtained by normalizing the superposition of the target's gravitational force and the obstacle's repulsive force. Drones that have already reached the target are excluded from the APF repulsive force calculation to prevent them from interfering with drones that are still navigating.

[0168] S4.1.3. Dynamic scheduling of APF weights: The APF weights hw are dynamically adjusted with each training round ep according to the following piecewise function:

[0169] hw(ep) = max(0.75, 1.0 - ep × (1.0 - 0.75) / 400);

[0170] That is, hw linearly decays from 1.0 at ep=1 (fully APF-guided) to 0.75 at ep=400 and then remains fixed. This ensures a 75% safety margin for APF throughout training, and RL only needs to learn an additional 25% residual optimization space based on APF. Compared with existing methods (Zhao et al. 2024, PGAT method) that completely withdraw APF (hw→0%), this method retains a 75% APF safety margin, increasing the convergence success rate from ~0% to 89.5%.

[0171] S4.1.4. Execution of Action Calculation and Trajectory Collection: This involves calculating the mixed action AI... exec It operates on the environment, collecting trajectory data for each step. The PPO loss function uses log π(aiexec|oi) instead of log π(aiRL|oi) to ensure that the policy gradient is correctly aligned with the reward signal corresponding to the actual action performed.

[0172] S4.2. After collecting the trajectory of a complete round, calculate the advantage function using generalized advantage estimation, and adopt the PPO truncation objective function update strategy:

[0173] L_CLIP(θ) = E[min(ρt(θ)·Ât, clip(ρt(θ), 1-ε, 1+ε)·Ât)]

[0174] Where L_CLIP(θ) is the objective function of the PPO pruning strategy, E is the expectation of the mini-batch sample, ρt(θ) is the probability ratio of the new and old strategies, Ât is the advantage estimate after mini-batch normalization, clip is the cutoff function, and ε is the PPO pruning hyperparameter.

[0175] In each policy update cycle, the Actor policy network is updated once, and the centralized Critic value network is updated six times. The gradient norm pruning method is used to suppress parameter update fluctuations, and the maximum gradient norm is set to 0.5.

[0176] The training consisted of 500 simulation rounds, with a maximum number of steps per round set to Tmax = 400 steps, and a 30-second timeout protection was set.

[0177] Table 2 shows a comparison between this embodiment and the prior art:

[0178] Table 2

[0179]

[0180] The following is an example illustrating its application:

[0181] (1) Experimental setup:

[0182] The experiment was conducted in a 3D space of 30×30×15 m. N=20 drones needed to navigate from their initial positions to their respective target points in an environment with M=12 randomly placed spherical obstacles (radius 0.8–1.8 m). Evaluation was based on 10 fixed random seeds (SEED+i, i=0,1,...,9) running independently. Here, "fixed random seeds" means that each method was run under exactly the same 10 scenario configurations during evaluation. With the seed values ​​fixed, the spatial positions and sizes of the obstacles, and the starting and target coordinates of each drone, remained constant, thus eliminating the interference of scenario randomness on the comparison results of different methods. This ensured that differences in success rate, collision rate, and other metrics stemmed solely from the performance of the strategy itself, rather than the favorable or unfavorable nature of the random scenario. The maximum number of steps per episode was Tmax=400. Three core metrics were recorded: mission success rate ρs (the proportion of drones reaching the target), collision rate ρc (the proportion of drones colliding), and average number of steps completed. (Average number of steps for all drones to arrive or time out). Comparison methods include: APF baseline (a traditional artificial potential field method with fully optimized parameters, including repulsion weights and perception range optimized for this scenario) and basic MAPPO (a pure reinforcement learning method without GAT graph attention module and without APF baseline).

[0183] (2) Comparison results:

[0184] The table below lists the quantization performance comparison of the three methods on 10 test seeds, as shown in Table 3, and the experimental results compared with the existing methods (N=20, M=12, mean of 10 test seeds).

[0185] Table 3

[0186]

[0187] (3) Results Analysis

[0188] First, in terms of mission success rate, the method in this embodiment achieves 89.5%, an improvement of 3.5 percentage points compared to the APF baseline (86.0%) and 85.3 percentage points compared to the basic MAPPO (4.2%). The APF baseline, as a well-tuned traditional method, is highly competitive in this scenario; however, its fixed potential field parameters still suffer from local minima when facing 12 randomly distributed dense obstacles, causing some drones to oscillate and fail to reach the target. The basic MAPPO method, lacking APF safety guidance, fails to converge completely in a 3D scene with sparse rewards and dense obstacles, achieving a success rate of only 4.2%, thus verifying the necessity of the APF safety mechanism in this invention.

[0189] Secondly, regarding the collision rate, the collision rate of the method in this embodiment is 5.0%, which is 5.0 percentage points lower than the APF baseline (10.0%) and a significant 76.2 percentage points lower than the basic MAPPO (81.2%). Although the APF baseline has a good success rate, due to the oscillation caused by the superposition of repulsive forces between agents in multi-agent scenarios, the actual collision rate (10.0%) is higher than that of the method in this invention (5.0%). This invention models the geometric relationships between agents through the GAT graph attention mechanism and combines it with the hierarchical penalty and reward of soft constraints between agents, enabling the UAV to predict the trajectory of its neighbors and avoid them in advance, thereby significantly reducing the collision rate while improving the success rate.

[0190] Furthermore, considering the average number of steps completed, the method in this embodiment averages approximately 359 steps, a reduction of about 7% compared to the APF baseline (approximately 386 steps), demonstrating the advantage of the method in path planning efficiency. This is mainly due to two factors: first, the target direction unit vector dg,i in the 108-dimensional extended observation and the proximity exponential acceleration reward rprox,i(t) jointly guide the UAV to adopt a more direct target-approaching trajectory; second, the collaborative team reward rcoop,i(t) = 0.05 × nr(t) / N promotes coordination among UAVs, avoids unnecessary waiting or detours, and improves the overall navigation efficiency of the swarm.

[0191] Finally, from a communication perspective, none of the three methods require real-time communication during the execution phase. This embodiment employs the CTDE framework (centralized training-distributed execution) during the training phase. The centralized Critic obtains global information through global mean pooling, eliminating the need for direct communication between agents. During the execution phase, each UAV relies solely on its own 108-dimensional local observations for independent decision-making, resulting in complete decentralization and excellent engineering deployability. In summary, this invention outperforms the comparative methods in success rate, collision rate, and path efficiency without introducing additional communication overhead, validating the comprehensive effectiveness of the proposed method.

[0192] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0193] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for autonomous cooperative obstacle avoidance of unmanned aerial vehicle (UAV) swarms based on multi-agent reinforcement learning, characterized in that, It includes the following steps: S1. Construct a three-dimensional extended observation vector of the UAV, including its own state vector, neighbor feature vector, obstacle feature vector, and extended safety perception feature vector; S2. Construct a MAPPO+GAT-GNN policy network with distance-biased attention, including an MLP encoder module, a distance-biased graph attention network module, an Actor policy head module, a residual action fusion module, and a centralized Critic value head module connected in sequence. Process the 108-dimensional three-dimensional extended observation vector of the UAV obtained in step S1 and output enhanced features; S3. Design a hierarchical multi-objective reward function; S4. Based on the hierarchical multi-objective reward function obtained in step S3, use the residual policy curriculum training with APF assistance to train the MAPPO+GAT-GNN policy network with distance-biased attention. The training data comes from online interactive simulation data; After completing one training round, perform MAPPO policy optimization, update the policy network parameters using the PPO truncation objective function, and then perform repeated iterative training until the set number of training times is reached.

2. The method for autonomous cooperative obstacle avoidance of unmanned aerial vehicle (UAV) swarms based on multi-agent reinforcement learning according to claim 1, characterized in that, The specific implementation method of step S1 includes the following steps: S1.

1. Construct a 10-dimensional own state vector, a 56-dimensional neighbor feature vector, and a 30-dimensional obstacle feature vector; S1.

2. Construct a 12-dimensional extended safety perception feature vector, including a 3-dimensional unit vector of the direction of the nearest obstacle, a 3-dimensional unit vector of the target direction, a 1-dimensional normalized distance of the nearest obstacle, a 1-dimensional normalized distance of the nearest neighbor, a 1-dimensional current speed magnitude, a 1-dimensional target distance emphasis, a 1-dimensional step progress, and a 1-dimensional team arrival ratio; S1.

3. Combining steps S1.1 and S1.2, obtain the three-dimensional extended observation vector of UAV i. , It is a real matrix.

3. The method for autonomous cooperative obstacle avoidance of UAV swarms based on multi-agent reinforcement learning according to claim 2, characterized in that, The specific implementation method of step S2 includes the following steps: S2.

1. Construct an MLP encoder module consisting of three linear layers connected in series. Each linear layer is followed by a normalized LN and a linear rectified activation function ReLU, which maps the three-dimensional extended observation vector oi of UAV i to the hidden state feature vector. ; S2.

2. Construct a distance-biased graph attention network module. N drones are abstracted as N agents. An interaction graph is constructed from the hidden state feature vectors {h1, h2, ..., hN} of the N agents, with each agent acting as a node in the graph. Neighborhood collaboration is modeled through two layers of graph attention updates. A distance bias term is introduced on top of the standard multi-head attention mechanism. After obtaining the attention coefficients, weighted aggregation and residual connections are used to obtain the output of the m-th layer. After two layers of GAT updates, the final output is an enhanced feature vector. ; S2.

3. Construct the Actor policy head module to enhance feature vectors. The motion is mapped to a 3D normalized acceleration motion using a three-layer MLP. The first and second layers of the three-layer MLP are ReLU functions, and the third layer is a hyperbolic tangent function, ultimately outputting the RL motion. ; S2.

4. Construct a residual motion fusion module to fuse RL actions. The APF-guided action is linearly mixed with the APF-weighted action to form the actual action to be executed, expressed as: ； in, Let be the final action vector of drone i. This is an element-wise truncation function. These are the APF weighting coefficients. The three-dimensional guiding action is calculated in real time based on the current state using the artificial potential field method, and is obtained by normalizing the combined target attraction and obstacle repulsion. S2.

5. Construct a centralized Critic value head module, which integrates the enhanced feature vector hi' with global features. After concatenation, the scalar value estimate Vi of the current state of UAV i is output through a three-layer MLP. The centralized Critic value head module calculates Vi during the training phase for advantage function estimation, but does not participate in decision-making during the execution phase.

4. The method for autonomous cooperative obstacle avoidance of unmanned aerial vehicle (UAV) swarms based on multi-agent reinforcement learning according to claim 3, characterized in that, The specific implementation method of step S3 includes the following steps: S3.

1. Design a reward for reaching the target. When drone i satisfies the arrival condition ‖Δgi(t)‖≤ragent at time t, a reward is given for reaching the target. Where Δgi(t) is the Euclidean distance between UAV i and target point gi at time t, and ragent is the collision radius of the UAV. The reward for drone i reaching the target is a one-time trigger reward; S3.

2. Design a progress reward, and the expression is: ； in, The progress reward for drone i at time step t. is the progress weighting coefficient, with a value of 1.5; ‖Δgi(t-1)‖ is the Euclidean distance between UAV i and the target point at the previous time step t-1; Let be the Euclidean distance between UAV i and the target point at time step t; S3. ； in, A bonus is awarded to drone i for accelerating its approach to the target at time step t. For attenuation scale parameters; ​ ​ ​ ； in, Rewards for the collaborative team that uses drones at time step t. For collaboration weighting coefficients, Let N be the total number of drones in the cluster that have successfully reached their respective target points at time t, and let N be the total number of drones in the cluster. ​ ； in, The orientation alignment reward for drone i at time step t. For orientation weighting coefficients, Let be the angle between the current velocity direction of UAV i and the target direction; ​ ； in, For the boundary penalty of drone i at time step t, The boundary penalty weight coefficient; S3.

9. Design a hierarchical multi-objective reward function It is a linear superposition of the eight components of steps S3.1 and S3.

8.

5. The method for autonomous cooperative obstacle avoidance of unmanned aerial vehicle (UAV) swarms based on multi-agent reinforcement learning according to claim 4, characterized in that, ​ S4.

1. Based on online interactive simulation to collect training data, in each training round, N=20 drones are set to interact in real time with 12 random spherical obstacles in a 3D space of 30×30×15 m. In each step, the oi and ai of drone i are generated. exec R i Vi, the quintuple trajectory of the logarithmic probability of actions logπ(aiexec|oi); ​ ​ Where L_CLIP(θ) is the objective function of the PPO pruning strategy, E is the expectation of the mini-batch sample, ρt(θ) is the probability ratio of the new and old strategies, Ât is the advantage estimate after mini-batch normalization, clip is the cutoff function, and ε is the PPO pruning hyperparameter. In each policy update cycle, the Actor policy network is updated once, and the centralized Critic value network is updated six times. The gradient norm pruning method is used to suppress parameter update fluctuations, and the maximum gradient norm is set to 0.

5. The training consisted of 500 simulation rounds, with a maximum number of steps per round set to Tmax = 400 steps, and a 30-second timeout protection was set.

Images