A cooperative trajectory planning method for large-scale UAV swarms based on fractal graph reinforcement learning

Abstract

This invention discloses a large-scale UAV swarm cooperative trajectory planning method based on fractal graph reinforcement learning. First, each UAV in the swarm is treated as a low-level node, and a low-level graph is constructed based on the communication or cooperative relationships between nodes. Then, a learnable soft allocation method is used to recursively aggregate the low-level nodes into multiple layers of upper-level group nodes, forming a fractal graph structure. Next, at each layer of the fractal graph structure, the global cooperative features of the divided UAV groups are extracted using a graph attention network. Finally, through recursive aggregation layer by layer, at least one top-level global cooperative feature is formed to generate the target task, which is then passed down to modulate the feature representations of the low-level nodes, fusing the target task with the low-level node feature representations to obtain a new low-level feature vector. Finally, at the low level, the control actions of each UAV are output based on the modulated node features to generate the flight trajectory of each UAV.

Description

Technical Field

[0001] This invention relates to the field of UAV swarm collaborative planning technology, specifically to a large-scale UAV swarm collaborative trajectory planning method based on fractal graph reinforcement learning. Background Technology

[0002] Cooperative trajectory planning for unmanned aerial vehicle (UAV) swarms is a current research hotspot in the field of intelligent unmanned systems, showing broad application prospects in complex tasks such as reconnaissance and surveillance, target acquisition, and cooperative search. However, with the expansion of swarm size, the dynamic changes in factors such as the number of UAVs, communication topology, and mission objectives lead to a sharp increase in the complexity of the decision space. Existing methods generally face challenges in computational efficiency, structural generalization, and balancing global and local coordination when dealing with cooperative trajectory planning for large-scale swarms.

[0003] Currently, the existing technologies related to this technical solution mainly include the following categories:

[0004] 1. Collaborative planning methods based on graph neural networks or graph attention networks

[0005] This type of method models the drone and the target as nodes in a graph structure, uses graph neural networks to extract the interaction features between nodes, and then outputs decision actions. This method can capture the local relationships between drones relatively well, but it has the following limitations:

[0006] The computational complexity is too high: when using global attention or dense graph modeling, the computational cost of relationships between nodes increases quadratically with the size of the UAV, making it difficult to meet the real-time requirements of online planning for large-scale clusters.

[0007] The disconnect between global and local collaboration: Most methods process information only at a single-layer graph structure, lacking a multi-layered, self-similar, nested organizational structure. This results in models that, while capable of handling global collaboration, neglect local squad-level collaboration, whereas in real-world missions, UAVs often need to operate in dispersed squad formations.

[0008] Simplified neighborhood strategies lose global information: Fixed neighborhood graphs or K-nearest neighbor graphs used to reduce complexity can reduce computation, but they ignore interactions between distant or different groups of nodes, resulting in the loss of global relationship information, which is not conducive to overall collaboration.

[0009] The structure cannot be recursively extended: Existing graph pooling or coarsening methods usually only perform single compression, and the network structure changes before and after compression, making it difficult to achieve nested recursive expansion, cross-layer parameter sharing, and model transfer between different scales.

[0010] 2. Hierarchical Reinforcement Learning-Based Approach

[0011] This type of approach divides swarm control into multiple layers, such as high-level decision-making and low-level control (e.g., the high-level layer determines the target point, and the low-level layer implements specific flight maneuvers), in an attempt to achieve scalability. However, this method has inherent limitations:

[0012] Heterogeneous networks and complex designs: The upper and lower layers often use different network structures and optimization objectives, resulting in complex overall design and high training difficulty.

[0013] Difficulty in dynamically scaling: The heterogeneity of the network structure at each layer makes it impossible to take advantage of the structural homogeneity in the drone swarm. When the swarm size or hierarchy changes, it is often necessary to redesign and retrain the network, making it difficult to adapt dynamically.

[0014] Training is difficult and performance is unstable: the high interlayer coupling and large differences in convergence speed can easily lead to training oscillations, difficulties in credit allocation, and performance degradation.

[0015] Weak cross-scale transfer capability: Models trained on small-scale clusters usually cannot be directly transferred to large-scale clusters due to differences in network structure, and the cost of retraining is high.

[0016] 3. Other existing technical methods

[0017] Fixed-rule-based methods: These methods achieve trajectory planning by manually defining control rules. While highly interpretable, these rules have limited expressive power and cannot handle complex dynamic environments; they lack self-learning and evolutionary capabilities, and performance improvements rely on manual adjustments; when the environment or task changes, the rules must be redesigned.

[0018] The centralized planning approach involves constructing a central node to uniformly acquire the status of all UAVs and solve for the globally optimal trajectory. Theoretically, it can obtain the globally optimal solution, but it suffers from fatal flaws such as high communication load, computational overhead increasing dramatically with scale, poor real-time performance, and single-point failure of the central node.

[0019] In summary, existing technologies generally suffer from the following technical problems when dealing with large-scale UAV swarm collaborative trajectory planning tasks: excessively high computational complexity in large-scale scenarios, making it difficult to guarantee real-time online planning; lack of a unified multi-scale structure, making it difficult to balance local flexibility and global consistency; inconsistent hierarchical decision-making mechanisms, leading to difficulties in cross-layer training and complex credit allocation; weak model generalization ability across scales, difficulty in parameter reuse, and high training costs; and a lack of a collaborative planning framework that can simultaneously balance computational efficiency, structural uniformity, robustness, and strong generalization ability.

[0020] Therefore, how to construct a large-scale UAV swarm collaborative trajectory planning method that is structurally unified, recursively extensible, can simultaneously integrate local and global collaborative information, and supports cross-scale parameter sharing and model transfer while maintaining acceptable computational complexity is a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0021] This invention aims to provide a large-scale UAV swarm collaborative trajectory planning method based on fractal graph reinforcement learning, in order to solve the technical problems existing in the prior art, such as high computational complexity, inconsistent hierarchical structure, difficulty in balancing global and local collaboration, and weak model cross-scale transfer capability, so as to achieve efficient, scalable and robust swarm collaborative trajectory planning.

[0022] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0023] A large-scale UAV swarm cooperative trajectory planning method based on fractal graph reinforcement learning, comprising the following steps:

[0024] Step 1: Construct the fractal graph structure

[0025] Each drone in the drone swarm is treated as a bottom-level node, and a bottom-level graph is constructed based on the communication or cooperative relationships between the nodes. Then, a learnable soft allocation method is used to recursively aggregate the bottom-level nodes into multiple levels of upper-level group nodes, thereby forming a fractal graph structure with self-similar properties.

[0026] Specifically, during each aggregation, the matching score between low-level nodes and upper-level prototype nodes is calculated and normalized to obtain a soft assignment matrix. This soft assignment matrix is ​​then used to coarseen the features and adjacency relationships of low-level nodes to obtain the features and adjacency relationships of upper-level nodes in the current layer. This allows for the construction of a fractal graph structure by connecting node features from different levels through adjacency relationships. This process can be repeated to form higher-level abstract representations.

[0027] Step 2: Hierarchical Feature Extraction and Modulation

[0028] At each layer of the fractal graph structure, local features of the divided UAV groups are extracted using a graph attention network. Low-level features mainly reflect local interaction details, while high-level features reflect the overall situation and group-level collaborative trends. Local features are recursively aggregated layer by layer to generate global collaborative features containing at least one top-level target task. These global collaborative features are then passed down to modulate the feature representations of lower-level nodes, allowing them to fuse with the target task from higher levels to obtain new lower-level feature vectors. This enables the global target to guide local actions.

[0029] Step 3: Action Generation

[0030] At the lower level, control actions for each UAV are output based on modulated node features. These control actions include, but are not limited to, turning, climbing, descending, accelerating, and decelerating. Each UAV updates its flight status according to the action commands, gradually forming a swarm cooperative flight path.

[0031] Step 4: Joint Training

[0032] A reward function is designed, and a reinforcement learning algorithm is used to jointly train the policy network, value network, and soft assignment matrix from the above steps. During training, the action policy, multi-level value evaluation, and graph aggregation constraint terms are simultaneously optimized to improve the consistency between hierarchical decision-making and graph structure learning.

[0033] Preferably, the learnable soft allocation method specifically includes:

[0034] First, the feature representation of the bottom-level nodes is obtained by upscaling the bottom-level nodes through a fully connected layer;

[0035] Let the underlying node embedding be represented as ,in Indicates the number of underlying nodes. This represents the dimension of node features. Let the number of upper-level prototype nodes be... Then the matching score from the node to the upper prototype can be expressed as:

[0036]

[0037] in Query feature vectors for the bottom-level nodes. This represents the upper-level prototype. After calculating the matching score, the matching score is normalized according to the prototype dimension to obtain the soft assignment matrix:

[0038]

[0039] in , Indicates the first The bottom-level node belongs to the first The weights of each upper-level group node are determined. After obtaining the weights, the coarsened features of the upper-level nodes are obtained based on the soft assignment matrix.

[0040]

[0041] in, This represents the normalized quality term for each group. The corresponding upper-level adjacency relation can be represented as:

[0042]

[0043] in, This is the adjacency matrix of the underlying graph. This is the coarsened upper-level adjacency matrix.

[0044] Preferably, a constraint term is introduced into the soft allocation matrix during training. This constraint term includes at least one of allocation entropy constraints, orthogonality constraints, and structure preservation constraints. Wherein:

[0045] The allocation entropy constraint is ,

[0046] Orthogonal constraints are ,

[0047] The structure maintains the constraint. .

[0048] The final aggregate constraint of the entire graph can be expressed as:

[0049]

[0050] in, These are the weight parameters for each constraint term.

[0051] Preferably, the step of passing the target task down and modulating the features of lower-level nodes specifically involves: using the target task in the global collaborative features as additional guiding information, and combining it with the lower-level feature vectors of the lower-level nodes, so that the features of the lower-level nodes contain the high-level collaborative target by reweighting, superimposing features, or gating modulation of the features of the lower-level nodes.

[0052] Preferably, the method for generating the control action instructions is as follows:

[0053]

[0054] in, Indicates the first Modulated features of individual drone nodes This indicates the control actions of the drone. Indicates the network parameters of the strategy.

[0055] Preferably, the reward function is designed based on the collaborative coordination effect, track quality, and safety constraints, and its expression is as follows:

[0056] Let the team reward at time t be Then it can be expressed as:

[0057]

[0058] in Represents the coordinated allocation of rewards. Represents the award for advancing the flight path. Rewards for consistency in direction. Represents task load balancing rewards. Represents smooth action reward. Represents collision penalty, Represents punishment for crossing the line. This represents the penalty for approaching an obstacle, while This represents the weighting coefficients for each item, which are used to adjust the focus.

[0059] Among them, the reward for coordinated allocation effect is used to measure the rationality of task allocation, the effectiveness of group organization, and the balance of resource utilization; the reward for track quality is used to measure the effectiveness, smoothness, and execution quality of the track; and the penalty for safety constraints is used to measure behaviors such as collisions, boundary crossings, insufficient safety intervals, or excessive changes in actions.

[0060] Preferably, the reinforcement learning algorithm employs a proximal policy optimization algorithm, and its total loss function is:

[0061]

[0062] in, For strategic losses, For the loss of value, For policy entropy regularization, For graph aggregation constraints, These are the weighting coefficients.

[0063] In a preferred embodiment of the present invention, when the size of the drone cluster changes, the model parameters trained on a small-scale cluster can be directly transferred to a larger-scale cluster for use by reconstructing the number of nodes in the underlying graph and the corresponding hierarchical aggregation relationship, thereby achieving cross-scale parameter sharing.

[0064] In a preferred embodiment of the present invention, the number of layers in the fractal graph structure is dynamically adjusted according to the size of the drone swarm.

[0065] In a preferred embodiment of the present invention, the features of the underlying node include position, speed, heading, altitude, target direction, local obstacle information, boundary information, search status, and target confidence information.

[0066] This invention has the following characteristics and beneficial effects:

[0067] This invention abandons the traditional single-layer global graph modeling approach. By recursively coarsening large-scale UAV swarms into multiple layers of local subgraphs (fractal graph structure), it effectively reduces the number of edges in the graph structure. Compared to the problem of computational complexity increasing quadratically with the number of nodes when using global attention or dense graph modeling, this invention significantly reduces the computational overhead of graph modeling and feature extraction while maintaining key collaborative information, thus meeting the needs of online real-time planning for large-scale swarms.

[0068] This invention employs isomorphic graph modeling, intent generation, and policy evaluation structures at each layer (individual layer, squad layer, sub-cluster layer, etc.), forming a unified fractal graph recursive framework. This structural consistency allows model parameters trained on small-scale clusters to be directly transferred to larger-scale clusters without requiring network redesign or retraining from scratch. This not only significantly reduces training costs in large-scale scenarios but also achieves cross-layer parameter reuse and cross-scale generalization, overcoming the shortcomings of existing hierarchical reinforcement learning methods that are difficult to transfer due to network heterogeneity.

[0069] This invention uses lower-level nodes to handle fine-grained action output, while higher-level nodes aggregate to generate the overall situation and target tasks. A layer-by-layer modulation mechanism is then used to transmit the higher-level intentions downwards, thus organically integrating local interactive details with global collaborative goals. When executing actions, the lower-level UAVs simultaneously perceive their surrounding environment and receive strategic guidance from higher levels, effectively solving the problem of the separation between global collaboration and local control in existing methods. This achieves a good balance between local maneuverability and overall task consistency.

[0070] This invention does not rely on fixed grouping or preset formations, but instead maps lower-level nodes to upper-level groups through a learnable soft-assignment matrix. When the number of drones, their locations, mission relationships, or the environment changes, the soft-assignment weights can be dynamically adjusted, thereby flexibly reconstructing hierarchical organizational relationships. Compared to rigid grouping strategies such as fixed neighborhoods or K-nearest neighbor graphs, this invention exhibits stronger robustness and adaptability to dynamic and uncertain combat or operational scenarios.

[0071] This invention simultaneously optimizes the underlying action policy, multi-layer value evaluation function, and allocation entropy constraints, orthogonality constraints, and structure preservation constraints during graph aggregation during training. This joint optimization mechanism helps alleviate common problems in hierarchical reinforcement learning, such as training oscillations, difficulties in credit allocation, and performance degradation, resulting in a more stable overall learning process, faster convergence, and higher policy quality.

[0072] In this invention, strategic intent (target task) is generated at a high level and transmitted from top to bottom. For different task requirements (such as cooperative search, target encirclement, area coverage, formation maneuvering, multi-stage cooperation, etc.), only the top-level target representation or high-level input needs to be modified, without changing the network structure or retraining the entire model. This modular design greatly improves the method's versatility and task transferability, and lowers the application threshold for new tasks.

[0073] The model trained on a 16-drone swarm can still generate reasonable, smooth, and safe cooperative trajectories when directly transferred to a 100-drone swarm, achieving zero-sample transfer. This experimental result strongly demonstrates the significant advantage of the proposed method in cross-scale generalization ability, which is difficult to achieve with existing graph neural networks and hierarchical reinforcement learning methods.

[0074] In summary, this invention achieves superior results compared to existing technologies in terms of computational efficiency, structural uniformity, dynamic adaptability, training stability, task generalization ability, and cross-scale migration capability, providing an efficient, robust, and easy-to-deploy solution for collaborative trajectory planning of large-scale UAV swarms. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 This is a flowchart illustrating the overall process of fractal graph reinforcement learning collaborative trajectory planning according to an embodiment of the present invention.

[0077] Figure 2 This is a network architecture diagram of the fractal graph reinforcement learning collaborative trajectory planning system in an embodiment of the present invention;

[0078] Figure 3 This is a schematic diagram of the fractal layer hierarchy structure in an embodiment of the present invention;

[0079] Figure 4 This is a flowchart illustrating the training closed-loop process according to an embodiment of the present invention.

[0080] Figure 5 This is a schematic diagram illustrating the results of collaborative flight path planning for 100 UAVs in an embodiment of the present invention. Detailed Implementation

[0081] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0082] This invention provides a large-scale UAV swarm cooperative trajectory planning method based on fractal graph reinforcement learning, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0083] Step 1: Node Feature Construction

[0084] At each decision-making moment, each UAV acquires its own state and local environmental perception information. In this embodiment, the original features of each UAV node include: position (x, y, z), velocity (vx, vy, vz), heading angle (yaw), altitude (h); relative target direction (dx, dy, dz), relative target distance; local obstacle information (obstacle distance vectors returned by LiDAR or visual sensors, reduced to 8 dimensions); boundary information (distance to the boundary of the task area); search status (whether the area has been searched); and target confidence information (such as the probability of target presence).

[0085] These raw features are encoded into fixed-dimensional (e.g., 128-dimensional) embedding vectors through a shared fully connected layer (Multilayer Perceptron, MLP), resulting in the feature matrix of the bottom-level nodes. ,in , .

[0086] Step 2: Graph state construction. Treat each drone in the drone swarm as a low-level node and construct the underlying graph based on the communication or collaboration relationships between the nodes.

[0087] Specifically, a bottom-level graph is constructed based on the communication relationships between drones, their spatial proximity, and the relationships between drones and mission targets and obstacles. In this embodiment, as shown... Figure 3 As shown, each drone, obstacle, and target task is represented as a node. If the Euclidean distance between two drones is less than the communication radius... (Based on the preset communication radius according to the actual communication equipment performance), corresponding bidirectional edges are added to the sparse edge index; if the UAV is within the obstacle perception range, an edge is established between the UAV and the obstacle; simultaneously, to enable the UAV to obtain mission target information, an association edge is established between the UAV node and the target node. Furthermore, to ensure that isolated nodes can still transmit their own characteristics, self-loop edges can be added to each node. This results in a graph structure stored in the form of a sparse edge index, whose node features are represented as follows:

[0088]

[0089] The edge connection relationship is represented as:

[0090]

[0091] Where N represents the number of nodes, D represents the node feature dimension, and E represents the number of edges.

[0092] Step 3: Fractal Graph Construction (Soft Assignment Recursive Aggregation). A learnable soft assignment method is used to recursively aggregate low-level nodes into multiple levels of upper-level group nodes, forming a fractal graph structure. During each aggregation, the matching score between the low-level node and the upper-level prototype node is calculated and normalized to obtain the soft assignment matrix. This soft assignment matrix is ​​used to coarseen the features and adjacency relationships of the low-level nodes, obtaining the features and adjacency relationships of the upper-level nodes in the current layer. Thus, different levels of fractal graph structures are constructed by using the node features of different levels through adjacency relationships.

[0093] Specifically, this embodiment adopts a two-layer fractal structure: bottom layer (individual layer, N=100) → middle layer (squad layer, K1=20) → top layer (cluster layer, K2=1). The number of prototype nodes in the upper layer, K1 and K2, are hyperparameters that can be set according to task requirements.

[0094] First aggregation (bottom layer → middle layer):

[0095] Low-level node embedding The query vector is obtained after linear transformation. ( ), upper-level prototype representation This is a learnable parameter matrix. Calculate the matching score:

[0096]

[0097] Softmax normalization is performed on the Score column-wise (prototype dimension) to obtain the soft allocation matrix. :

[0098]

[0099] Then, the normalized mass term for each group is calculated. ( (This is a very small constant, avoiding division by zero). The characteristics of middle-level nodes are:

[0100] The adjacency relationship of the middle layer is as follows:

[0101]

[0102] Second aggregation (middle layer → upper layer):

[0103] Features of middle-layer nodes As a new input, the upper-level prototype number Similarly, calculate the soft allocation matrix. (At this point, all intermediate nodes are assigned to a single high-level node with different weights). This yields the characteristics of the high-level nodes. .

[0104] Furthermore, during training, constraint terms are introduced into the soft allocation matrix, including at least one of allocation entropy constraints, orthogonality constraints, and structure preservation constraints; wherein,

[0105] The allocation entropy constraint is:

[0106] ,

[0107] Orthogonal constraints are:

[0108] ,

[0109] The structural preservation constraints are:

[0110] ;

[0111] The final aggregate constraint of the entire graph can be expressed as:

[0112]

[0113] in, , , The weight parameters for each constraint term.

[0114] Step 4: Hierarchical feature extraction. At each layer of the fractal graph structure, local features within the divided drone groups are extracted using a graph attention network.

[0115] At each layer (the number of layers is dynamically determined by soft aggregation based on the scale of the drones), a graph attention network is used to extract the overall features of each group. These features, through different attention weights, aggregate the strategic intent of the group or formation, thereby representing the current focus of the group or formation, the overall direction of movement, and other information. A graph attention network (GAT) is used to extract the local features of each layer (understandably, since it is necessary to obtain the current focus information of a group and the overall coordination information of the group formation, and graph attention networks can achieve good coordination, this embodiment uses graph attention networks to achieve this) to extract the local features of that layer. Taking the bottom layer as an example:

[0116] Input: Node features Adjacency matrix .

[0117] Calculate multi-head attention (e.g., 4 heads), with each head calculating the attention coefficient between nodes. Then, weighted aggregation of neighbor features yields the updated node features. .

[0118] Finally, the overall representation of this layer is obtained through global average pooling. .

[0119] Similarly, perform the same GAT operation on the middle layer (number of nodes K1) and the high layer (number of nodes 1) to obtain the overall representation of the middle layer. and the overall expression of the senior management .

[0120] Step 5: The high-level target task is modulated downwards. Local features are recursively aggregated layer by layer to generate a global collaborative feature containing at least one top-level target task. The global collaborative feature is then passed down to modulate the feature representation of the lower-level nodes, so that the feature representation of the lower-level nodes is fused with the target task of the higher level to obtain a new lower-level feature vector.

[0121] In the process of down-modulating the high-level target mission, FiLM modulation is used. First, scaling parameters and bias parameters are generated based on the high-level intent vector. , Each floor , The calculation formula is as follows:

[0122]

[0123]

[0124] in, , These represent modulation parameters, which control feature scaling and offset respectively; Represents Hadamard element-wise multiplication; Represents the feature matrix of low-level nodes; This represents the modulated low-level feature vector.

[0125] FiLM modulation is used to transmit the intentions of the upper layer to the lower layer nodes layer by layer, so as to realize the downward modulation of the target tasks of the higher layer.

[0126] Step 6: Action Output and Track Generation

[0127] At the underlying level, based on the modulated node features Each node outputs a probability distribution of discrete actions through a linear mapping layer and a softmax function. The control actions are represented by a discrete action set, which includes at least actions such as maintaining state, steering adjustment, climbing, descending, and speed adjustment. In this embodiment, the action set can be further subdivided into left turns, right turns, acceleration, and deceleration actions of different magnitudes to adapt to the maneuverability requirements of the UAV in complex environments.

[0128] For the i-th drone, its action probability distribution can be expressed as:

[0129]

[0130] in, This represents the feature representation of the i-th UAV node after being modulated by the high-level target task. This indicates the control actions of the drone. The parameters represent the policy network parameters, where W and b represent the linear mapping parameters, respectively.

[0131] During the training phase, actions are randomly sampled according to the above policy distribution, i.e.:

[0132]

[0133] This allows the strategy to retain necessary exploratory capabilities during training; and during the inference or deployment phase, the action with the highest probability can be directly selected as the output, i.e.:

[0134]

[0135] Each UAV updates its flight status based on the selected action. The status update is based on the UAV's kinematic or dynamic model, including changes in heading angle, speed, and altitude, and is integrated to obtain the position at the next moment, thereby gradually forming a continuous swarm cooperative flight path.

[0136] Step 7: Reward Design

[0137] This embodiment designs a combined reward function, which is based on the collaborative coordination effect, track quality and safety constraints, and its specific formula is as follows;

[0138] Let the team reward at time t be Then it can be expressed as:

[0139]

[0140] in Represents the coordinated allocation of rewards. Represents the award for advancing the flight path. Rewards for consistency in direction. Represents task load balancing rewards. Represents smooth action reward. Represents collision penalty, Represents punishment for crossing the line. This represents the penalty for approaching an obstacle, while This represents the weighting coefficients for each item, which are used to adjust the focus.

[0141] The reward for the collaborative allocation effect is used to measure the rationality of task allocation, the effectiveness of group organization, and the balance of resource utilization;

[0142] The reward for track quality is used to measure the effectiveness, smoothness, and execution quality of the track;

[0143] The penalties for the safety constraints are used to measure actions such as collisions, crossing boundaries, insufficient safety distances, or excessive changes in movement.

[0144] Step 8: Joint Training (Reinforcement Learning)

[0145] This embodiment uses the Proximal Policy Optimization (PPO) algorithm for training, and the training process is as follows: Figure 4 As shown.

[0146] The network parameters mainly include policy network, value assessment network, and fractal graph aggregation parameters.

[0147] The policy network comprises: an input feature mapping layer, a low-level graph encoder, an upper-level graph encoder, a soft allocation aggregation module, a target task generation module, a high-level intent down-modulation module, and a low-level action output layer. Both the low-level and upper-level graph encoders can employ graph attention network structures to extract collaborative features from different graph structures. The parameters in the soft allocation aggregation module include node query mapping parameters and upper-level prototype parameters. The low-level action output layer generates the action probability distribution for each UAV based on the modulated low-level node features.

[0148] The valuation network is used to estimate the value of the overall state at different levels. In this embodiment, the overall representation of each level is processed by the corresponding valuation head to obtain the level value. The underlying value is represented by the overall underlying value, the middle-level value by the overall middle-level value, and the upper-level value by the overall upper-level value. The overall value can be estimated by weighting and integrating the values ​​of each level according to preset weights, or by using an average integration method. For example, it can be represented as:

[0149]

[0150] Where L represents the number of layers, Indicates the first The weighting coefficients of the layer value can be set according to the task requirements.

[0151] To ensure the stability of policy training, this invention optimizes not only the underlying action policies during training but also the multi-layer value evaluation results and related constraints in the graph aggregation process, thereby achieving joint optimization of fractal graph structure learning, hierarchical collaborative decision-making, and trajectory planning policy learning. Furthermore, during training, this invention uses a policy pruning method based on the PPO algorithm to update the model, avoiding instability during training. The stability of training controlled by PPO lies in the policy pruning method, the core formula of which is as follows: For time t, the importance sampling ratio can be expressed as...

[0152]

[0153] The policy objective function can be expressed as:

[0154] ,

[0155] in, For the dominant function, This is the cutting factor.

[0156] Value loss can be expressed as

[0157]

[0158] in, The cumulative return for discounts (discount factor γ=0.99).

[0159] The final total loss function can be expressed as:

[0160]

[0161] in, The coefficient is the policy entropy regularization term. , Example , .

[0162] Training process: Collect a batch of trajectories (e.g., 2048 time steps) and calculate the advantage function. Update the policy and value network, and repeat until convergence.

[0163] Step 9: Cross-scale migration

[0164] This embodiment verifies the effectiveness of directly transferring a model trained on 16 drones to 100 drones. During the transfer, only the number of bottom-level nodes N needs to be changed to 100, the adjacency matrix A needs to be reconstructed, and the remaining network parameters (including feature encoding MLP, GAT weights, and soft assignment) remain unchanged. (etc.) Fully reused. The number of upper-layer prototype nodes K1 and K2 remains unchanged. Experimental results show that the transferred model can generate reasonable cooperative tracks without additional training, such as... Figure 5 As shown.

[0165] It should be noted that since all learnable parameters are independent of the number of underlying nodes and depend only on the feature dimension D, the model naturally supports changes in the number of nodes.

[0166] Based on the above technical solution, in this embodiment, as follows: Figure 5As shown, in a two-dimensional simulation environment, a task area of ​​4000×4000 meters was set up, with 100 UAVs and 32 targets. This method achieves dynamic coordination, where the UAVs dynamically adjust their formation and target tracking based on the actual scenario, achieving balanced allocation and task coordination. In this example, the number of UAVs allocated to each of the 32 targets was 3-4, achieving dynamic balance and coordinated target tasks. Furthermore, related simulation verification was simultaneously conducted in the AFSIM simulation engine. This example also showed good results in the AFSIM high-fidelity simulation platform, laying the foundation for real-world deployment.

[0167] The method proposed in this invention can be applied to tasks such as collaborative search, reconnaissance, strike, target acquisition, formation flight, and area coverage of military / civilian UAV swarms, and has broad application prospects.

[0168] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A large-scale UAV swarm cooperative trajectory planning method based on fractal graph reinforcement learning, characterized in that, Includes the following steps: Step 1: Constructing a fractal graph structure: Each drone in the drone swarm is treated as a low-level node. A low-level graph is constructed based on the communication or collaboration relationships between nodes. A learnable soft allocation method is used to recursively aggregate low-level nodes into multiple levels of upper-level group nodes, forming a fractal graph structure. During each aggregation, the matching score between the low-level node and the upper-level prototype node is calculated and normalized to obtain a soft allocation matrix. This soft allocation matrix is ​​used to coarseen the features and adjacency relationships of the low-level nodes, obtaining the features and adjacency relationships of the upper-level nodes in the current layer. Thus, different levels of fractal graph structures are constructed by using the node features of different levels through adjacency relationships. Step 2, Layered Feature Extraction and Modulation: Local features within the group of drones are extracted at each layer of the fractal graph structure using a graph attention network; the local features are recursively aggregated layer by layer to generate a global collaborative feature containing at least one top-level target task, and the global collaborative feature is passed down to modulate the feature representation of the bottom-level nodes, so that the feature representation of the bottom-level nodes is fused with the target task of the high-level nodes to obtain a new bottom-level feature vector. Step 3, Action Generation: At the bottom layer, the control actions of each UAV are output based on the modulated node features to generate the flight path of each UAV. Step 4, Joint Training: Design a reward function and use a reinforcement learning algorithm to jointly train the policy network, value network, and soft assignment matrix from the above steps. During the training process, the action policy, multi-level value evaluation, and graph aggregation constraint terms are optimized simultaneously.

2. The method according to claim 1, characterized in that, The learnable soft allocation method specifically includes: First, the feature representation of the bottom-level nodes is obtained by upscaling the bottom-level nodes through a fully connected layer; After obtaining the feature representation of the underlying nodes, let the feature representation of the underlying nodes be: Where N represents the number of underlying nodes and D represents the node feature dimension; Let K be the number of upper-level prototype nodes. Then the matching score from a node to the upper-level prototype can be expressed as: Where Q is the query feature vector of the bottom-level node, and P is the prototype representation of the upper layer. After calculating the matching score, the matching score is normalized according to the prototype dimension to obtain the soft assignment matrix: in, , This represents the weight of the i-th bottom-level node belonging to the k-th upper-level group node. After obtaining the weight, the coarsened features of the upper-level nodes are obtained based on the soft assignment matrix. Where mass represents the normalized mass term for each group; The corresponding upper-level adjacency relationship can then be represented as: Where A is the adjacency matrix of the bottom-level graph. This is the coarsened upper-level adjacency matrix.

3. The method according to claim 2, characterized in that, During training, constraints are introduced into the soft allocation matrix, including at least one of allocation entropy constraints, orthogonality constraints, and structure preservation constraints; wherein... The allocation entropy constraint is: ， Orthogonal constraints are: ， The structural preservation constraints are: ； The final aggregate constraint of the entire graph can be expressed as: in, , , The weight parameters for each constraint term.

4. The method according to claim 1, characterized in that, The process of passing the target task down and modulating the features of lower-level nodes specifically involves: using the target task in the global collaborative features as additional guiding information, and combining it with the lower-level feature vectors of the lower-level nodes; and using methods such as reweighting, feature superposition, or gating modulation of the lower-level node features to make the lower-level node features contain the high-level collaborative target.

5. The method according to claim 1, characterized in that, The control actions include steering, climbing, descending, accelerating, and decelerating.

6. The method according to claim 1, characterized in that, The method for generating the control action instructions is as follows: in, Indicates the first Modulated features of individual drone nodes This indicates the control actions of the drone. Indicates the network parameters of the strategy.

7. The method according to claim 1, characterized in that, The reward function is designed based on the collaborative deployment effect, track quality, and safety constraints, and its expression is as follows: Let the team reward at time t be Then it can be expressed as: in Represents the coordinated allocation of rewards. Represents the award for advancing the flight path. Rewards for consistency in direction. Represents task load balancing rewards. Represents smooth action reward. Represents collision penalty, Represents punishment for crossing the line. This represents the penalty for approaching an obstacle, while This represents the weighting coefficients for each item, which are used to adjust the focus. The rewards for the collaborative allocation effect are used to measure the rationality of task allocation, the effectiveness of group organization, and the balance of resource utilization; The reward for track quality is used to measure the effectiveness, smoothness, and execution quality of the track; The penalties for the safety constraints are used to measure actions such as collisions, crossing boundaries, insufficient safety distances, or excessive changes in movement.

8. The method according to claim 1, characterized in that, The reinforcement learning algorithm employs a proximal policy optimization algorithm, and its total loss function is: in, For strategic losses, For the loss of value, For policy entropy regularization, For graph aggregation constraints, , , These are the weighting coefficients.

9. The method according to claim 1, characterized in that, When the size of the drone swarm changes, by reconstructing the number of nodes in the underlying graph and the corresponding hierarchical aggregation relationships, the model parameters trained on the small-scale swarm can be directly transferred to the larger-scale swarm for use, thus achieving cross-scale parameter sharing.

10. The method according to claim 1, characterized in that, The number of layers in the fractal diagram structure is dynamically adjusted according to the size of the drone swarm.

11. The method according to claim 1, characterized in that, The features of the underlying nodes include position, velocity, heading, altitude, target direction, local obstacle information, boundary information, search status, and target confidence information.

Images