Unmanned aerial vehicle group cooperative movement method based on reinforcement learning in combination with bionic cluster

Abstract

The invention provides an unmanned aerial vehicle group cooperative movement method based on reinforcement learning in combination with a bionic cluster, and belongs to the field of unmanned aerial vehicle cooperative control and multi-agent systems. According to the method, a local interaction rule of an unmanned aerial vehicle group is constructed based on a Couzin-Leader model, and exclusion, attraction, perception and target tradeoff mechanisms are established; local observation modeling is carried out on the cooperative control process, a decision output space is constructed, and a profit feedback mechanism is designed; a multi-agent double-delay depth deterministic strategy gradient algorithm is adopted, an independent decision network is configured for each unmanned aerial vehicle, and target tradeoff parameters of guide nodes and sensing range parameters of following nodes are dynamically optimized, so that a self-adaptive cooperative control strategy is obtained; and deploying a training strategy to the unmanned aerial vehicle group to realize distributed cooperative movement of the unmanned aerial vehicle group in a dynamic and partially observable environment. The method can effectively reduce the group splitting risk, improves the group connectivity and the target arrival rate, and is suitable for disaster rescue, cluster logistics and other scenes.

Description

Technical Field

[0001] This invention belongs to the technical field of UAV cooperative control and multi-agent systems, specifically relating to a UAV swarm cooperative movement method based on reinforcement learning combined with biomimetic swarms, applicable to the cooperative control of UAV swarms in application scenarios such as distributed navigation, target tracking, material delivery, and disaster relief. Background Technology

[0002] With the rapid development of drone technology, multi-drone collaborative systems have shown broad application prospects in fields such as military reconnaissance, logistics delivery, disaster relief, and environmental monitoring. Through distributed collaboration mechanisms, drone swarms can complete complex tasks that are difficult for a single drone to accomplish without relying on a central control node, thereby significantly improving mission execution efficiency and system robustness.

[0003] However, in practical applications, UAV swarms still face many key technical challenges in cooperative movement, especially in dynamically changing environments with only locally observable information. On the one hand, the swarm needs to maintain a stable connectivity structure to avoid disintegration or splitting due to communication limitations, environmental interference, or individual behavioral differences. On the other hand, the swarm also needs to efficiently guide itself towards the target area while maintaining cohesion to meet mission completion requirements. Existing swarm cooperative control methods, such as schemes based on artificial potential fields, consensus control, or traditional optimization control, typically rely on global information acquisition, fixed interaction rules, or complex parameter tuning. These methods struggle to achieve stable and reliable swarm cooperative behavior when faced with environmental uncertainties, scalability, and partially observable constraints.

[0004] Bionic swarming theory offers important insights into multi-agent collaboration. Numerous studies have demonstrated that natural groups such as flocks of birds and schools of fish can achieve highly coordinated collective movement relying solely on local information exchange. The Cousin-Leader model, by introducing a small number of goal-aware guide nodes (informed agents), simulates the collective phenomenon where a key individual leads the entire group towards a goal. Based on local rules such as repulsion, attraction, and alignment, this model imbues the group with a certain goal orientation while maintaining group cohesion. However, such bionic models typically rely on pre-defined behavioral parameters and fixed interaction mechanisms, lacking adaptive adjustment capabilities in complex dynamic environments and struggling to perform online optimization based on task changes and environmental disturbances.

[0005] Meanwhile, deep reinforcement learning, especially multi-agent reinforcement learning, has provided a new technical path for the adaptive cooperative control of UAV swarms. Through continuous interaction with the environment, agents can autonomously learn cooperative strategies based on reward signals, thereby forming a robust distributed decision-making mechanism in unknown or dynamic environments. However, directly applying reinforcement learning to large-scale UAV swarms still faces problems such as the curse of dimensionality, training instability, degradation of cooperative strategies, and excessive homogenization, which limit its engineering application in complex swarm systems.

[0006] Therefore, how to introduce the adaptive optimization capability of multi-agent reinforcement learning while maintaining the natural cohesive characteristics of the biomimetic swarm model, and construct a UAV swarm cooperative control method and system that can both maintain the swarm connectivity structure and guide the swarm to efficiently approach the target, has become a key technical problem that urgently needs to be solved in the field of UAV swarm cooperative control. Summary of the Invention

[0007] This invention aims to address the problems of group splitting, insufficient target-oriented ability, and strong parameter dependence that existing UAV swarms easily encounter when moving collaboratively in dynamic and partially observable environments. It proposes a UAV swarm collaborative movement method and system based on reinforcement learning combined with biomimetic swarms to improve the swarm's adaptive collaborative ability, group cohesion, and overall target arrival efficiency in complex environments.

[0008] Firstly, this invention provides a method for cooperative movement of unmanned aerial vehicle (UAV) swarms based on reinforcement learning combined with biomimetic clustering, comprising the following steps:

[0009] Step S11: Obtain the scene parameters and target settings for the UAV swarm cooperative movement task. The scene parameters include swarm size, guide node ratio, target location, and two-dimensional motion environment range. Construct individual behavior rules for the UAV swarm based on the Cousin-Leader biomimetic swarm model, and define the repulsion radius of each individual. attraction radius Perception range and the target trade-off parameters of the bootstrap node This enables collision avoidance, orientation alignment, and group cohesion among individuals, relying solely on local neighbor information.

[0010] Step S12: Construct a cooperative control model based on the multi-agent dual-delay deep deterministic policy gradient (MATD3) algorithm, configuring an independent policy mapping unit (Actor network) and a value evaluation unit (Critic network) for each agent; wherein, the Actor network outputs its control parameters based on the agent's local observation state: when the agent is the guiding node, it outputs the target-oriented balance parameters. When the agent is a follower node, output the perception range. The local perception field of view of the guiding node is The Critic network is used to evaluate the value function of state-action pairs to guide the policy optimization of the Actor network.

[0011] Step S13: Based on the benefit constraint mechanism, design a composite reward function, and train the collaborative control model based on the composite reward function until the model converges to obtain the trained collaborative control strategy; the composite reward function comprehensively considers the group's goal attainment performance, the stability of the group's connectivity structure, and the penalty term for isolated individuals, thereby simultaneously promoting goal convergence and group cohesion.

[0012] Step S14: Deploy the trained cooperative control strategy to the UAV swarm to achieve adaptive, highly cohesive cooperative navigation in dynamic, partially observable environments;

[0013] Furthermore, in the individual behavior rules constructed based on the Cousin-Leader model, the agent... At any moment direction of movement Update as follows:

[0014] When the neighbor is excluded When not empty, a weighted approach is used to adjust the direction to avoid collisions:

[0015]

[0016] in, For time step, Retain weighting coefficients for direction;

[0017] When the repulsive neighbor set is empty and the attractive neighbor set is empty When not empty, the following node aligns with the neighbor's position direction and velocity direction:

[0018]

[0019] When the intelligent agent When guiding a node, introduce the target direction into its direction of motion. The correction direction is obtained:

[0020]

[0021] Where the equilibrium parameter Dynamically output by the Actor network. It is used to balance group following behavior with goal orientation.

[0022] Furthermore, the cooperative control model adopts the MATD3 algorithm framework, whose core mechanism includes:

[0023] Dual Critic Network Structure: Two independent Critic networks are set up for each agent. , And in the target value calculation, the smaller of the two values ​​is taken to construct the target. This reduces the value, thereby suppressing overestimation of the value function;

[0024] Target policy smoothing: in calculating the target When the value is reached, truncated Gaussian noise is superimposed on the output of the target Actor network. The target action is smoothed to improve the stability of the value function learning.

[0025] Delayed policy update: The update frequency of the Actor network is set to be lower than that of the Critic network, so that the policy update is based on relatively stable value estimation results, thereby improving the convergence and reliability of the training process.

[0026] Furthermore, the composite reward function Defined as:

[0027]

[0028] in: This represents the proportion of agents that have reached the target area, used to measure the group's goal achievement rate. Group connectivity is used to characterize the degree of internal connectivity within a subgroup containing the bootstrap node; This represents the number of isolated agents without neighbors within the current sensing range. The total size of the group; These are adjustable positive weighting coefficients, used to weigh the importance of target convergence, group connectivity, and split inhibition, respectively.

[0029] Secondly, the present invention further provides a collaborative control system for unmanned aerial vehicle swarms based on reinforcement learning combined with biomimetic swarms, for implementing the aforementioned method, comprising:

[0030] The parameter acquisition module is used to acquire scene parameters and target settings for the UAV swarm cooperative movement mission. The scene parameters include at least the swarm size, the proportion of guiding nodes, the target position, and the boundary of the two-dimensional motion environment, providing an initial configuration for swarm modeling and training.

[0031] The behavior modeling module is used to construct individual behavior rules for UAV swarms based on the Cousin-Leader biomimetic swarm model. The rules include exclusion interaction, attraction interaction, local perception mechanism, and goal-oriented balancing mechanism for guiding nodes, so as to achieve collision avoidance, alignment and group cohesion under the condition of relying only on local neighbor information.

[0032] The reinforcement learning control module is used to construct and train a cooperative control model based on the multi-agent dual-delay deep deterministic policy gradient (MATD3) algorithm. The model sets up independent Actor and Critic networks for each agent, where the Actor network outputs control parameters based on local observations (the guiding node outputs target equilibrium parameters). Follow the node output sensing range ), the Critic network is used to evaluate the value of state-action pairs to guide policy optimization;

[0033] The strategy deployment module is used to load the trained cooperative control strategy into each agent of the UAV swarm. During the actual operation phase, it makes distributed decisions based on local perception information to achieve adaptive cooperative navigation in a dynamic, partially observable environment.

[0034] The task management and evaluation module is used to manage and evaluate the performance of simulation or actual operation processes, including: resetting the task environment, calculating the composite reward function, statistically analyzing group performance indicators, and determining the task completion status; among which, the group connectivity indicator is defined as... Total arrival rate is defined as It is used to quantitatively evaluate group collaborative behavior from two dimensions: structural stability and task completion.

[0035] Compared with the prior art, the present invention has at least the following beneficial effects:

[0036] The synergistic integration of biomimetic rules and learning mechanisms significantly enhances adaptive capabilities: This invention couples the local interaction mechanism of the Cousin-Leader biomimetic swarm model with the policy learning capability of the multi-agent dual-delay deep deterministic policy gradient (MATD3) algorithm: the guiding node adaptively outputs the target equilibrium parameters through the Actor network. Dynamically balancing goal orientation and herd following; following nodes adaptively adjust their perception range through the Actor network. Therefore, without the need for global information or complex manual rule configuration, a group can achieve stable and efficient cooperative movement in a dynamic, partially observable environment.

[0037] Effectively suppressing group splitting and enhancing group cohesion: By explicitly introducing a group connectivity term and an isolated individual penalty term into the reward feedback mechanism, and combining it with learned adaptive control parameters, this invention can continuously maintain a high level of group connectivity during movement. This invention significantly reduces the risk of group splits caused by individuals leaving the group. Simulation results show that, compared to traditional or improved Cousin-Leader models, this invention can reduce the group split rate by approximately 13%.

[0038] Improve goal achievement efficiency and guide nodes to make more rational decisions: guide nodes to dynamically adjust through learning mechanisms. This invention avoids the swarm becoming detached from the group due to excessive target orientation, and prevents navigation efficiency from being weakened by excessive swarm following, enabling the swarm to efficiently approach the target while maintaining cohesion. In multiple simulation missions with swarm sizes ranging from 50 to 200 UAVs, the overall swarm arrival rate achieved by this invention is [not specified]. The average performance reached 92%, which is about 6% higher than the comparison model, and the guiding effect of the guiding node on the following node is more stable and reliable.

[0039] It exhibits good scalability and robustness: This invention employs a distributed decision-making architecture that configures independent Actor and Critic networks for each agent, enhancing the diversity and generalization ability of strategies. Even when conditions such as increased group size and environmental complexity change, the system can still maintain high connectivity. With arrival rate It demonstrates good scalability and resistance to environmental disturbances.

[0040] A general swarm collaborative control framework is formed, which is easy to transfer to engineering: The method of this invention does not rely on a specific UAV dynamics model. The effectiveness of the core algorithm has been verified in a two-dimensional abstract environment, and it can be directly extended to three-dimensional simulation platforms (such as Gazebo and AirSim) or actual UAV systems. This framework is applicable to various UAV swarm application scenarios such as distributed navigation, target tracking, material delivery, and post-disaster collaborative search, providing a general and reusable technical solution for engineering deployment. Attached Figure Description

[0041] Figure 1 This is a flowchart of a method for cooperative movement of unmanned aerial vehicle swarms based on reinforcement learning combined with biomimetic swarms, provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the MATD3 algorithm framework (independent network architecture) used in the embodiments of the present invention;

[0043] Figure 3 This is a comparison graph showing the change of population connectivity over time between different population sizes for the embodiments and comparative models of the present invention.

[0044] Figure 4 This is a comparison chart of the average group arrival rates of the embodiments of the present invention and the comparative model under different group sizes;

[0045] Figure 5 This is a histogram showing the distribution of population arrival rate in 200 repeated experiments according to an embodiment of the present invention. Detailed Implementation

[0046] The technical solution of the present invention will now be described in detail and completely with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0047] Example 1

[0048] like Figure 1 As shown, this embodiment provides a method for cooperative movement of unmanned aerial vehicle (UAV) swarms based on reinforcement learning combined with biomimetic swarms, including the following steps:

[0049] Step S101: Task scenario initialization.

[0050] Construct a two-dimensional task region Initialized within the region by A swarm of drones. According to a pre-set ratio. A subset of drones were selected from the group as guide nodes, and the remainder as follower nodes, where the proportion was... The initial positions of all drones are set in a sub-region on one side of the area (e.g., The initial velocity magnitude is set to (here) ), direction is random; target points are randomly generated within the area. As a destination for coordinated group movement.

[0051] Step S102: Construction and initialization of the collaborative control model.

[0052] like Figure 2 As shown, based on the Multi-Agent Dual-Delay Deep Deterministic Policy Gradient Algorithm (MATD3), an independent decision-making unit is constructed for each UAV in the swarm. Indicates the current state. This indicates the decision-making action to be taken in the current state. Indicates the state Take action below The instant reward value obtained, Indicates the next state. Indicates the target policy network in state The reference action for the output is below. This represents the minimum value among the outputs of the two objective value assessment networks. Each decision unit includes:

[0053] Actor Network Its network structure includes an input layer, hidden layers, and an output layer. Input layer: The input dimension can be 4N, where N is the population size; Hidden layers: Contain two fully connected layers. The first hidden layer contains 256 neurons, and the second hidden layer contains 128 neurons, each followed by a ReLU activation function; Output layer: One fully connected layer with 1 neuron, corresponding to the output control parameters, guiding the node's output. Follow the node output For output The Sigmoid activation function is used to constrain it to the range [0,1]; for the output It can be mapped to using a scaled Tanh activation function. Within a reasonable range.

[0054] Dual Critic Network Its network structure includes an input layer, hidden layers, and an output layer. The input layer receives joint observations and actions from all agents. In practice, each agent's observation and action are concatenated and used as part of the input. The hidden layer also uses a deep structure, consisting of two fully connected layers with 512 and 256 neurons respectively, activated by the ReLU function. The output layer contains one neuron, outputting a scalar value evaluation (Q-score) for the current joint state-action pair.

[0055] Target Network The structure is exactly the same as the corresponding main network, and the parameters are initialized using the same random seed to stabilize the training process.

[0056] Initialize network parameters and establish an experience replay pool. Used to store interactive samples.

[0057] Step S103: Local observation acquisition.

[0058] At time step Each intelligent agent Based on its own position and sensing range parameters (Following nodes are output from the Actor network, while the guiding node uses a preset, relatively large range) Determine the neighbor set. Its local observations are represented as follows:

[0059]

[0060] And uniformly fill to a fixed dimension to meet network input requirements.

[0061] Step S104: Adaptive action generation and execution.

[0062] Each intelligent agent will observe the measurement Input the corresponding Actor network to obtain the control input.

[0063]

[0064] During the training phase, exploration noise is added to the output action. Based on the control parameters, and combined with the Cousin-Leader biomimetic swarming rules, the motion direction is calculated and the position is updated to achieve cooperative movement based on local interaction.

[0065] Step S105: Reward calculation and experience storage.

[0066] Calculate instant rewards based on the current system status The reward function integrates group arrival rate, group connectivity, and isolated node penalty. The weighting coefficients of the composite reward function are... The values ​​and dynamic adjustment rules for arrival rate weight are as follows: The arrival rate weight is set to 100. When the entire group enters the target area, the environment provides a corresponding terminal reward based on the arrival rate. As connectivity changes dynamically, when At that time, the connectivity weights Setting it to 0.5 is used to maintain high connectivity but avoid over-rewarding stagnation; when At that time, the connectivity weights Setting it to -2 imposes a penalty on potential splitting risk; when If the task is deemed a failure, the training round is terminated early, and subsequent rewards are no longer calculated; isolation penalty weight. Setting it to 10 means that throughout the entire movement, if an isolated node loses communication with any of its neighbors, it will be continuously penalized according to this coefficient to prevent individuals from leaving the group. Interaction samples... Store in the experience replay pool .

[0067] Step S106: Model training and update.

[0068] When the number of samples in the replay pool meets the training requirements, perform a parameter update:

[0069] Critic Update: Calculating the target value using a bi-objective Critic network Minimize the mean squared error loss to update ;

[0070] Actor Delayed Updates: Updates the Actor network at a preset frequency. Perform policy gradient updates to maximize the value function;

[0071] Target network soft update: using coefficients The target network parameters are updated smoothly, whereby... .

[0072] Step S107: Strategy Deployment and Application.

[0073] Repeat steps S103 to S106 until training converges. Deploy the trained Actor network parameters to the UAV swarm system. During actual operation, each UAV can generate control parameters online by calling its own Actor network based solely on local observation information, achieving distributed, adaptive swarm cooperative movement.

[0074] Example 2

[0075] This embodiment provides a specific implementation and effect verification of the method in a simulation experiment.

[0076] Experimental setup:

[0077] Simulation platform: A two-dimensional discrete-time simulation environment is built based on Python and reinforcement learning libraries (such as PyTorch).

[0078] Group size: tested separately Four sizes.

[0079] Compare to baseline:

[0080] C-Model: An improved Cuzin-Leader model with fixed or heuristic tuning. parameter.

[0081] Fixed-ωModels: A series A model with values ​​fixed between 0.1 and 1.0 was used as a control.

[0082] Traditional methods: artificial potential field method, consensus algorithm.

[0083] Evaluation indicators:

[0084] Group connectivity : Used to characterize the tightness of connections between individuals within a subgroup containing the guide node. The closer the value is to 1, the more complete the group structure.

[0085] Total group arrival rate The percentage of agents that reached the target area at the end of the task.

[0086] Success rate: The proportion of experiments in which the population arrival rate exceeds 90% in multiple random initialization experiments.

[0087] Analysis of experimental results:

[0088] Group connectivity maintenance: such as Figure 3 As shown, at different scales, the swarm connectivity of the method of this invention (TD3-Model) can maintain a high level (>92.5%) throughout the entire movement process, and the decline trend over time is the slowest. Especially in a large swarm of 200 aircraft, its connectivity is significantly better than C-Model and all fixed models. The model demonstrates its superior resistance to splitting.

[0089] Overall group arrival rate: such as Figure 4 As shown, when N = 200, the overall arrival rate of the group using the method of the present invention (TD3-Model) can be maintained at a high level under different proportions of guiding nodes; even when the proportion of guiding nodes is low, its arrival rate is about 6% higher than that of C-Model.

[0090] Task success rate: such as Figure 5 As shown, the TD3-Model proposed in this invention exhibits a significantly higher task success rate than the comparative models. Extensive simulation experiments demonstrate that the group arrival rate is mainly concentrated in the interval [0.9, 1], proving a high task success rate.

[0091] Compared with traditional methods, TD3-Model significantly outperforms artificial potential field method and consensus algorithm in terms of arrival rate and formation stability, verifying the advantages of combining biomimetic rules and reinforcement learning in complex dynamic environments.

[0092] Algorithm effectiveness verification: By replacing the MATD3 algorithm with other reinforcement learning algorithms such as MADDPG and DDPG for comparative experiments, it was found that MATD3 has a faster convergence speed and more stable final performance, proving the effectiveness of MATD3's dual Critic, target smoothing and delayed update mechanism in this task.

[0093] in conclusion

[0094] Simulation results demonstrate that the proposed method effectively addresses the challenges of maintaining cohesion and achieving efficient navigation in collaborative movement of UAV swarms. By learning adaptive balancing strategies for guiding nodes and perception strategies for following nodes, the swarm can achieve highly cohesive, high-success-rate target-oriented motion relying solely on local information, providing an effective intelligent collaborative solution for practical UAV swarm applications.

[0095] Example 3

[0096] This embodiment provides a collaborative control system for UAV swarms based on reinforcement learning combined with biomimetic clustering. This system can be used to implement the methods described above. The system can be deployed at a ground control station or distributed and embedded in each UAV, and includes software modules and necessary hardware communication interfaces.

[0097] The system consists of the following functional units:

[0098] Parameter acquisition module: Used to read or set scene configuration parameters for collaborative tasks, including group size, proportion of guiding nodes, target location, and movement space boundaries, providing basic input for subsequent behavior modeling and strategy reasoning.

[0099] Behavior modeling module: Based on the Cousin-Leader bionic swarm model, establish individual interaction rules and construct repulsion, attraction, perception and goal trade-off mechanisms; the module receives environmental and role setting information from the parameter acquisition module and forms corresponding local motion update constraints according to the different attributes of the guiding node and the following node.

[0100] The reinforcement learning control module is used to load and run the offline-trained multi-agent dual-delay deep deterministic policy gradient (MATD3) model. This module configures independent Actor and Critic network structures for each UAV, supports centralized value evaluation during the training phase, and retains only the Actor networks of each UAV for distributed inference during the execution phase, thereby outputting the corresponding control parameters—the guide node outputs the target trade-off parameters. Follow the node to output the sensing range parameters .

[0101] The strategy deployment module combines the control parameters generated by the reinforcement learning control module with the biomimetic motion rules in the behavior modeling module to complete the update calculation of the UAV's motion direction and position at the next moment, and converts the results into flight control commands to drive the group to perform coordinated movement.

[0102] Task Management and Evaluation Module: Used for scheduling simulated or actual task flows, including task initialization, runtime status monitoring, and termination condition determination; simultaneously, it calculates group performance metrics online, such as group connectivity. Total arrival rate of the group And record and analyze key data during the operation.

[0103] During the training or validation phase, each module can achieve necessary information exchange through communication interfaces. For example, during centralized learning, it can provide joint state information to the Critic network. During the actual deployment phase, each UAV can only call its own Actor network for inference based on local perception information, thereby achieving distributed collaborative control.

[0104] During the algorithm development phase, the system verifies the collaborative decision-making algorithm through a closed-loop test environment based on computer simulation. The test environment includes verification of the control algorithm operation based on the software model and real-time interactive verification based on the actual flight control computing unit. After completing the above verification, the decision engine is deployed to the flight control computer of a real UAV, and the collaborative motion demonstration and verification of a physical UAV swarm is realized by integrating actual navigation sensors and obstacle avoidance sensors.

[0105] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for cooperative mobile operation of unmanned aerial vehicle (UAV) swarms based on reinforcement learning combined with biomimetic swarming, characterized in that, The technical steps include the following: Step S11: Construct a scenario description for the UAV swarm cooperative movement mission. The scenario description includes at least the swarm size, the proportion of guiding nodes, the target area location, and the flight space boundary. Under the constraints of the scenario description, introduce a biomimetic swarm dynamics model to model the local interaction mechanism of individual UAVs. The interaction mechanism includes at least repulsion constraints, attraction constraints, perception range adjustment, and trade-off parameters for balancing swarm convergence and target guidance, thereby forming basic rules for swarm movement under the condition of relying only on neighborhood information. Step S12: Based on the basic rules of the group movement, establish a multi-agent adaptive decision-making framework, and configure a learning structure including a policy mapping unit and a value evaluation unit for each UAV. The policy mapping unit outputs continuous control quantities based on the local observation information of the UAV. The control quantities represent the target orientation trade-off parameters in the guiding nodes and the individual perception range parameters in the non-guiding nodes. The value evaluation unit is used to measure the contribution of individual behavior to the group's collaborative effect. Step S13: Construct a benefit constraint mechanism for group cooperative mobility tasks. The benefit constraint mechanism simultaneously characterizes the group's target approach, the group's structural connectivity, and the penalty factor for isolated individuals. By embedding the benefit constraint mechanism into the multi-agent adaptive decision-making framework, the policy mapping unit and the value evaluation unit are jointly optimized, so that the policy mapping unit and the value evaluation unit of each UAV converge collaboratively during the interaction process, thereby forming a stable cooperative behavior pattern. Step S14: After completing the joint optimization, the obtained swarm cooperative control strategy is embedded into the UAV execution system. During the actual operation phase, each UAV only calls the corresponding strategy mapping unit to output control quantity based on local perception information, and completes motion update in combination with the biomimetic swarm dynamics model, thereby realizing adaptive cooperative navigation of UAV swarm in dynamic and partially observable environment.

2. The method for cooperative mobile unmanned aerial vehicle swarms based on reinforcement learning combined with biomimetic swarms according to claim 1, characterized in that, Before introducing the Cousin-Leader biomimetic swarm dynamics model, the method first formalizes the UAV swarm cooperative movement task as a partially observable Markov decision process, and its modeling methods include: Each UAV in the swarm is treated as an independent decision-making entity. Under limited perception conditions, it makes decisions based solely on its own state and local neighborhood information. The observations of each UAV include at least its own position and velocity vectors, as well as the position and velocity distribution of neighboring UAVs within its perception range. The control input is limited to continuous parameter form rather than discrete maneuver commands. The control variables of the guiding nodes are defined as balance parameters used to adjust the target guidance intensity, and the control variables of the non-guiding nodes are defined as perception range parameters used to adjust the neighborhood interaction scale. Furthermore, the state evolution process of the system is embedded into the Cousin-Leader biomimetic swarm dynamics model, so that the position update of each UAV at any time is jointly determined by its role attributes, local neighbor distribution and continuous control parameters, thus forming a state transition mechanism that satisfies some observable characteristics.

3. The method for cooperative mobile unmanned aerial vehicle swarms based on reinforcement learning combined with biomimetic swarms as described in claim 2, characterized in that, In the state transition mechanism, the motion updates of each UAV follow the improved Cousin-Leader biomimetic interaction mechanism, and its individual motion update rules are defined as follows: Let the first An intelligent agent at time The position vector is The unit vector of the direction of motion is The amplitude of the flight speed is constant. Based on neighborhood distance relationships, the local interaction objects are divided into repulsive neighborhoods and attractive neighborhoods, defined as follows: in, For the repulsion radius, For the attraction radius; when the repulsive neighborhood... When not empty, the agent prioritizes avoidance adjustment, and its direction vector is modified according to a weighted form with a history preservation term: in, For time step, This is the direction-maintaining coefficient, used to balance the original motion trend with the neighborhood repulsion effect; when Empty and attracting neighbors When not empty, ordinary follower nodes coordinate their adjustments based on a combination of information regarding the relative positions of their neighbors and their movement trends. Their direction updates are as follows: When the intelligent agent When identified as a guiding node, a reference direction vector pointing to the task objective is introduced in addition to the aforementioned group interaction directions. And based on the balance parameters output by the strategy mapping unit described in step S12 Target fusion is performed to obtain the correction direction: Based on this, the position iterative update is completed: for the guide node... For non-boot nodes, in, and Representing intelligent agents respectively At any moment and Spatial location.

4. The method for cooperative mobile unmanned aerial vehicle swarms based on reinforcement learning combined with biomimetic swarms according to claim 2, characterized in that, The cooperative mobile task of the drone swarm is characterized as a partially observable sequential decision-making process, in which each agent makes decisions based only on local perception information. Specific modeling methods include: Local observation modeling, for any intelligent agent The observation information it obtains at the current moment consists of its own state and the motion states of its neighbors within its perception range. The local observation is represented in matrix form: in and Representing intelligent agents respectively The spatial position and velocity components are represented in one row, while the remaining rows correspond to the state information of neighboring agents within its perception range. When the number of neighbors is insufficient, the observation matrix is ​​expanded to a unified dimension using zero-padding. ,in For the size parameters of the drone swarm; The decision output space, wherein the decision process does not directly generate motion control commands, but outputs continuous parameters for adjusting group interaction rules, wherein: in, This represents the trade-off parameter used by the guiding node to balance goal orientation and group consistency. This represents the local perception field of view of non-guide nodes, while the local perception field of view of guide nodes is... The trade-off parameters and the local perception view range are both adaptively generated by the corresponding strategy mapping unit based on local observations. The reward feedback mechanism, targeting the overall collaborative behavior of the group, constructs a global feedback signal decoupled from individual control parameters, determines the reward function, and guides the joint optimization of multi-agent policies. The reward function is defined as follows: in, This indicates the proportion of agents that have reached the target area. This represents the connectivity level of the group at the current moment. This indicates the number of isolated agents that have not detected any neighbors. For the total size of the group, These are weighting coefficients used to adjust the relative importance of different performance indicators; By adopting the above method, a unified decision modeling of the cooperative movement behavior of UAV swarms can be achieved by relying only on local observable information, and a constraint basis can be provided for subsequent policy training based on multi-agent reinforcement learning.

5. The method for cooperative mobile unmanned aerial vehicle swarms based on reinforcement learning combined with biomimetic swarms according to claim 4, characterized in that, The cooperative control strategy employs a multi-agent, dual-delay deterministic strategy optimization structure, and models each UAV separately as an independent decision-making unit. Each agent is configured with the following neural network components: two sets of master value evaluation networks. Two sets of target value assessment networks A set of master policy networks and a set of target policy networks ; During the training phase, the value evaluation network updates itself by minimizing the deviation between the predicted and target values ​​based on empirical replay data. Its loss function is defined as: in, For experience replay pool, This is the current state. The decision to be made at the present moment. In the state Take action below The feedback reward value obtained, For the next state, the target item It is given by the following formula: To introduce truncated random perturbation terms, This is used to suppress excessive bias in value estimation. This indicates that the function value will be truncated to... Within the range, here Represents the lower and upper bounds of the decision output space; The main policy network updates its parameters by performing reverse optimization on the expected output of the current value evaluation function. Its objective function is: To maintain numerical stability during training, the parameters of the target value evaluation network and the target policy network are updated recursively from the main network parameters using a smooth iterative method. The update rule is as follows: in Preset update coefficients; By optimizing the multi-agent dual-delay deterministic strategy structure, each UAV can achieve stable convergence of the multi-agent cooperative strategy without sharing control variables and relying solely on local observations and parameterized action outputs.

6. The method for cooperative mobile unmanned aerial vehicle swarms based on reinforcement learning combined with biomimetic swarms according to claim 5, characterized in that, The cooperative control strategy introduces an update constraint mechanism during training to suppress policy instability and value function bias. This mechanism includes at least the following: Target policy perturbation constraints, when constructing the target value term, are applied to the target policy network. The output of introduces amplitude-limited random perturbations to form a smoothed reference motion, expressed as: By locally perturbating the reference action, the variance fluctuation of the target value during the estimation process is reduced, thereby mitigating the numerical instability caused by the output of a single strategy. The separate parameter update constraint sets different parameter refresh rates for the policy network and the value assessment network. This allows the value assessment module to prioritize fitting the current experience distribution and then adjust the policy parameters based on its output. This avoids the instability caused by prematurely modifying the control policy before the value function has converged. The dual-channel value pruning mechanism provides evaluation results for the same state-action pair from two independently constructed value evaluation networks, and selects the smaller value as the reference output when calculating the target item. This provides a lower bound constraint on the risk of overestimation, reducing the possibility of systematic bias in the value function during iteration. Through the target policy perturbation constraint, the separate parameter update constraint, and the dual-channel value pruning mechanism, each UAV can achieve stable learning and consistent convergence of the multi-agent cooperative strategy based solely on local observations and outputting parameterized control quantities.

7. The method for cooperative mobile unmanned aerial vehicle swarms based on reinforcement learning combined with biomimetic swarms according to claim 1, characterized in that, The method also includes parameter initialization rules for policy verification and a group performance quantification evaluation step, specifically including: The construction of the initial state of the population in a two-dimensional continuous space The initial distribution state of the drone swarm is generated internally, in which the spatial position of each agent is determined by random sampling and the initial motion direction is set independently and randomly. Differentiated constraints are applied to the flight parameters of different roles, so that the velocity modulus of the guiding node is set to a preset ratio value of the non-guiding node, thereby forming a heterogeneous group structure, which is used to evaluate the adaptability of the cooperative control strategy under heterogeneous conditions. Group connectivity metric, which uses a subgroup containing at least one bootstrap node as the effective unit of cooperation, defines the time interval. The group connectivity is: in, This represents the set of connected cluster indices containing the bootstrap node at the current time. Abstracting drone swarms as undirected graphs , where the set of nodes This represents all the agents in the group. Let represent the set of edges. If the Euclidean distance between any two agents does not exceed their attraction radius threshold, then an undirected edge is established between them. Based on this adjacency relationship, the graph is... Connectivity analysis can divide the entire population into several non-overlapping connected clusters; Indicates the first The number of agents contained in a connected cluster. The total size of the group. The group connectivity is used to characterize the group's ability to maintain overall coordination during movement. The goal completion metric uses the proportion of agents reaching the target area as the global completion rate, defining the overall group arrival rate as: in, and These represent the number of guiding nodes and non-guiding nodes that arrived at the target area at the time the task terminated, respectively. The total group size; the total group arrival rate is used to measure the overall effectiveness of the collaborative control strategy in achieving target convergence while maintaining group consistency. By using the aforementioned initialization rules and evaluation metrics, a unified performance evaluation is conducted on the cooperative movement process of UAV swarms based on parameterized control and local interaction mechanisms, thereby verifying the effectiveness and stability of the proposed method in dynamic environments.

8. A collaborative control system for unmanned aerial vehicle (UAV) swarms based on reinforcement learning combined with biomimetic swarming, characterized in that, The system is configured to perform all the steps of the method as described in any one of claims 1-7, and accordingly includes: The environmental parameter and target setting module is used to perform the scene description construction described in step S11, and to obtain and define the group size, the proportion of guiding nodes, the target area location and the flight space boundary. The biomimetic dynamics and adaptive decision-making module integrates the functions of execution steps S11 to S13, specifically including: A behavior rule calculation submodule, which embeds the aforementioned biomimetic swarm dynamics rules, is used to calculate the basic motion direction based on local neighbor information and individual roles; A policy network submodule, which is set independently for each agent, is used to implement step S12, outputting corresponding control parameters based on local observations and the role type of the agent; A value network and optimization submodule employs the multi-agent dual-delay deep deterministic policy gradient algorithm and performs joint optimization training on the parameters of the policy network submodule based on the reward function determined by step S13 and the aforementioned profit constraint mechanism. The collaborative strategy execution module is used to deploy the trained policy network sub-module to each UAV, so that the UAV relies only on local perception during operation. The module calls the policy network to output control parameters in real time and hands them over to the behavior rule calculation sub-module to complete the final motion calculation. The task management and performance evaluation module is used to implement the parameter initialization rules and the group performance quantification evaluation process. It is responsible for initializing the simulation environment, resetting the task state, and calculating the overall performance indicators based on the group evolution state, including group connectivity and total group arrival rate.

Images