Multi-agent depth deterministic strategy gradient method based on attention mechanism

Abstract

The invention relates to the technical field of unmanned aerial vehicle control, in particular to a multi-agent depth deterministic strategy gradient method based on an attention mechanism, and the method comprises the steps: analyzing the motion state of an unmanned aerial vehicle cluster, and building a dynamic model; multiple attention mechanisms are fused, an unmanned aerial vehicle cluster formation model is constructed in combination with a kinetic model, and order parameters are defined according to an unmanned aerial vehicle cluster; multiple agents are adopted to perform formation control of the unmanned aerial vehicle cluster through deep reinforcement learning, rewards corresponding to the unmanned aerial vehicles are determined, and the rewards and order parameters are synthesized to train an unmanned aerial vehicle cluster formation model; and verifying the feasibility of the trained unmanned aerial vehicle cluster formation model. The unmanned aerial vehicle cluster can synchronously integrate local information and task information, flexible self-adaption to a complex environment is achieved, and the generalization ability of the unmanned aerial vehicle cluster in the complex environment is improved; through the synergistic effect of multiple attention mechanisms, the problems of information redundancy, insufficient generalization and the like faced by a multi-agent system can be effectively relieved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and more specifically to a multi-agent deep deterministic policy gradient method based on an attention mechanism. Background Technology

[0002] With the rapid development of drone technology, individual drones, limited by their own hardware and software constraints, can no longer meet the increasingly complex application scenarios. Furthermore, with the widespread application of drone swarms in disaster relief, area search, and target tracking, achieving efficient and stable swarm collaboration and formation control has become a core challenge in multi-agent system research. Traditional formation methods such as the Leader-Follower method, virtual structure method, consensus theory method, behavioral model method, and artificial potential field method suffer from rapidly increasing model complexity and computational consumption in large-scale swarms, making them unsuitable for practical needs and severely restricting the application of drone swarms.

[0003] To address the coordination and control challenges of traditional formation methods in large-scale cluster conditions, a self-organizing control method is proposed. Drawing inspiration from biological group behavior, this method employs a decentralized self-organizing control strategy, enabling each UAV to achieve efficient and stable formation control at the cluster level by adhering to simple local interaction rules. However, in complex tasks such as dynamic obstacle avoidance, multi-target tracking and group collaboration, heterogeneous cluster collaboration, and complex formation transformations, simple interaction rules alone are insufficient. For example, in obstacle avoidance in dynamic environments, traditional rules struggle to effectively perceive and respond to environmental changes, easily leading to collisions or formation loss of control. In multi-target tracking or task grouping scenarios, local rules are inadequate for efficient task allocation and flexible reorganization. Therefore, to handle complex tasks, this method introduces additional control laws and behavioral rules, resulting in a rapid increase in the dimensionality of control parameters such as weights, thresholds, and priorities. This not only exacerbates the difficulty of parameter tuning and maintenance but may also lead to parameter coupling, reducing the cluster's collaborative performance and system robustness. Furthermore, the high-dimensional parameter space significantly increases the demand for computing power and communication resources, further restricting the system's scalability.

[0004] Furthermore, a deep reinforcement learning method was proposed, which brought a new breakthrough to the control of UAV swarms with its powerful policy learning capability. It improved the stability and cooperation of multi-agent systems by introducing global information. However, early deep reinforcement learning algorithms were usually applied to single UAVs and then migrated to multi-UAV environments. In multi-UAV environments, the policies of all UAVs are updated simultaneously, leading to dynamic changes in the environment and thus causing non-stationarity of the state space.

[0005] Based on traditional deep reinforcement learning algorithms, the best performing algorithms are MADDPG (Multi-Agent Deep Deterministic Policy Gradient) and MAPPO (Multi-Agent Proximal Policy Optimization), especially MADDPG, which is based on the "centralized training, distributed execution" framework. Although both algorithms have made some progress in improving system scalability and training efficiency, they still have certain limitations and challenges in large-scale agent cluster applications: First, due to the multi-agent game characteristics of the environment, when the number of agents increases and the interaction relationships become more complex, both MADDPG and MAPPO algorithms are prone to falling into suboptimal Nash equilibrium. In addition, as the cluster size increases, MADDPG and MAPPO also face the problems of insufficient generalization ability and policy overfitting. Summary of the Invention

[0006] To address the technical problems of existing deep reinforcement learning methods based on drone swarms, such as susceptibility to inferior Nash equilibrium and insufficient generalization ability, this invention aims to provide a multi-agent deep deterministic policy gradient method based on an attention mechanism. The specific technical solution adopted is as follows:

[0007] Analyze the motion state of the drone swarm and establish a dynamic model;

[0008] A drone swarm formation model is constructed by integrating multiple attention mechanisms and combining them with a dynamic model, and the order parameters are defined according to the drone swarm definition.

[0009] A multi-agent approach is adopted to control the formation of a drone swarm through deep reinforcement learning, determine the corresponding reward for each drone, and train the drone swarm formation model by combining the reward and the order parameters.

[0010] Verify the feasibility of the trained drone swarm formation model.

[0011] Preferably, the motion state of the drone swarm is analyzed, and a dynamic model is established, including:

[0012] Based on the motion state of the drone swarm, the velocity components of the drones in the horizontal and vertical planes are determined, and the motion direction angle of the drones is obtained.

[0013] The spatial position of all drones in the drone swarm is determined by analyzing the velocity components, and the drone velocity is updated by combining the spatial drag.

[0014] Preferably, the multiple attention mechanism includes a bottom-up attention mechanism and a top-down attention mechanism, wherein the bottom-up attention mechanism includes cohesion rules, separation rules and alignment rules, and the top-down attention mechanism includes target consistency rules, global aggregation rules and clustering penalty rules.

[0015] Preferably, a multi-attention mechanism is integrated with a dynamic model to construct a drone swarm formation model, and the order parameters are defined according to the drone swarm, including:

[0016] The set of neighbors for any drone in a drone swarm is determined using a dynamic model;

[0017] The analysis integrates dynamic models, UAVs, and their corresponding neighbor sets. The Bottom-Up attention mechanism emphasizes the autonomous decision-making of individual UAVs based on local perception and neighborhood information. The Top-Down attention mechanism incorporates the global objective and the structural characteristics of the UAV swarm into the decision-making process of each UAV.

[0018] The order parameters are defined according to the drone cluster.

[0019] Preferably, the Bottom-Up attention mechanism emphasizes the autonomous decision-making of individual drones based on local perception and neighborhood information, including:

[0020] Based on the cohesion rule, the relative position vector of the centroid of the drone's neighbors is obtained by analyzing the drone and its corresponding neighbor set, thus obtaining the relative deviation distance of the drone.

[0021] Based on the separation rule, the analysis shows that the drone and its corresponding set of neighbors generate forces that point away from the neighbors.

[0022] Based on alignment rules, the drone adjusts its current direction of motion by analyzing the dynamic model through neighbor sets.

[0023] Preferably, a top-down attention mechanism is used to integrate the global target and the structural characteristics of the drone swarm into the decision-making process of each drone, including:

[0024] Based on the target consistency rule, the target motion direction vector of the UAV cluster is obtained, and the cosine value of the angle between the current motion direction of the UAV and the target motion direction is obtained by combining the dynamic model.

[0025] Based on global aggregation rules, the total number of drones in the drone cluster is counted, and the average distance in the drone cluster is determined by combining the drones and their corresponding neighbor sets.

[0026] Based on clustering penalty rules, drones in a drone swarm are clustered to obtain multiple clusters. These clusters are then analyzed to obtain cluster profile coefficients. Penalty coefficients and penalty thresholds are set, and the penalties for each drone are determined by combining the cluster profile coefficients, penalty coefficients, and penalty thresholds.

[0027] Preferably, the order parameters are defined according to the drone swarm, specifically as follows:

[0028] The activity area of ​​the drone swarm is divided into multiple grids, and the number of drones in each grid is counted. An order parameter is defined based on the number of grids.

[0029] Preferably, a multi-agent approach is used to perform formation control of the drone swarm through deep reinforcement learning, and to determine the corresponding rewards for the drones, specifically as follows:

[0030] When performing formation control of a drone swarm, the drones are defined as intelligent agents. The interaction between each intelligent agent and its neighboring intelligent agents is analyzed to make autonomous decisions. A reward and punishment mechanism is set up to determine the corresponding reward for the drone based on the autonomous decision-making process.

[0031] Preferably, the interaction between each agent and its neighboring agents is analyzed to make autonomous decisions, and a reward and punishment mechanism is set to determine the corresponding reward for the drone based on the autonomous decision-making process, including:

[0032] The spatial position and velocity of the drone are determined based on the feedback from the dynamic model, and the state space of the drone swarm is constructed.

[0033] Establish a motion space for the drone's actions;

[0034] The state space and action space together determine the deterministic transition;

[0035] The reward for the drone is determined by defining a reward component for each rule in the Bottom-Up and Top-Down attention mechanisms and using a reward weight coefficient to combine the reward components of each rule.

[0036] Preferably, the UAV swarm formation model is trained by combining rewards and order parameters, specifically as follows:

[0037] Each agent is configured with an Actor network and a Critic network. The Critic network outputs the reward corresponding to the drone. During the training process of the drone swarm formation model based on the reward and the order parameters, the Actor network adjusts the parameters of the drone swarm formation model by maximizing the reward output by the Critic network, and assists the order parameters in performing global joint evaluation.

[0038] The present invention has the following beneficial effects:

[0039] This application proposes a multi-agent deep deterministic policy gradient method based on attention mechanisms. It integrates multiple attention mechanisms to improve the MADDPG algorithm, enabling multi-agent swarms, specifically UAV swarms, to simultaneously integrate local and task information, achieving flexible adaptation to complex environments and enhancing the generalization ability of UAV swarms in complex environments. Furthermore, the synergistic effect of multiple attention mechanisms effectively alleviates the problems of information redundancy and insufficient generalization faced by multi-agent systems. Verification confirms that this method significantly improves convergence speed, training stability, and collaborative control capabilities in large-scale swarm environments. It provides a new approach for the autonomous collaboration and efficient control of multi-agent swarm intelligence. Future research can further combine graph neural networks, meta-learning, and other methods to explore higher-level adaptive and generalization mechanisms, promoting the application of multi-agent systems in a wider range of real-world scenarios. Attached Figure Description

[0040] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating the steps of a multi-agent deep deterministic policy gradient method based on an attention mechanism, as provided in an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the DA-MADDPG algorithm framework, a multi-agent deep deterministic policy gradient method based on an attention mechanism, provided in an embodiment of the present invention.

[0043] Figure 3 A schematic diagram of UAV spatial coordinates for a multi-agent deep deterministic policy gradient method based on an attention mechanism provided in an embodiment of the present invention;

[0044] Figure 4 A schematic diagram of the DA-MADDPG consensus convergence algorithm framework, a multi-agent deep deterministic policy gradient method based on an attention mechanism, provided in an embodiment of the present invention.

[0045] Figure 5 A comparison chart of the consistency convergence curves of a multi-agent deep deterministic policy gradient method based on an attention mechanism provided in an embodiment of the present invention;

[0046] Figure 6This is a comparison chart of the robustness curves of a multi-agent deep deterministic policy gradient method based on an attention mechanism provided in an embodiment of the present invention. Detailed Implementation

[0047] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a multi-agent deep deterministic policy gradient method based on an attention mechanism proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] The following description, in conjunction with the accompanying drawings, details the specific scheme of the multi-agent deep deterministic policy gradient method based on attention mechanism provided by this invention.

[0050] To better illustrate this, we will use a multi-agent deep reinforcement learning approach. First, the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm employs an "Actor-Critic" architecture and proposes the concept of "centralized training and distributed execution." During the training phase, the Critic network has access to global information containing the observations and actions of all agents (UAVs, Unmanned Aerial Vehicles). During execution, the agents rely only on local information for decision-making, thus addressing the policy instability problem caused by dynamic environmental changes. In the MADDPG algorithm, the Critic network input includes the joint actions of all agents, avoiding the policy overfitting problem in traditional independent Q-learning, and achieving good results in multi-agent target tracking problems. Based on this algorithm, existing technologies have proposed several improved algorithms. For example, the concept of parameter sharing is introduced into the MADDPG algorithm, which improves the algorithm's scalability and reduces training costs. Alternatively, the attention mechanism in deep learning is transferred to the MADDPG algorithm, enabling agents in the swarm to selectively focus on the information most relevant to their own decisions from other agents, rather than processing all information indiscriminately. This significantly reduces computational complexity and improves training efficiency. Furthermore, this research integrates the attention mechanism into the centralized Q-function of each agent, allowing each agent to dynamically allocate attention weights based on the similarity of its state and actions with other agents. This enhances the scalability and generalization ability of the multi-agent system in large-scale UAV swarms, enabling the algorithm to effectively achieve action coordination and formation control even when the number of UAVs is large.

[0051] Secondly, the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm expands the application boundaries of multi-agent systems. MAPPO employs an on-policy Proximal Policy Optimization (PPO) policy gradient method, which also follows the "Centralized Training with Decentralized Execution" (CTDE) framework. During the training phase, it utilizes global information to optimize the policies of each agent, while during the execution phase, it ensures that each agent makes decisions based solely on its local observations. The core advantage of this algorithm lies in introducing the PPO clip mechanism to limit the policy update magnitude, significantly improving the stability and convergence speed of the multi-agent training process, making it more robust in large-scale, multi-type agent collaborative and competitive environments. Furthermore, MAPPO uses an on-policy sampling method, simplifying the algorithm implementation and improving scalability. In large-scale UAV swarms and other multi-agent tasks, it can effectively handle high-dimensional state spaces and complex dynamic environments, demonstrating good generalization ability and high policy coordination efficiency.

[0052] However, both existing algorithms have certain drawbacks. First, they are prone to falling into suboptimal Nash equilibrium. During training, agents may converge to suboptimal policy combinations, leading to limited overall system performance and difficulty in achieving globally optimal cooperative or competitive results. This phenomenon is particularly prominent in large-scale cluster scenarios where cooperation and adversarial relationships are highly coupled, restricting the practical application value of the algorithms. Second, they suffer from insufficient generalization ability and policy overfitting. The MADDPG algorithm, due to its off-policy training and experience replay mechanism, tends to perform poorly in new environments or unseen agent combinations, resulting in limited generalization ability. While MAPPO improves training stability, its on-policy sampling method is highly dependent on environmental distribution, which may also lead to overfitting of policies to specific training environments, reducing adaptability in dynamic and complex environments. This results in cooperative action failures and policy transfer difficulties in real-world scenarios such as drone swarms, limiting the promotion and application of multi-agent reinforcement learning algorithms in large-scale, variable environments.

[0053] To address the problems of existing methods, this application proposes a multi-agent deep deterministic policy gradient method based on an attention mechanism, namely, the DA-MADDPG (Decentralized Actor-Multi-Agent Deep Deterministic Policy Gradient) algorithm, which is an improvement on the multi-agent deep deterministic policy gradient algorithm.

[0054] Please combine Figure 1 and Figure 2 It illustrates a flowchart of the steps and a schematic diagram of the DA-MADDPG algorithm framework for a multi-agent deep deterministic policy gradient method based on an attention mechanism provided in the first embodiment of the present invention. The method includes:

[0055] Step S1: Analyze the motion state of the drone swarm and establish a dynamic model;

[0056] Step S2: Integrate multiple attention mechanisms, combine with dynamic models to construct UAV swarm formation models, and define order parameters according to UAV swarm definitions;

[0057] Step S3: Use multi-agent deep reinforcement learning to control the formation of the UAV swarm, determine the corresponding reward for each UAV, and train the UAV swarm formation model by combining the reward and the order parameters.

[0058] Step S4: Verify the feasibility of the trained drone swarm formation model.

[0059] It should be noted that in this application, intelligent agent refers to unmanned aerial vehicle (UAV), that is, UAV with autonomous flight and intelligent decision-making capabilities. In this case, multiple intelligent agents correspond to UAV swarms to achieve more complex and efficient flight missions.

[0060] Further, step S1 includes:

[0061] Step S11: Based on the motion state of the drone swarm, determine the velocity components of the drones in the horizontal and vertical planes respectively, and obtain the motion direction angle of the drones.

[0062] Preferably, in this embodiment, since the UAV is a low-altitude, low-speed aircraft, physical details such as gravity, altitude, and air pressure are ignored, and the dynamic characteristics of the UAV are analyzed in a focused manner, which will not affect the state of the aircraft.

[0063] Please see Figure 3 Specifically, based on the motion state of any drone in the drone swarm, the velocity component of the drone in the horizontal plane is determined, denoted as... The velocity component in the vertical plane is denoted as... Then, the motion direction angle of the drone can be obtained, and the corresponding calculation formula is:

[0064]

[0065] in, , Both represent the direction angle of motion of the drone; This represents the velocity component of the drone in the vertical plane; This represents the velocity component of the drone in the horizontal plane; , These respectively represent the drones in shaft and Velocity components on the axis.

[0066] It can be explained that the horizontal plane is Plane; perpendicular plane is , flat.

[0067] Step S12: Analyze the velocity components to determine the spatial position of all drones in the drone swarm, and update the drone velocity by combining the spatial drag.

[0068] Specifically, first, the relationship between the various velocity components of the drone is determined, and the corresponding calculation formula is as follows:

[0069]

[0070] in, Indicates that drones are in Velocity components on the axis.

[0071] Next, the spatial location of the drone is determined, and the corresponding calculation formula is:

[0072]

[0073] in, Indicates that the drone is in Spatial coordinates at any given time; Indicates that the drone is in Spatial coordinates at any given time; Indicates the first time.

[0074] Finally, the speed update of the drone is determined, and the corresponding calculation formula is as follows:

[0075]

[0076] in, , , These represent the drones in the [number]th [month]. At that moment, axis, axis, Velocity components on the axis; Indicates the air drag coefficient; , , This represents the acceleration vector of the drone.

[0077] In particular, since drones are low-altitude, low-speed aircraft, their workspace can be considered isotropic, and therefore the air drag coefficient can be assumed to be... It has the same value in all directions.

[0078] Understandably, traditional drone swarm formation methods such as the Leader-Follower method, virtual structure method, consensus theory method, behavioral model method, and artificial potential field method, while achieving good drone formation in small-scale swarms, often experience exponential growth in model complexity in large-scale swarm environments. This leads to a significant increase in the system's computational and energy consumption requirements, making it difficult to meet real-time and scalability requirements. Furthermore, if a centralized or semi-centralized control architecture is used, robustness and scalability are severely challenged under conditions of communication constraints or node failures. Therefore, to achieve efficient, flexible, and robust formation control of large-scale drone swarms, a multi-attention mechanism is proposed.

[0079] Furthermore, the multiple attention mechanisms include a bottom-up attention mechanism and a top-down attention mechanism. The bottom-up attention mechanism includes cohesion rules, separation rules, and alignment rules, while the top-down attention mechanism includes goal consistency rules, global aggregation rules, and clustering penalty rules. In other words, a decentralized self-organizing control strategy that integrates bottom-up and top-down attention mechanisms is proposed to achieve efficient collaborative formation of large-scale UAV swarms.

[0080] Further, step S2 includes:

[0081] Step S21: Determine the neighbor set of any drone in the drone swarm using a dynamic model.

[0082] Specifically, assuming that the drone swarm being analyzed currently contains The space of each drone is recorded as follows: Speed ​​is denoted as In this embodiment, based on the first If we use a drone for analysis, then its neighbor set is:

[0083]

[0084] in, Indicates the first The set of neighbors corresponding to a drone; Indicates the first in the drone swarm A drone; , They represent the first frame and the first The spatial location of the drone; Indicates the sensing radius.

[0085] Step S22: Analyze the integrated dynamics model, UAVs, and their corresponding neighbor sets. Emphasize the autonomous decision-making of individual UAVs based on local perception and neighborhood information through the Bottom-Up attention mechanism; and integrate the global target and the structural characteristics of the UAV swarm into the decision-making process of each UAV through the Top-Down attention mechanism.

[0086] To explain, the Bottom-Up attention mechanism is typically represented by the three behavioral principles of cohesion, separation, and alignment in the Boids model. These principles collectively influence the spatial position changes of the drone. The drones are acquired in an orderly distribution. During training, each training iteration simultaneously calculates these three rules. Through these local interaction rules, each drone selects actions based solely on the state information of its neighbors, giving the cluster good self-organization and local robustness.

[0087] Furthermore, in step S22, the Bottom-Up attention mechanism emphasizes the autonomous decision-making of individual drones based on local perception and neighborhood information, including:

[0088] Step SA221: Based on the cohesion rule, analyze the drone and its corresponding neighbor set to obtain the relative position vector of the drone's neighbor centroids, and obtain the relative deviation distance of the drone.

[0089] Specifically, the corresponding calculation formula is:

[0090]

[0091] in, Indicates the first The relative deviation distance of the drone; Indicates the first The relative position vector of the centroids of the neighboring drones; Indicates the first The modulo operation of the neighbor set corresponding to the drone; , They represent the first frame and the first The spatial location of the drone; Indicates the distance between targets in the formation.

[0092] It can be explained that the target spacing in the formation It is a human-given rule that the cohesion rule means that each drone tends to move closer to the spatial center of its neighbors, that is, the centroid of the drone swarm, in order to maintain the integrity and compactness of the drone swarm. Based on the cohesion rule, it can prevent individuals in the drone swarm from leaving the swarm and ensure that the swarm will not disperse due to the movement of individuals.

[0093] Step SA222: Based on the separation rule, analyze the forces generated by the drone and its corresponding neighbor set that point away from the neighbor.

[0094] Specifically, the corresponding calculation formula is:

[0095]

[0096] in, Indicates the first The force generated by the drone pointing away from the neighbors; , They represent the first frame and the first The spatial location of the drone; Indicates the first The set of neighbors corresponding to a drone; It represents a very small positive number.

[0097] It can be explained that the smallest positive number Used to prevent the denominator from being zero or too small; based on the separation rule, it can ensure that each drone will actively keep its distance from its neighbors that are too close to prevent collisions; in actual operation, when the drones are close to each other, a force is generated that points away from the neighbors.

[0098] Step SA223: Based on the alignment rules, the UAV analyzes the dynamic model through the neighbor set and adjusts its current motion direction.

[0099] Specifically, the corresponding calculation formula is:

[0100]

[0101] in, Indicates the first The average velocity vector of the neighboring drone; Represents the alignment vector; , They represent the first frame and the first The speed of the drone; Indicates the first The modulo operation of the neighbor set corresponding to the drone.

[0102] It can be explained that the alignment rule refers to each drone referencing the movement direction and speed of its neighbors to adjust its own movement direction as closely as possible to its neighbors. This ensures the coordinated movement and overall directionality of drones in the drone swarm. During operation, the alignment vector is used to adjust the drone's own speed to approach the average speed of its neighbors.

[0103] Understandably, relying solely on the Bottom-Up attention mechanism is insufficient to fully address the needs of large-scale clusters in terms of global consistency, task orientation, and structural integrity. Therefore, the Top-Down attention mechanism is introduced to integrate global goals and overall structural characteristics into the decision-making process of each UAV. Specifically, goal consistency rules guide all UAVs toward a common mission objective; global aggregation rules promote overall cluster spatial convergence, improving global orderliness and stability; and clustering penalty rules suppress cluster splitting and fragmentation, ensuring the integrity of the cluster structure.

[0104] Furthermore, in step S22, the global target and the structural characteristics of the drone swarm are integrated into the decision-making process of each drone through a Top-Down attention mechanism, including:

[0105] Step SB221: Based on the target consistency rule, obtain the target motion direction vector of the UAV cluster, and combine it with the dynamic model to obtain the cosine value of the angle between the current motion direction of the UAV and the target motion direction.

[0106] Specifically, the corresponding calculation formula is:

[0107]

[0108] in, Indicates the current analysis of the first The cosine of the angle between the current direction of motion of the drone and the direction of motion of the target; Indicates the first The speed of the drone; This represents the expected target motion direction vector of the entire drone swarm; Indicates the first The speed modulus of the drone; This represents the magnitude of the target motion direction vector expected by the entire drone swarm.

[0109] It can be explained that the cosine value of the included angle The larger the result value, the higher the number of the current analysis. The more consistent the target movement direction of the individual drone and the drone swarm as a whole, the better.

[0110] Step SB222: Based on the global aggregation rules, count the total number of drones in the drone cluster, and determine the average distance in the drone cluster by combining the drones and their corresponding neighbor sets.

[0111] Specifically, the corresponding calculation formula is:

[0112]

[0113] in, This indicates the average distance of the drone swarm; Indicates the number of drones in a drone swarm; , They represent the first frame and the first The spatial location of the drone.

[0114] In other words, spatial aggregation degree measures the overall density of a drone swarm. It's a global statistic used to measure the spatial compactness of the entire swarm; by calculating the average distance between all drones, it characterizes the density of the swarm's overall distribution. In contrast, cohesion rules are local control mechanisms for drones, directly affecting individual motion decisions, while global aggregation rules are a global measure of the entire swarm's state, typically used to evaluate or monitor swarm performance. In other words, cohesion rules are the "method" for achieving swarm aggregation, while global aggregation rules are the "result" for measuring the aggregation effect.

[0115] Step SB223: Based on the clustering penalty rule, cluster the drones in the drone swarm to obtain multiple clusters, analyze them, obtain the cluster profile coefficients, set the penalty coefficients and penalty thresholds, and determine the corresponding penalty for the drones by combining the cluster profile coefficients, penalty coefficients and penalty thresholds.

[0116] Specifically, the corresponding calculation formula is:

[0117]

[0118]

[0119] in, Indicates punishment; Indicates the penalty coefficient; Represents the cluster profile coefficient; Indicates the penalty threshold; Indicates the number of drones in a drone swarm; Indicates the first A drone; Indicates the first Clustering profile coefficients corresponding to drones; Indicates the first The average distance from the drone to the nearest cluster among all remaining clusters; Indicates the first The average distance from one drone to all other drones in the same cluster; , They represent the first frame and the first Clusters corresponding to drones; Indicates the first frame and the first The distance of the drone.

[0120] It can be explained that the clustering penalty rule is used to suppress formation splitting of UAV swarms due to external disturbances during mission execution; the... Clustering profile coefficients corresponding to drones The closer it is to 1, the better it represents the first... A drone is closer to other drones in its own cluster and farther from other clusters, indicating a good clustering effect; a value close to 0 indicates that the first drone is closer to other drones in its own cluster and farther from other clusters. The drone is located at the boundary; a value less than 0 indicates that the first drone is at the boundary. The drone may have been misclassified.

[0121] The model demonstrates that the UAV swarm formation model fully integrates the local self-organizing characteristics of the Bottom-Up attention mechanism with the global target control of the Top-Down attention mechanism. This model not only possesses high scalability and robustness but also enables efficient collaboration and task-driven formation of large-scale swarms in dynamic and complex environments, providing a new theoretical foundation and methodological support for the autonomous collaborative control of multi-agent systems.

[0122] Step S23: Define the sequence parameters according to the drone cluster definition.

[0123] Understandably, in UAV swarm formation missions, the consistency of the swarm's motion state is fundamental to mission completion. To describe the degree of consistency in the motion state of a UAV swarm, the concept of an order parameter is introduced. This parameter describes the degree of order in a physical system. Its core definition is: during a continuous phase transition, the value of the order parameter gradually increases from zero (disordered phase) to a non-zero value (ordered phase), reflecting the system's degree of order and symmetry breaking. Optionally, the order parameter can be magnetization, density difference, macroscopic wave function, etc. A value of zero indicates that the system is in a high-symmetry, disordered state, while a non-zero value indicates that the system has entered a low-symmetry, ordered state.

[0124] To better illustrate this, we will conduct a study based on order parameters. First, we will obtain the speed consistency characteristic of the drone swarm, and the corresponding calculation formula is as follows:

[0125]

[0126] in, This indicates the degree of speed consistency among drone swarms; Indicates the number of drones in a drone swarm; Indicates the first The speed of the drone.

[0127] This illustrates the degree of speed consistency in a drone swarm. ,when When this occurs, it indicates that the drone swarm is in a completely disordered state; when At this time, it indicates that the drone swarm is in a completely ordered state; although the order parameter While it can characterize the speed consistency of a drone swarm, it cannot characterize the uniformity of the swarm's spatial distribution. Therefore, based on the drone swarm formation model in this application, the order parameter is redefined to measure the degree of orderliness of the swarm's spatial distribution.

[0128] Furthermore, in step S23, specifically:

[0129] The activity area of ​​the drone swarm is divided into multiple grids, and the number of drones in each grid is counted. An order parameter is defined based on the number of grids.

[0130] Specifically, the corresponding calculation formula is:

[0131]

[0132] in, Indicates the ordinal parameter; Indicates the number of grid cells; Indicates the first The number of drones in each grid; This represents the average number of drones across all grids.

[0133] It can be explained that the activity area based on the drone swarm is divided into One grid; order parameter When the ordinal parameter When the value is 0, it indicates that the drone swarm is evenly distributed throughout the activity area; when the sequence parameter is 0, it indicates that the drone swarm is evenly distributed throughout the activity area. A value of 1 indicates that the drone swarm is clustered in a single grid.

[0134] As an alternative implementation method, the state consistency of the swarm can also be analyzed by using order parameters, which are used in some condensed matter physics. From the perspective of condensed matter physics, changes in order parameters usually indicate that matter breaks through the critical state and undergoes a phase transition. Similarly, in the field of swarm intelligence, changes in order parameters can be regarded as the spontaneous emergence of intelligent behavior in the swarm. Therefore, the ordered state of the UAV swarm can be reflected by order parameters, and the state of the UAV swarm can be effectively controlled and regulated by adjusting the order parameters.

[0135] Furthermore, in step S3, multi-agent deep reinforcement learning is used to perform formation control of the drone swarm and determine the corresponding reward for each drone, specifically:

[0136] When performing formation control of a drone swarm, the drones are defined as intelligent agents. The interaction between each intelligent agent and its neighboring intelligent agents is analyzed to make autonomous decisions. A reward and punishment mechanism is set up to determine the corresponding reward for the drone based on the autonomous decision-making process.

[0137] To clarify, using multi-agent deep reinforcement learning for drone formation control can be considered a Markov decision process. During formation control, all drones are treated as agents with perception and decision-making capabilities. Each agent makes autonomous decisions based on its interactions with the environment and with other agents. In this process, each agent receives a reward value related to the target task for its decisions, and the agent tends to learn decisions that yield higher rewards.

[0138] Preferably, the Markov decision process is characterized as a quintuple, i.e. ,in, Representing the state space; Represents the action space; Indicates a deterministic transfer; Indicates a reward; This represents the discount factor.

[0139] Furthermore, the interaction between each agent and its neighboring agents is analyzed to make autonomous decisions, and a reward and punishment mechanism is set to determine the corresponding reward for the drone based on the autonomous decision-making process, including:

[0140] Step S31: Determine the spatial position and velocity of the UAV based on the feedback from the dynamic model, and construct the state space of the UAV swarm.

[0141] It can be explained that the state space contains global observation information for the entire UAV swarm. This observation information primarily includes two types of data: spatial position and velocity. The corresponding calculation formulas are as follows:

[0142]

[0143] in, Representing the state space; Indicates the spatial location of the drone; Indicates the speed of the drone; Indicates the number of drones in a drone swarm; This represents the dimension of the state space.

[0144] Step S32: Establish motion space for the drone's actions.

[0145] It should be noted that, since drones need to perform continuous actions, the action space is defined as a continuous real number space in this invention, and the corresponding calculation formula is as follows:

[0146]

[0147] in, Represents the action space; Indicates the actions of the drone; Indicates the number of drones in a drone swarm; This represents the dimension of the action space.

[0148] Step S33: Combine the state space and action space to determine the deterministic transition.

[0149] It can be explained that, The state transition function is determined by both system dynamics and environmental constraints. In this invention, it is denoted as a deterministic transition, and the corresponding calculation formula is:

[0150]

[0151] in, Indicates the first Real-time drone status; Represents a deterministic transfer function; , They represent the first The status and space of the drone at all times.

[0152] Step S34: Define a reward component for each rule in the Bottom-Up Attention Mechanism and Top-Down Attention Mechanism, and use a reward weight coefficient to combine the reward components of each rule to determine the reward for the drone.

[0153] It can be explained that the reward refers to the reward obtained by the drone during the learning process. The reward amount for each rule is determined based on the different rules; and in this embodiment, the reward is still based on the first rule. The drone was used for explanation.

[0154] Specifically, the reward component for the cohesion rule is calculated using the following formula:

[0155]

[0156] in, Indicates the first Cohesive rewards for drones; Indicates the first The relative position vector of the centroids of the neighboring drones; Indicates the first The spatial location of the drone; Indicates the distance between targets in the formation; Indicates the first The number of neighbors of the drone.

[0157] The penalty component of the separation rule is calculated using the following formula:

[0158]

[0159] in, Indicates the first Separation penalty for drones; Indicates an indicator function; Indicates the number of drones in a drone swarm; Indicates the first frame and the first The distance of the drone; Indicates the collision radius; This indicates a tiny error.

[0160] The reward component for alignment rules is calculated using the following formula:

[0161]

[0162] in, Indicates the first Alignment rewards for drones; Indicates the first The velocity vector of the drone; Indicates the first The average velocity vector of the neighboring drone; Indicates the first The number of neighbors of the drone.

[0163] The reward component for the goal consistency rule is calculated using the following formula:

[0164]

[0165] in, Indicates the first Rewards for achieving target consistency in drone deployment; Indicates the first The velocity vector of the drone; This represents the expected target motion direction vector of the entire drone swarm; Indicates the first The speed modulus of the drone; This represents the magnitude of the target motion direction vector expected by the entire drone swarm.

[0166] The reward component of the global aggregation rule is calculated using the following formula:

[0167]

[0168] in, Indicates the first Spatial aggregation reward for drones; This indicates the average distance of the drone swarm; Indicates the distance between targets in the formation; This indicates a tiny error.

[0169] The calculation formula for the penalty component corresponding to the clustering penalty rule is the same as that for the clustering penalty rule, and the clustering penalty is denoted as... , indicating the first Clustering penalty for drones.

[0170] Based on the reward component corresponding to each of the aforementioned rules, the reward for the currently analyzed drone is determined using the following formula:

[0171]

[0172] in, Indicates the first Rewards for each drone; , , , , , This represents the reward weight coefficient corresponding to the reward component of each rule.

[0173] Please see Figure 2 Furthermore, in step S3, the drone swarm formation model is trained by combining the reward and order parameters, specifically as follows:

[0174] Each agent is configured with an Actor network and a Critic network. The Critic network outputs the reward corresponding to the drone. During the training process of the drone swarm formation model based on the reward and the order parameters, the Actor network adjusts the parameters of the drone swarm formation model by maximizing the reward output by the Critic network, and assists the order parameters in performing global joint evaluation.

[0175] Understandably, during the training process, the agent's policy learning is based on the "Actor-Critic" framework, which equips each agent with an independent Actor network and Critic network. The Actor network generates action decisions based on its own local observation information, while the Critic network is used to evaluate the joint actions of the agent and its environment, and outputs the corresponding Q-value, which is the reward for the drone, to measure the merits of the current policy.

[0176] To clarify, during the training process of the UAV swarm formation model, i.e. the update process of the model parameters, the Actor network optimizes its policy by maximizing the Q value output by the Critic network. That is, it adjusts the network parameters through the policy gradient method so that the agent can obtain higher long-term rewards in a given environment. The Critic network uses the Temporal Difference Error (TD-Error) mechanism to update its parameters and improves its evaluation ability by minimizing the difference between the estimated Q value and the target Q value.

[0177] Specifically, in this embodiment, the DA-MADDPG algorithm introduces a bottom-up attention mechanism, namely the Bottom-Up attention mechanism. By modeling the local interaction rules in the drone swarm, each agent can perceive important information in its own neighborhood. It draws on the idea of ​​the Boids model, emphasizing that individual drones should prioritize their neighbors that are closely related to their own behavior during the decision-making process, so as to achieve efficient self-organization and local collaboration. This not only improves the pertinence of individual decisions, but also helps the overall adaptive capability of the swarm.

[0178] Then, a top-down attention mechanism was introduced, which integrates the global objective and the overall structural features of the cluster into the decision-making of each UAV. Through the objective consistency rule, all UAVs are guided toward the common mission objective, improving the consistency of movement direction. Through the global aggregation rule, the convergence of the overall cluster space is promoted, enhancing the orderliness and stability of the cluster. At the same time, a clustering penalty rule is set to effectively suppress the cluster from splitting under external disturbances, ensuring the integrity of the overall structure.

[0179] Next, by modifying the network structure in the Critic network and introducing a multi-head architecture, a dual attention mechanism was also implemented. The dual attention mechanism enables the Critic network to not only evaluate the Q-value, but also to perform a global joint evaluation of phase transition parameters such as global order parameters. This improvement allows the agent to better balance individual goals and cluster collaboration in complex environments, ultimately achieving efficient collaboration in large-scale systems and task-driven formation control.

[0180] Preferably, the multi-head architecture introduced in the Critic network is a multi-task learning mechanism that jointly models the Q-value and auxiliary indicators such as global order parameters, essentially introducing the idea of ​​multi-task learning. Unlike traditional Critic networks, multi-head Critic networks not only output the Q-value but also multiple target values ​​related to the main task. Theoretical analysis shows that multi-head Critic networks, through the regularization effect of auxiliary tasks, can effectively reduce the risk of overfitting in the parameter space of the main task, improving generalization ability and convergence speed. Related gradient variance analysis shows that when there is a positive correlation between the auxiliary task and the main task, the gradient variance of the multi-head Critic network is smaller than that of the traditional single-head Critic network, resulting in a more stable and efficient parameter update process. In this embodiment, the order parameter, representing the overall state of the cluster, is used to introduce auxiliary supervision signals, enabling the Critic network to simultaneously possess Top-Down and Bottom-Up attention, ultimately improving the generalization ability, convergence speed, and stability of the main tasks such as Q-value prediction.

[0181] It can be explained that in step S4, the feasibility of the trained UAV swarm formation model is verified, that is, the performance of the DA-MADDPG algorithm is verified. Specifically, a series of simulation experiments are carried out using an experimental platform. In actual operation, the experimental platform parameters and the relevant training parameters of the DA-MADDPG algorithm are set to ensure the operation of the simulation experiments.

[0182] Please see Figure 4First, we need to verify the consistency convergence performance of the UAV swarm. This is a key indicator used to measure whether a UAV swarm can efficiently achieve a consistent goal within a limited time through local information interaction during collaborative control or distributed task execution. Consistency refers to the ability of all members within the swarm to gradually converge to the same state variables, such as position, speed, heading, or task assignment, or to satisfy a certain consistency relationship through collaborative strategies. Convergence performance is reflected in the time required for the swarm to reach a consistent goal from its initial dispersed state, the speed of convergence, the stability of the convergence process, and its robustness to external disturbances and communication delays. Good consistency convergence performance means that the swarm can efficiently and accurately achieve its collaborative goal in a short time and can adapt to changes in the number of members or external environmental interference, improving the overall collaborative operation efficiency and task completion quality of the swarm.

[0183] Please see Figure 5 The curves represent dimensionless data, describing the average value after multiple samplings, while the shaded areas represent the statistical variance.

[0184] Specifically, consistency convergence analysis of the algorithm was conducted for small-scale clusters (3 drones), medium-scale clusters (9 drones), and large-scale clusters (18 drones). The consistency convergence of the MADDPG and MAPPO algorithms was also compared and analyzed to demonstrate that the DA-MADDPG algorithm has efficient policy learning capabilities, fast convergence, and good scalability in multi-agent systems, and can adapt to the collaborative needs of clusters of different sizes.

[0185] The results show that in small-scale clusters, the DA-MADDPG algorithm exhibits a rapid increase in its order parameter during the initial training phase, quickly approaching 1 and maintaining a high and stable level throughout subsequent training. This demonstrates that the algorithm effectively drives highly consistent collaborative behavior in UAV clusters, exhibiting fast convergence speed and strong robustness, as shown in the curve with almost no significant fluctuations throughout the experiment. In contrast, the MADDPG algorithm shows significant fluctuations in its order parameter during training, with a slower overall convergence speed and a significantly lower average order parameter than DA-MADDPG. Furthermore, its larger variance during convergence indicates that MADDPG struggles to achieve stable cluster consistency and is susceptible to environmental disturbances or policy instability. Although the order parameters of the MAPPO algorithm showed some improvement in the early stages of training, they eventually stabilized at a moderate level with relatively small fluctuations. Overall, its performance was still inferior to DA-MADDPG, indicating that while MAPPO can improve cluster consistency to some extent, its collaborative efficiency and eventual consistency level are limited in small-scale multi-agent tasks. Therefore, DA-MADDPG's consistency convergence performance in small-scale UAV swarms is significantly better than that of MADDPG and MAPPO algorithms. Its efficient convergence speed and excellent stability make it more suitable for practical scenarios with high requirements for cluster collaborative consistency, verifying the effectiveness and superiority of the DA-MADDPG algorithm in improving the collaborative capabilities of multi-agent swarms.

[0186] Regarding reward convergence performance, the DA-MADDPG algorithm rapidly increases and maintains a high level in the early stages of training, exhibiting strong stability and high returns with relatively small overall fluctuations. As training progresses, it consistently achieves high rewards, indicating superior policy learning ability and stronger environmental adaptability. While the MADDPG algorithm shows some improvement in the early stages of training, its overall performance fluctuates significantly, with marked curve undulations, and ultimately falls significantly below the DA-MADDPG algorithm. This suggests that the MADDPG algorithm struggles to consistently achieve high returns in small-scale cluster environments, potentially being significantly affected by policy instability or environmental disturbances. Furthermore, the wide range of the MADDPG reward curve indicates poor robustness. The MAPPO algorithm, on the other hand, remains essentially near zero, showing almost no significant improvement or fluctuations. This indicates that MAPPO failed to effectively learn valuable policies in this experimental scenario, resulting in low overall returns for the agent group and limited collaborative capabilities. In conclusion, DA-MADDPG significantly outperforms both MADDPG and MAPPO algorithms in small-scale multi-agent tasks, demonstrating its ability to more efficiently improve the collaborative returns of agents.

[0187] In medium-sized clusters, regarding convergence performance of the order parameter, the DA-MADDPG algorithm shows a rapid increase in the average order parameter during the initial training phase, quickly reaching a high level close to 1.0, and maintaining high stability throughout the training process with almost no significant fluctuations. This indicates that the algorithm can efficiently drive the cluster to achieve highly consistent cooperative behavior, with fast convergence speed and strong robustness. While the MADDPG algorithm shows a relatively rapid increase in the order parameter during the initial training phase, it experiences a significant drop after reaching its peak, exhibiting large fluctuations during subsequent convergence, eventually stabilizing between 0.6 and 0.7 with a large variance. This suggests that the algorithm struggles to maintain high consistency in medium-sized clusters and is significantly affected by environmental disturbances and policy instability. The MAPPO algorithm shows a slower increase in the average order parameter and the lowest overall convergence level, eventually stabilizing around 0.4 with relatively smaller fluctuations. However, its overall consistency and cooperative efficiency are significantly weaker than the DA-MADDPG and MADDPG algorithms, indicating that this algorithm struggles to achieve efficient cluster cooperation in medium-sized multi-agent systems. In summary, this demonstrates that DA-MADDPG significantly outperforms the MADDPG and MAPPO algorithms in terms of consensus convergence performance in medium-sized clusters, achieving a higher level and more stable cluster collaborative consensus.

[0188] Regarding the convergence performance of rewards, the DA-MADDPG algorithm rapidly increased in the early stages of training, quickly reaching a high level, and then remained between 300 and 400 for a long period. Although there were some fluctuations, the overall trend was relatively stable and consistently higher than other algorithms. This indicates that the algorithm can effectively improve the overall return of the multi-agent system, and its strategy has good adaptability and robustness. The MADDPG algorithm also showed a slight increase in the early stages of training, but quickly experienced a significant drop, and remained below 0 throughout the subsequent training process, with large fluctuations. This indicates that its strategy has poor stability and is easily affected by environmental disturbances or poor cooperation among agents. The final average reward curve is significantly lower than that of DA-MADDPG. The MAPPO algorithm almost always remained near 0, with minimal overall fluctuations and no obvious upward trend. This indicates that MAPPO failed to learn an effective cooperative strategy in this scenario, resulting in extremely low overall cluster returns and the worst performance. In summary, the results show that the DA-MADDPG algorithm not only significantly outperforms the MADDPG and MAPPO algorithms in terms of reward level, but also has a faster convergence speed, more stable long-term performance, and demonstrates stronger policy learning ability and cluster collaboration efficiency. Specifically, the MADDPG algorithm's reward and stability decrease significantly as the cluster size increases, while MAPPO becomes almost unsuitable for the current task. This demonstrates that the DA-MADDPG algorithm exhibits significant advantages in medium-sized multi-agent reinforcement learning tasks, enabling efficient and stable cluster collaboration and reward improvement.

[0189] In large-scale clusters, the DA-MADDPG algorithm still demonstrates a strong advantage in terms of convergence performance regarding the order parameter. Its average order parameter rapidly rises to nearly 1.0 in the early stages of training and remains at a very high and stable level throughout subsequent training, with almost no significant fluctuations. This indicates that even in large-scale cluster collaborative tasks, the algorithm can effectively drive all agents to achieve highly consistent collaborative behavior, exhibiting excellent convergence speed and robustness. While the MADDPG algorithm's average order parameter also increases rapidly in the early stages of training, it quickly declines to a certain extent and eventually stabilizes between 0.7 and 0.8, with significant fluctuations. This reflects that as the number of agents increases, its cluster consistency and policy stability are significantly affected. The MAPPO algorithm performs the worst, with a slow increase in the order parameter, gradually declining after reaching a peak, and eventually converging between 0.3 and 0.4. The overall fluctuation is small, but the consistency level is low, indicating that it is difficult to adapt to the complex collaborative requirements of large-scale multi-agent systems. In summary, as the cluster size increases, only the DA-MADDPG algorithm can still maintain extremely high consistency and stability, while the performance of MADDPG and MAPPO both declines significantly, verifying the superiority of DA-MADDPG in large-scale multi-agent cluster collaborative tasks.

[0190] Regarding reward convergence performance, the DA-MADDPG algorithm is almost unaffected by the increase in cluster size. Its average reward curve rises rapidly in the early stages of training and stabilizes at around 400, maintaining a high level thereafter. Although there are some fluctuations, the overall trend is very stable, indicating that the algorithm can consistently obtain high returns when facing large-scale intelligent agent systems, demonstrating good policy learning ability and system robustness. The MADDPG algorithm, while showing some improvement in the early stages, quickly falls back to near 0 and hovers around 0 for a long time in subsequent training, exhibiting large fluctuations and high reward variance, reflecting a significant decrease in its collaborative ability and reward acquisition ability in large-scale scenarios. The MAPPO algorithm's average reward curve remains close to 0 throughout, showing almost no improvement and minimal fluctuations, indicating that it struggles to learn effective policies in large-scale multi-agent tasks, resulting in extremely low overall returns. In summary, as the cluster size increases, the DA-MADDPG algorithm maintains efficient and stable reward performance, while the reward levels of the MADDPG and MAPPO algorithms decrease significantly, validating the significant advantages of the DA-MADDPG algorithm in large-scale multi-agent collaborative scenarios.

[0191] It can be noted that under different cluster sizes, the three algorithms DA-MADDPG, MADDPG, and MAPPO exhibit significant performance differences. Considering small-scale, medium-scale, and large-scale scenarios, the DA-MADDPG algorithm performs better overall. MADDPG is competitive in small-scale tasks, but its performance declines significantly as the cluster size increases. MAPPO, on the other hand, does not achieve ideal results under any scale condition.

[0192] Secondly, the swarm formation control capability of the UAV swarm was verified. Through formation control, UAV swarms can achieve efficient collaborative operations among multiple UAVs. A reasonable formation design ensures clear division of labor and close cooperation among the UAVs, improving the overall mission completion efficiency. For example, in large-area inspections, regional searches, and emergency rescue missions, a good swarm formation can cover a wider area, reduce duplication and omissions, and thus improve the comprehensiveness and timeliness of information collection.

[0193] In this embodiment, simulation experiments were conducted in two-dimensional and three-dimensional spaces. Specifically, for the two-dimensional space experiment, the DA-MADDPG algorithm was used to realize a circular formation of 6 drones, with each drone evenly distributed on the designated target circumference. The experiment was conducted by setting formation reward rules and performing multiple independent repeated trials to avoid accidental deviations. The experiment showed that most drones could move to the assigned target location quickly and stably, but some agents exhibited fluctuations or delays during the movement, revealing the widespread individual differences and environmental complexity in multi-agent systems.

[0194] For the three-dimensional space experiment, the DA-MADDPG algorithm is still used to implement a 6-machine spherical formation. The Fibonacci lattice algorithm is used to generate uniformly distributed points on the sphere, and these points are used as the formation target points of the algorithm. During the three-dimensional space formation training, the formation reward rules determined in the two-dimensional space experiment are still used, but three-dimensional space vectors are used for calculation. This experiment is used to illustrate the cooperative control capability and global convergence characteristics of the DA-MADDPG algorithm in multi-dimensional space formation tasks.

[0195] Finally, the robustness of the UAV swarm formation model was verified. It ensures that even when some UAVs fail due to malfunction, energy depletion, or external interference, the remaining agents can still adaptively adjust and complete the intended task, avoiding mission failure. Furthermore, robustness helps the system maintain its formation structure and cooperative relationships when facing environmental changes, communication interruptions, or local disturbances, ensuring the stability and reliability of the swarm operation. A highly robust swarm can effectively cope with various emergencies and uncertainties, reducing the impact of single-point failures on the overall model and improving its survivability and adaptability in complex dynamic environments. It also enhances the model's scalability and flexibility, enabling the swarm to flexibly respond to member additions or removals or functional changes, facilitating the upgrade and expansion of the UAV swarm formation model. This is fundamental to achieving high reliability, high adaptability, and high autonomy in multi-agent swarms, and is crucial for ensuring the effective operation of UAV systems in practical applications.

[0196] In this embodiment, during the flight of the UAV swarm, a certain number of UAVs are randomly selected to fail, and the operational status of the UAV swarm is examined to see if this changes. Specifically, this is based on the process of six UAVs coordinating their movement in two-dimensional space. In the initial stage, all agents maintain an orderly formation and move towards the designated direction. Starting from step 30, the 5th and 6th UAVs are selected to fail and not participate in the swarm formation. According to experimental feedback, although some UAVs suddenly fail, the remaining UAVs can still dynamically adjust their positions, maintain the overall formation structure, and continue to move towards the target direction. This demonstrates that the DA-MADDPG algorithm still exhibits strong self-organization ability and formation stability after local member failures, without significant formation disintegration or major deviations. This fully reflects the robustness of the DA-MADDPG algorithm, that is, the model as a whole can maintain stable operation and task continuity when facing local disturbances or individual failures.

[0197] Please see Figure 6 Figure (a) shows the reward curve, and Figure (b) shows the order parameter curve. It can be seen that during the robustness test, regarding the reward curve, although the DA-MADDPG algorithm exhibits some fluctuations, the overall reward level remains in a high range and does not show a continuous decline due to the failure of some agents. This indicates that the algorithm can maintain high task completion efficiency and stability even after encountering disturbances. Regarding the order parameter curve, it shows high consistency, with its mean always close to 1 and its standard deviation extremely small, reflecting that the algorithm's cooperative consistency and formation order are effectively maintained throughout the test. In summary, this shows that the DA-MADDPG algorithm has strong robustness. Whether from the perspective of reward, cooperative consistency, or actual formation evolution, the algorithm can effectively maintain the continuity of the overall task and the stability of the cooperative structure through the adaptive adjustment of the remaining drones even when some members fail.

[0198] As an alternative implementation, the multi-agent deep deterministic policy gradient method based on the attention mechanism proposed in this application can also be used to conduct performance studies under highly realistic conditions. This further illustrates that the UAV swarm formation model proposed by this method can move most UAVs to the assigned target location quickly and stably. However, some agents experience fluctuations or delays during the movement, which indicates the widespread individual differences and environmental complexity in multi-agent systems.

[0199] Understandably, the DA-MADDPG algorithm achieves efficient group behavior control from local rules to global goals by introducing a dual attention mechanism and a multi-head Critic network. The bottom-up attention mechanism enables each agent to dynamically perceive information about other individuals within its local neighborhood; this mechanism enhances local self-organization capabilities while also strengthening the model's adaptive adjustment ability to local perturbations. Then, a top-down attention mechanism is introduced, which effectively integrates global goals and structural information into the individual decision-making process through rules such as goal consistency, global aggregation, and clustering penalties, achieving coordinated adjustment of local and global information. This entire approach not only ensures that the cluster can maintain overall order and stability in complex environments but also theoretically provides a new paradigm for the cooperative control of multi-agent systems.

[0200] From a systemic perspective, the DA-MADDPG algorithm effectively balances the local optima of individual agents with the global optima of the cluster, overcoming the bottlenecks of insufficient collaborative consistency, robustness, and generalization ability in existing multi-agent reinforcement learning under large-scale and complex environments. Furthermore, experimental results on indicators such as reward and order parameters intuitively verify the algorithm's adaptive adjustment capability under agent failure or environmental disturbances. This not only enriches the theoretical foundation of multi-agent systems but also provides an effective solution for complex tasks such as large-scale UAV swarms and robot formations in engineering applications.

[0201] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0202] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A multi-agent deep deterministic policy gradient method based on attention mechanism, characterized in that, The method includes: Analyze the motion state of the drone swarm and establish a dynamic model; A drone swarm formation model is constructed by integrating multiple attention mechanisms and combining them with a dynamic model, and the order parameters are defined according to the drone swarm definition. A multi-agent approach is adopted to control the formation of a drone swarm through deep reinforcement learning, determine the corresponding reward for each drone, and train the drone swarm formation model by combining the reward and the order parameters. Verify the feasibility of the trained drone swarm formation model.

2. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 1, characterized in that, Analyze the motion state of the drone swarm and establish a dynamic model, including: Based on the motion state of the drone swarm, the velocity components of the drones in the horizontal and vertical planes are determined, and the motion direction angle of the drones is obtained. The spatial position of all drones in the drone swarm is determined by analyzing the velocity components, and the drone velocity is updated by combining the spatial drag.

3. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 1, characterized in that, The multiple attention mechanism includes a bottom-up attention mechanism and a top-down attention mechanism. The bottom-up attention mechanism includes cohesion rules, separation rules, and alignment rules. The top-down attention mechanism includes target consistency rules, global aggregation rules, and clustering penalty rules.

4. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 3, characterized in that, A drone swarm formation model is constructed by integrating multiple attention mechanisms and combining them with a dynamic model. The model is then defined based on the drone swarm's order parameters, including: The set of neighbors for any drone in a drone swarm is determined using a dynamic model; The analysis integrates dynamic models, UAVs, and their corresponding neighbor sets. The Bottom-Up attention mechanism emphasizes the autonomous decision-making of individual UAVs based on local perception and neighborhood information. The Top-Down attention mechanism incorporates the global objective and the structural characteristics of the UAV swarm into the decision-making process of each UAV. The order parameters are defined according to the drone cluster.

5. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 4, characterized in that, The Bottom-Up attention mechanism emphasizes the autonomous decision-making of individual drones based on local perception and neighborhood information, including: Based on the cohesion rule, the relative position vector of the centroid of the drone's neighbors is obtained by analyzing the drone and its corresponding neighbor set, thus obtaining the relative deviation distance of the drone. Based on the separation rule, the analysis shows that the drone and its corresponding set of neighbors generate forces that point away from the neighbors. Based on alignment rules, the drone adjusts its current direction of motion by analyzing the dynamic model through neighbor sets.

6. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 4, characterized in that, The top-down attention mechanism incorporates the global objective and the structural characteristics of the drone swarm into the decision-making process of each drone, including: Based on the target consistency rule, the target motion direction vector of the UAV cluster is obtained, and the cosine value of the angle between the current motion direction of the UAV and the target motion direction is obtained by combining the dynamic model. Based on global aggregation rules, the total number of drones in the drone cluster is counted, and the average distance in the drone cluster is determined by combining the drones and their corresponding neighbor sets. Based on clustering penalty rules, drones in a drone swarm are clustered to obtain multiple clusters. These clusters are then analyzed to obtain cluster profile coefficients. Penalty coefficients and penalty thresholds are set, and the penalties for each drone are determined by combining the cluster profile coefficients, penalty coefficients, and penalty thresholds.

7. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 4, characterized in that, According to the drone swarm definition sequence parameters, specifically: The activity area of ​​the drone swarm is divided into multiple grids, and the number of drones in each grid is counted. An order parameter is defined based on the number of grids.

8. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 3, characterized in that, A multi-agent approach using deep reinforcement learning is employed for the formation control of a drone swarm, determining the corresponding rewards for each drone. Specifically: When performing formation control of a drone swarm, the drones are defined as intelligent agents. The interaction between each intelligent agent and its neighboring intelligent agents is analyzed to make autonomous decisions. A reward and punishment mechanism is set up to determine the corresponding reward for the drone based on the autonomous decision-making process.

9. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 8, characterized in that, The system analyzes the interactions between each agent and its neighbors to make autonomous decisions, and establishes a reward and punishment mechanism. Based on the autonomous decision-making process, the system determines the corresponding reward for the drone, including: The spatial position and velocity of the drone are determined based on the feedback from the dynamic model, and the state space of the drone swarm is constructed. Establish a motion space for the drone's actions; The state space and action space together determine the deterministic transition; The reward for the drone is determined by defining a reward component for each rule in the Bottom-Up and Top-Down attention mechanisms and using a reward weight coefficient to combine the reward components of each rule.

10. The multi-agent deep deterministic policy gradient method based on attention mechanism according to claim 9, characterized in that, The drone swarm formation model is trained by combining rewards and order parameters, specifically as follows: Each agent is configured with an Actor network and a Critic network. The Critic network outputs the reward corresponding to the drone. During the training process of the drone swarm formation model based on the reward and the order parameters, the Actor network adjusts the parameters of the drone swarm formation model by maximizing the reward output by the Critic network, and assists the order parameters in performing global joint evaluation.

Images