Multi-leader follower cluster control method adopting multi-layer graph attention mechanism enhancement

Abstract

The invention relates to the technical field of multi-agent cooperative control, discloses a multi-leader follower cluster control method enhanced by adopting a multilayer graph attention mechanism, and solves the problems of poor topology adaptability and low cross-level cooperative efficiency in a dynamic environment. According to the main scheme, the method comprises the following steps: designing a state coding module adapted to role differences and a strategy network and evaluation network model based on graph attention differentiation; designing a role separation playback pool for leader-follower cooperative control; and designing a strategy gradient optimization method of deep reinforcement learning. Aiming at a cluster control task in a continuous action space, a graph attention enhanced multi-agent collaborative framework is provided, different role state input is unified through dynamic topology modeling, cross-level information fusion is realized by using a graph attention mechanism, the adaptive capacity of agents to a dynamic environment and the distinguishing capacity of the agents to a level relationship are improved, and the clustering control efficiency is improved. A role isolation experience storage mechanism is designed, and mutual interference between strategies of a leader and a follower is reduced.

Description

Technical Field

[0001] This invention relates to the fields of multi-agent systems and reinforcement learning, and specifically provides a multi-leader follower cluster control method enhanced by a multi-layer graph attention mechanism. Background Technology

[0002] Multi-leader follower swarm control technology is a crucial research area in the cooperative control of multi-agent systems and has been extensively studied. In recent years, breakthroughs in graph neural networks and deep learning technologies have propelled the level of intelligence in the field of intelligent swarm control, while also providing more powerful modeling and control capabilities for multi-leader follower systems in complex dynamic environments. Graph neural networks, especially graph attention networks, as a learning method that can effectively model complex dependencies between agents, have been attempted to be applied to swarm cooperative control. Because graph neural networks can dynamically adjust information transmission weights based on neighbor relationships, the learned cooperative strategies exhibit stronger environmental adaptability and robustness compared to traditional control methods based on fixed topologies or rules. Therefore, graph neural networks have become an important tool for solving multi-leader follower cooperative control problems.

[0003] However, current multi-leader follower cluster control algorithms based on graph neural networks still suffer from insufficient topology adaptability and weak cross-level information fusion capabilities. These topology limitations and bottlenecks in cross-level information transmission directly affect followers' ability to distinguish instructions from multiple leaders and their perception of the dynamic environment, weakening the overall collaborative effect and convergence speed of cluster control. Therefore, enhancing the dynamic adaptability of topology among agents and improving the efficiency of multi-level (especially between leader levels and between leader-follower levels) information fusion has become a crucial technical challenge. To address this issue, this invention proposes a multi-leader follower cluster control method enhanced by a multi-layer graph attention mechanism, addressing the shortcomings of existing methods in topology adaptability and hierarchical information fusion. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in traditional multi-leader follower cluster control methods, such as insufficient ability of followers to distinguish instructions from multiple sources of leaders, low efficiency of dynamic environment perception, and decreased cluster collaboration effect and convergence speed, caused by poor topology adaptability and weak cross-level information fusion capability.

[0005] To achieve the above objectives, the present invention employs the following technical means:

[0006] A multi-leader follower cluster control method enhanced by a multi-layer graph attention mechanism includes the following steps:

[0007] Step S1: Establish an intelligent agent point motion model, an obstacle detection model, and a leader-follower group perception model. The constructed model provides an interactive environment for training multi-agent cluster control based on reinforcement learning.

[0008] Step S2: Considering the task requirements and environmental constraints of controlling multi-leader follower clusters, design state, action control, and reward functions for deep reinforcement learning, as follows:

[0009] State design: Based on the intelligent agent mass model and obstacle detection model constructed in step S1, for the leader intelligent agent... Local observations are constructed using observation information from communicable neighboring leader agents, the relative position and velocity information of virtual leaders, and the position information of detected obstacles. For follower intelligent agents Local observations are constructed using the relative positions and velocities of communicable neighboring intelligent agents, the positional information of their respective real leaders, and the positional information of the detected obstacles. In cluster tasks, the global state is constructed by combining the local observations of all leaders and followers:

[0010] .

[0011] Action design: Define the agent's acceleration as the agent's action. Construct joint actions based on the actions of each agent. .

[0012] Reward function design: The control error generated in the multi-agent cluster control task is used as the reward value of agent control. The control error of the cluster task includes leader target penalty (penalizing the tracking error between the leader and the virtual leader), follower target penalty (penalizing the distance deviation between the follower and the real leader), neighboring agent distance penalty (penalizing the violation of the minimum safe distance between agents), acceleration mutation penalty (suppressing acceleration mutations of all agents), and obstacle avoidance penalty (grading the penalty for collision risk between agents and environmental obstacles).

[0013] Step S3: Based on the state designed in Step S2, at each time step, construct a dynamic topology graph according to the relative positions of the UAVs. Simultaneously, perform two-layer attention encoding for both the self-observation state and the communication observation state. Based on this, design and evaluate the network model and the policy network model, where:

[0014] Evaluate the network in global state and joint actions As input, after two-layer graph attention encoding, the output is a state-behavior evaluation value;

[0015] Policy network observes state As input, the acceleration control quantity is output after being encoded by a two-layer graph attention.

[0016] Step S4: By designing a multi-agent role separation replay pool, the experiences of the leader agent and the follower agent during the training process are stored separately to reduce experience interference between roles.

[0017] Step S5: Complete the parameter update by designing a multi-agent reinforcement learning deterministic policy optimization method based on centralized training and distributed execution;

[0018] Step S6, the multi-leader follower cluster control process based on the multi-layer graph attention mechanism includes the following steps:

[0019] Step S6.1: Using the dynamic graph topology construction and hierarchy differentiation module of step S3, process the observation input of the agent, generate unified graph structure parameters and inject hierarchy identification information;

[0020] Step S6.2: Using the state construction method in step S2 and the graph structure parameters generated in step S6.1, reconstruct the complete observation state of the agent;

[0021] Step S6.3: Input the agent observation state reconstructed in step S6.2 into the policy network model designed in step S5 to obtain the agent's acceleration control behavior;

[0022] Step S6.4: Input the agent behavior obtained in step S6.3 into the agent mass motion model in step S1 to complete the motion control execution of the agent.

[0023] Based on the above technical solution, step S1 includes:

[0024] Step S1.1: Establish a motion model of the intelligent agent's mass point: Treat each intelligent agent as having a radius A circular entity, described by a second-order differential system. The state of motion of the center of the circle:

[0025]

[0026] in, , and Representing intelligent agents respectively Position, velocity, and acceleration vectors and They represent and The differential, Indicates the total number of agents;

[0027] Step S1.2: Establish the following obstacle detection model:

[0028] Based on the agent's detection range, the agent The set of obstacles observed in the environment can be represented as:

[0029]

[0030] in, It is the collection of all obstacles in the current environment. It is an obstacle Location, The confidence radius of the detection sensors fitted to the intelligent agent, within which the detected obstacle motion information is reliable, wherein... Indicates the location of the agent.

[0031] Step S1.3: Based on the hierarchical collaborative control architecture, construct the jurisdictional relationship between leaders and followers and the group perception mechanism.

[0032] Jurisdiction definition: The agents are divided into One leader and One follower, one leader Governing a fixed-size subset of followers ,satisfy:

[0033] (Full coverage)

[0034] ( (No duplicate jurisdiction)

[0035] (Grouped into equal-sized groups)

[0036] Intra-group perception mechanism: Leaders accept virtual leaders and other leaders at the same level ( ) observation information. Followers The scope of their perception is limited to their leader. Within its jurisdiction, followers Leaders available Its precise location and speed can only detect other followers within the same group. The state.

[0037] In the above technical solution, step S2 includes:

[0038] Design State Space: Leader Agent By establishing communication with neighboring leader agents, the relative positions and velocities of all agents observed by the neighboring leader agents can be obtained. Based on the relative positions and velocities of the virtual leader observed by itself and its own flag, a leader agent can be constructed. The internal observation state of is represented as:

[0039] , ,…, , , ,…, ]

[0040] in, Represents the neighbor leader intelligent agent Relative to the leader intelligent agent The relative motion state, Represents follower intelligent agents Compared to the leader intelligent agent The relative motion state, This indicates that the virtual leader is relative to the leader intelligent agent. The relative motion states are represented as follows:

[0041]

[0042]

[0043]

[0044] in, Represented as the magnitude of the agent's maximum speed. Represented as the normalized parameter of the relative distance between agents. Indicates the position of the agent. Indicates the speed of the intelligent agent. This represents the maximum speed of the agent.

[0045] Similarly, follower agents By establishing communication with the leader and other followers within the group, obtaining relative position and velocity information, and combining this with its own flags, a follower intelligent agent can be constructed. The internal observation state of is represented as:

[0046] , , ,…, ]

[0047] in, The relative motion state between the follower agent and its leader agent.

[0048] Therefore, the system state of a multi-agent cluster is defined as the joint state:

[0049]

[0050] Designing the motion space: Based on the intelligent agent's point motion model in step S1, motion control of the intelligent agent can be achieved by simply providing acceleration control. Therefore, the actions of the intelligent agent are designed to... Defined as acceleration control quantity: ;

[0051] Design a reward function: Use the system control error of the multi-agent cooperative control system as the agent's reward. The control error includes the control error of each agent. , represented as ,include:

[0052] 1. Leaders track punishment items

[0053] Based on potential field theory, a tracking penalty term is designed to drive the leader toward the virtual leader through the potential energy gradient.

[0054]

[0055]

[0056] in, Let be the potential energy function, with parameters The potential energy field intensity coefficient, yes The Euclidean distance between the virtual leader and the current agent at any given moment. This represents the minimum target tracking penalty. A constant representing a negative number

[0057] 2. Follower tracking penalty items

[0058] Similarly, a tracking penalty term is designed based on potential field theory, which drives the followers to move toward the leader through the potential energy gradient.

[0059]

[0060] in, yes The European distance between a constant follower and their leader. A constant representing a negative number.

[0061] 3. Cluster control distance penalty

[0062] Clustering requires all agents to maintain a desired distance while moving together, and each agent needs to design a cluster control distance penalty term with other agents with which it has a connection.

[0063]

[0064] in, It is the expected distance between any two agents in the cluster, where Represents intelligent agents Location, Represents intelligent agents Location, A constant representing a negative number.

[0065] 4. Acceleration mutation penalty term

[0066] Suppress acceleration mutations in all agents.

[0067]

[0068] in A constant representing a positive number. This represents the current acceleration of the agent.

[0069] 5. Obstacle Avoidance Penalties

[0070] Intelligent agents need to avoid obstacles when they encounter them.

[0071]

[0072] in, It is a maximally negative constant. It is the distance between the center of the obstacle and the center of the agent. It is the obstacle avoidance limit distance. It is the radius of the obstacle avoidance buffer zone.

[0073] Therefore, in a cluster environment, the reward function of a multi-agent system is designed as follows:

[0074]

[0075] In the above technical solution, step S3 includes:

[0076] Step S3.1, Dynamic Topology Graph Construction:

[0077] The distance matrix is ​​obtained by Euclidean distance calculation of the agent position obtained in step S1.1 and the obstacle data obtained in step S1.2. The intra-group perception mechanism in step S1.3 is applied to perform binarization processing (the value is assigned as 1 within the perception range, otherwise it is 0) to generate a real-time adjacency matrix, which can reflect the changes in dynamic formation and record key neighbor information.

[0078] Step S3.2, Two-layer attention encoding:

[0079] The first layer is ontology observation attention, which performs linear transformation on the independent observation data (position, velocity, flags, etc.) of a single agent, calculates the weight distribution of its own state through a self-attention mechanism, and outputs an enhanced ontology feature vector.

[0080] The second layer is communication observation attention. Based on the adjacency matrix, neighboring drones within the communication range are selected. A trainable attention weight calculation module weights and aggregates the features of neighboring nodes. The weights are dynamically determined by the agent's hierarchical relationship, the relative distance between nodes, and feature similarity, outputting a communication feature vector that integrates neighbor information. A topology link breakage protection mechanism is also employed. When a drone loses all connections due to obstacles, the communication feature vector inherits historical values ​​with a certain decay factor, preventing sudden changes in action that could lead to loss of control. Based on this, individual state optimization and group collaborative adaptation are decoupled. The ontology layer ensures small-scale clustering, while the communication layer improves the efficiency of large-scale group collaboration.

[0081] Step S3.3: Evaluate the network model:

[0082] System status and agent behavior In the input evaluation network model, the observed state is first extracted, and then a two-layer graph attention encoding is performed according to step S3.2. Next, it is concatenated with the action feature encoding obtained from the fully connected layer to obtain the complete feature vector, which is then sent to the multilayer perceptron to obtain the evaluation value.

[0083] Step S3.4, Policy Network Model:

[0084] System status Similarly, in the input policy network model, the observed state is extracted and encoded using a two-layer graph attention method. Then, the processed features are fed into a multilayer perceptron to obtain the agent's behavior.

[0085] In the above technical solution, step S4 includes:

[0086] By reducing the interference of experience through a hierarchical storage mechanism, a leader experience replay pool is designed. Follower experience replay pool ,in Indicates the state at the next moment. It signifies a reward.

[0087] In the above technical solution, step S5 includes:

[0088] Network sharing mechanism design: In the multi-agent cooperative control framework, a parameter sharing strategy is adopted to achieve efficient training. Firstly, the policy network is shared, with all agents sharing the same policy network. Second, assess network sharing: adopt a dual-assessment network architecture. .

[0089] Evaluation of network parameter updates: The network is evaluated by minimizing the Bellman residual optimization.

[0090]

[0091] Indicates the state of the agent. Represents the acceleration of the intelligent agent;

[0092] The target value calculation uses a double... Inhibition mechanism:

[0093]

[0094] Indicates the first Value network, This indicates the state of the agent at the next moment. This indicates the agent's observation at the next moment. This indicates the agent's observation at the next moment.

[0095] Policy network parameter update: Gradient calculation introduces an entropy regularization term.

[0096]

[0097] in, , For policy entropy, This is the entropy coefficient.

[0098] Because the present invention employs the above-mentioned technical means, it has the following beneficial effects:

[0099] I. Dynamic Topology Adaptability and Cross-Level Collaborative Optimization. This invention targets multi-leader follower cluster systems and employs a multi-layer graph attention mechanism. By constructing a dynamic adjacency matrix in real time, it effectively solves the problem of poor adaptability of traditional fixed topologies in complex dynamic environments. This method significantly improves the cluster's response speed and collaborative efficiency in scenarios such as obstacle emergence and target conflict, enhancing the overall robustness of the system.

[0100] II. High-fidelity simulation of motion and perception models. By establishing a rigid body motion model for the agent and an obstacle detection model, a highly realistic interactive environment is provided for reinforcement learning. This design enables the agent to accurately perceive its own motion state and changes in the external environment, significantly improving its environmental adaptability.

[0101] III. Hierarchical Reinforcement Learning Element Design. For the leader-follower architecture, a hierarchical state space, action space, and reward function were innovatively designed. This scheme significantly improves the agent's decision-making accuracy and learning efficiency in complex tasks by differentiating between the needs of leader path planning and follower collaborative control.

[0102] IV. Graph Structure Representation and Role Perception Mechanism. A graph attention network is used to dynamically encode the system topology, and a role differentiation mechanism is combined to independently handle leader and follower strategies. This design achieves accurate modeling of group interaction relationships, significantly improving the adaptability and generalization ability of multi-agent collaborative control.

[0103] V. Role-isolated experience replay pool. By designing an experience storage mechanism that separates leaders and followers, interference between strategies of different roles is effectively reduced.

[0104] VI. Multi-layered Attention Decision Network Architecture. An innovative multi-layered graph attention network was constructed, achieving end-to-end optimization from micro-level individual behavior to macro-level group collaboration through a cascaded design of local interaction layers and global instruction layers. This architecture significantly improves control accuracy in complex scenarios while ensuring real-time performance.

[0105] This invention successfully solves the challenge of large-scale cluster control in multi-agent collaborative control by innovatively integrating dynamic graph topology modeling, role perception mechanisms, and hierarchical reinforcement learning architecture. It achieves real-time optimization of agent interaction based on a multi-layer graph attention mechanism, enhances environmental adaptability by combining kinematic models and obstacle perception frameworks, and significantly improves training efficiency through role isolation experience playback and hierarchical reward design. Attached Figure Description

[0106] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0107] Figure 2 This is a schematic diagram of the control flow for a multi-leader follower cluster control method enhanced with a multi-layer graph attention mechanism;

[0108] Figure 3 This is a schematic diagram of the network structure for strategy evaluation. Detailed Implementation

[0109] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.

[0110] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.

[0111] The core objective of this invention is to significantly enhance the topology dynamic adaptability of a cluster system and improve the efficiency of cross-level (leader-leader, leader-follower) information fusion by introducing an innovative control method. The ultimate goal is to solve a series of problems caused by poor information processing, such as unclear follower instruction differentiation, low environmental awareness efficiency, poor cluster collaboration, and slow convergence speed. To achieve this objective, the invention proposes two key technological innovations: a multi-layer graph attention mechanism and a role-separated replay pool.

[0112] Specifically, this invention provides a multi-leader follower cluster control method enhanced by a multi-layer graph attention mechanism, comprising the following steps:

[0113] Step S1: Establish an intelligent agent point motion model, an obstacle detection model, and a leader-follower group perception model. The constructed model provides an interactive environment for training multi-agent cluster control based on reinforcement learning.

[0114] Step S2: Considering the task requirements and environmental constraints of controlling multi-leader follower clusters, design state, action control, and reward functions for deep reinforcement learning, as follows:

[0115] State design: Based on the intelligent agent mass model and obstacle detection model constructed in step S1, for the leader intelligent agent... Local observations are constructed using observation information from communicable neighboring leader agents, the relative position and velocity information of virtual leaders, and the position information of detected obstacles. For follower intelligent agents Local observations are constructed using the relative positions and velocities of communicable neighboring intelligent agents, the positional information of their respective real leaders, and the positional information of the detected obstacles. In cluster tasks, the global state is constructed by combining the local observations of all leaders and followers:

[0116] .

[0117] Action design: Define the agent's acceleration as the agent's action. Construct joint actions based on the actions of each agent. .

[0118] Reward function design: The control error generated in the multi-agent cluster control task is used as the reward value of agent control. The control error of the cluster task includes leader target penalty (penalizing the tracking error between the leader and the virtual leader), follower target penalty (penalizing the distance deviation between the follower and the real leader), neighboring agent distance penalty (penalizing the violation of the minimum safe distance between agents), acceleration mutation penalty (suppressing acceleration mutations of all agents), and obstacle avoidance penalty (grading the penalty for collision risk between agents and environmental obstacles).

[0119] Step S3: Based on the state designed in Step S2, at each time step, construct a dynamic topology graph according to the relative positions of the UAVs. Simultaneously, perform two-layer attention encoding for both the self-observation state and the communication observation state. Based on this, design and evaluate the network model and the policy network model, where:

[0120] Evaluate the network in global state and joint actions As input, after two-layer graph attention encoding, the output is a state-behavior evaluation value;

[0121] Policy network observes state As input, the acceleration control quantity is output after being encoded by a two-layer graph attention.

[0122] Step S4: By designing a multi-agent role separation replay pool, the experiences of the leader agent and the follower agent during the training process are stored separately to reduce experience interference between roles.

[0123] Step S5: Complete the parameter update by designing a multi-agent reinforcement learning deterministic policy optimization method based on centralized training and distributed execution;

[0124] Step S6, the multi-leader follower cluster control process based on the multi-layer graph attention mechanism includes the following steps:

[0125] Step S6.1: Using the dynamic graph topology construction and hierarchy differentiation module of step S3, process the observation input of the agent, generate unified graph structure parameters and inject hierarchy identification information;

[0126] Step S6.2: Using the state construction method in step S2 and the graph structure parameters generated in step S6.1, reconstruct the complete observation state of the agent;

[0127] Step S6.3: Input the agent observation state reconstructed in step S6.2 into the policy network model designed in step S5 to obtain the agent's acceleration control behavior;

[0128] Step S6.4: Input the agent behavior obtained in step S6.3 into the agent mass motion model in step S1 to complete the motion control execution of the agent.

[0129] Based on the above technical solution, step S1 includes:

[0130] Step S1.1: Establish a motion model of the intelligent agent's mass point: Treat each intelligent agent as having a radius A circular entity, described by a second-order differential system. The state of motion of the center of the circle:

[0131]

[0132] in, , and Representing intelligent agents respectively Position, velocity, and acceleration vectors and They represent and The differential, Indicates the total number of agents;

[0133] Step S1.2: Establish the following obstacle detection model:

[0134] Based on the agent's detection range, the agent The set of obstacles observed in the environment can be represented as:

[0135]

[0136] in, It is the collection of all obstacles in the current environment. It is an obstacle Location, The confidence radius of the detection sensors fitted to the intelligent agent, within which the detected obstacle motion information is reliable, wherein... Indicates the location of the agent.

[0137] Step S1.3: Based on the hierarchical collaborative control architecture, construct the jurisdictional relationship between leaders and followers and the group perception mechanism.

[0138] Jurisdiction definition: The agents are divided into One leader and One follower, one leader Governing a fixed-size subset of followers ,satisfy:

[0139] (Full coverage)

[0140] ( (No duplicate jurisdiction)

[0141] (Grouped into equal-sized groups)

[0142] Intra-group perception mechanism: Leaders accept virtual leaders and other leaders at the same level ( ) observation information. Followers The scope of their perception is limited to their leader. Within its jurisdiction, followers Leaders available Its precise location and speed can only detect other followers within the same group. The state.

[0143] In the above technical solution, step S2 includes:

[0144] Design State Space: Leader Agent By establishing communication with neighboring leader agents, the relative positions and velocities of all agents observed by the neighboring leader agents can be obtained. Based on the relative positions and velocities of the virtual leader observed by itself and its own flag, a leader agent can be constructed. The internal observation state of is represented as:

[0145] , ,…, , , ,…, ]

[0146] in, Represents the neighbor leader intelligent agent Compared to the leader intelligent agent The relative motion state, Represents follower intelligent agents Compared to the leader intelligent agent The relative motion state, This indicates that the virtual leader is relative to the leader intelligent agent. The relative motion states are represented as follows:

[0147]

[0148]

[0149]

[0150] in, Represented as the magnitude of the agent's maximum speed. Represented as the normalized parameter of the relative distance between agents. Indicates the position of the agent. Indicates the speed of the intelligent agent. This represents the maximum speed of the agent.

[0151] Similarly, follower agents By establishing communication with the leader and other followers within the group, obtaining relative position and velocity information, and combining this with its own flags, a follower intelligent agent can be constructed. The internal observation state of is represented as:

[0152] , , ,…, ]

[0153] in, The relative motion state between the follower agent and its leader agent.

[0154] Therefore, the system state of a multi-agent cluster is defined as the joint state:

[0155]

[0156] Designing the motion space: Based on the intelligent agent's point motion model in step S1, motion control of the intelligent agent can be achieved by simply providing acceleration control. Therefore, the actions of the intelligent agent are designed to... Defined as acceleration control quantity: ;

[0157] Design a reward function: Use the system control error of the multi-agent cooperative control system as the agent's reward. The control error includes the control error of each agent. , represented as ,include:

[0158] 1. Leaders track punishment items

[0159] Based on potential field theory, a tracking penalty term is designed to drive the leader toward the virtual leader through the potential energy gradient.

[0160]

[0161]

[0162] in, Let be the potential energy function, with parameters The potential energy field intensity coefficient, yes The Euclidean distance between the virtual leader and the current agent at any given moment. This represents the minimum target tracking penalty. A constant representing a negative number

[0163] 2. Follower tracking penalty items

[0164] Similarly, a tracking penalty term is designed based on potential field theory, which drives the followers to move toward the leader through the potential energy gradient.

[0165]

[0166] in, yes The European distance between a constant follower and their leader. A constant representing a negative number.

[0167] 3. Cluster control distance penalty

[0168] Clustering requires all agents to maintain a desired distance while moving together, and each agent needs to design a cluster control distance penalty term with other agents with which it has a connection.

[0169]

[0170] in, It is the expected distance between any two agents in the cluster, where Represents intelligent agents Location, Represents intelligent agents Location, A constant representing a negative number.

[0171] 4. Acceleration mutation penalty term

[0172] Suppress acceleration mutations in all agents.

[0173]

[0174] in A constant representing a positive number. This represents the current acceleration of the agent.

[0175] 5. Obstacle Avoidance Penalties

[0176] Intelligent agents need to avoid obstacles when they encounter them.

[0177]

[0178] in, It is a maximally negative constant. It is the distance between the center of the obstacle and the center of the agent. It is the obstacle avoidance limit distance. It is the radius of the obstacle avoidance buffer zone.

[0179] Therefore, in a cluster environment, the reward function of a multi-agent system is designed as follows:

[0180]

[0181] In the above technical solution, step S3 includes:

[0182] Step S3 includes:

[0183] Step S3.1, Dynamic Topology Graph Construction:

[0184] The distance matrix is ​​obtained by Euclidean distance calculation of the agent position obtained in step S1.1 and the obstacle data obtained in step S1.2. The intra-group perception mechanism in step S1.3 is applied to perform binarization processing (the value is assigned as 1 within the perception range, otherwise it is 0) to generate a real-time adjacency matrix, which can reflect the changes in dynamic formation and record key neighbor information.

[0185] Step S3.2, Two-layer attention encoding:

[0186] The first layer is ontology observation attention, which performs linear transformation on the independent observation data (position, velocity, flags, etc.) of a single agent, calculates the weight distribution of its own state through a self-attention mechanism, and outputs an enhanced ontology feature vector.

[0187] The second layer is communication observation attention. Based on the adjacency matrix, neighboring drones within the communication range are selected. A trainable attention weight calculation module weights and aggregates the features of neighboring nodes. The weights are dynamically determined by the agent's hierarchical relationship, the relative distance between nodes, and feature similarity, outputting a communication feature vector that integrates neighbor information. A topology link breakage protection mechanism is also employed. When a drone loses all connections due to obstacles, the communication feature vector inherits historical values ​​with a certain decay factor, preventing sudden changes in action that could lead to loss of control. Based on this, individual state optimization and group collaborative adaptation are decoupled. The ontology layer ensures small-scale clustering, while the communication layer improves the efficiency of large-scale group collaboration.

[0188] Step S3.3: Evaluate the network model:

[0189] System status and agent behavior In the input evaluation network model, the observed state is first extracted, and then a two-layer graph attention encoding is performed according to step S3.2. Next, it is concatenated with the action feature encoding obtained from the fully connected layer to obtain the complete feature vector, which is then sent to the multilayer perceptron to obtain the evaluation value.

[0190] Step S3.4, Policy Network Model:

[0191] System status Similarly, in the input policy network model, the observed state is extracted and encoded using a two-layer graph attention method. Then, the processed features are fed into a multilayer perceptron to obtain the agent's behavior.

[0192] In the above technical solution, step S4 includes:

[0193] By reducing the interference of experience through a hierarchical storage mechanism, a leader experience replay pool is designed. Follower experience replay pool ,in Indicates the state at the next moment. It signifies a reward.

[0194] In the above technical solution, step S5 includes:

[0195] Network sharing mechanism design: In the multi-agent cooperative control framework, a parameter sharing strategy is adopted to achieve efficient training. Firstly, the policy network is shared, with all agents sharing the same policy network. Second, assess network sharing: adopt a dual-assessment network architecture. .

[0196] Evaluation of network parameter updates: The network is evaluated by minimizing the Bellman residual optimization.

[0197]

[0198] Indicates the state of the agent. Represents the acceleration of the intelligent agent;

[0199] The target value calculation uses a double... Inhibition mechanism:

[0200]

[0201] Indicates the first Value network, This indicates the state of the agent at the next moment. This indicates the agent's observation at the next moment. This indicates the agent's observation at the next moment.

[0202] Policy network parameter update: Gradient calculation introduces an entropy regularization term.

[0203]

[0204] in, , For policy entropy, This is the entropy coefficient.

[0205] To facilitate a better understanding of the technical concept of this invention by those skilled in the art, the innovative aspects of this invention will be further explained as follows:

[0206] Innovation Point 1:

[0207] Instead of adopting a general multi-agent model, this invention constructs a highly customized hierarchical collaborative control architecture with strictly defined roles and information flows, and designs a targeted reinforcement learning environment based on this architecture.

[0208] Detailed description of the proposed solutions:

[0209] Physical and perceptual modeling (step S1): This invention first abstracts each intelligent agent into a point mass following a second-order differential system through step S1.1. , This laid the kinematic foundation. Subsequently, step S1.2 established a kinematic basis based on the confidence radius. The obstacle detection model. Most importantly, step S1.3 explicitly establishes a leader-follower hierarchical collaborative control architecture:

[0210] Jurisdiction: All agents are strictly divided into m leaders and n followers. Each leader governs a fixed-size subset of followers that do not overlap, achieving "full coverage" and "no duplicate jurisdiction" over all followers.

[0211] Intra-group perception mechanism: Information flow is strictly limited. The leader can observe the virtual leader and other leaders; however, the perception range of follower j is strictly limited to its own leader. Within their jurisdiction, they can only obtain precise information about their direct superiors and the status of other followers in the same group.

[0212] Character differentiation state-action-reward design (step S2):

[0213] Based on the above model, this invention designs state, action, and reward functions that are strictly bound to the role for reinforcement learning.

[0214] State Design: Leader Local observation Local observations of follower j The constituent elements are drastically different: the former focuses on neighboring leaders and virtual leaders, while the latter focuses on its own real leader. Global State It is composed of the joint local observations of all agents.

[0215] Reward function design: A sophisticated composite reward function with five penalty terms was designed to penalize leader tracking error, follower spacing deviation, safe distance between agents, acceleration mutation, and obstacle avoidance risk, thereby quantifying the complex cluster task objective into an optimizable reward signal.

[0216] The essence lies in precisely mapping the inherent hierarchical structure and communication constraints of multi-leader cluster tasks in the real world to the modeling (environment) and solution (reward function) processes of reinforcement learning. Unlike many general multi-agent reinforcement learning (MARL) methods that treat all agents as homogeneous or only make simple distinctions, this invention creates an asymmetric and non-uniform information interaction environment by defining strict "governance relationships" and "intra-group perception mechanisms." This refined modeling makes subsequent control policy learning more targeted, directly addressing the challenge of "insufficient ability to distinguish instructions from multiple leaders." This design approach, which starts from the root of the problem and constructs a highly matched mathematical model and learning framework, is not self-evident.

[0217] Innovation Point Two:

[0218] Multi-layer graph attention network with injected hierarchical labels

[0219] The core of the invention, named "Multi-layer Graph Attention Mechanism," is innovative not simply by stacking Graph Attention (GAT) layers, but by designing a network mechanism that can perceive and utilize agent-level (role) information, thereby directly addressing the pain point of "weak cross-level information fusion capability."

[0220] Its core idea is clearly reflected in the control flow of step S6:

[0221] Dynamic Graph Construction and Hierarchical Differentiation (Step S6.1): Before the policy network processes the observation information, a "Dynamic Graph Topology Construction and Hierarchical Differentiation Module" is set up. The key function of this module is to "generate unified graph structure parameters and inject hierarchical identification information".

[0222] Complete observation state reconstruction (step S6.2): ​​Reconstruct the observation state of the agent using the graph structure parameters containing hierarchical information generated in the previous step.

[0223] Policy generation (step S6.3): The reconstructed and more complete observation state is fed into the policy network, and the final output control quantity is output.

[0224] "Injecting hierarchical identification information" is a unique innovation of this invention. Conventional GATs, such as the 2-3 layer architectures mentioned in some studies, dynamically learn the importance between nodes through attention weights, but they usually treat all nodes as the same type. In the scenario of this invention, the information interaction between leaders, the instruction transmission between leaders and followers, and the collision avoidance coordination between followers have completely different internal logics and importance.

[0225] This invention, by injecting hierarchical identifiers, enables graph attention networks to distinguish the hierarchy (e.g., leader-leader, leader-follower) of current information interactions when calculating attention weights. This allows the network to learn differentiated attention weights related to roles and hierarchies. For example, followers can learn to assign higher attention weights to instructions from their direct leaders while appropriately paying attention to the dynamics of other followers in the same group to avoid collisions. Compared to traditional GAT, this mechanism can more effectively integrate information from different levels and sources, thereby greatly enhancing the "cross-hierarchical information fusion capability," and is a direct and ingenious means to solve the core technical problems described in the background art. Although multi-layer GAT architectures have applications in other fields, this invention combines them with role-hierarchical identifiers for multi-leader follower control, constituting a highly innovative application.

[0226] Innovation Point Three:

[0227] Character Separation Experience Replay Pool

[0228] To further improve the learning efficiency and stability under complex hierarchical architectures, this invention introduces another key innovation in the training mechanism.

[0229] Role Separation Storage (Step S4): This invention explicitly proposes, "By designing a multi-agent role separation replay pool, the experiences of the leader agent and the follower agents during training are stored separately." Its core purpose is to "reduce interference between experiences of different roles."

[0230] In reinforcement learning, the experience replay pool is crucial for breaking data correlations and stabilizing the training process. In multi-agent systems, especially in heterogeneous role systems as defined in this invention, the experiences (i.e., (state, action, reward, next state) tuples) of leaders and followers follow drastically different distributions. Their observation spaces differ, their reward functions have different emphases, and their optimal policies are also quite different.

[0231] If these two vastly different experiences are mixed and stored in the same replay pool, the neural network will constantly switch between the two different data distributions during sampling training. This will undoubtedly increase the difficulty of fitting, lead to instability in the learning process, and even slow down the convergence speed. This is precisely the "inter-role experience interference" referred to in this invention.

[0232] Separating and storing experience by role (leader / follower) is equivalent to providing "refined" training data for the policy networks of different roles. When updating the leader's policy, it is sampled from the leader's experience pool; when updating the follower's policy, it is sampled from its own dedicated experience pool. This design, seemingly simple, directly addresses the pain points of heterogeneous multi-agent learning. It is consistent with innovations one (role definition) and two (role-aware network), forming a complete "role separation" closed loop from model definition and network design to training mechanism. Although the idea of ​​setting up independent experience pools for different agents or roles has existed in MARL research, this invention applies it to its original multi-leader-follower control framework as an organic component for solving specific problems (accelerating convergence and improving synergy), exhibiting a synergistic effect and forming an indispensable part of the overall solution of this invention.

Claims

1. A multi-leader follower cluster control method enhanced by a multi-layer graph attention mechanism, characterized in that, Includes the following steps: Step S1: Establish an intelligent body mass point motion model, an obstacle detection model, and a leader-follower group perception model to construct a reinforcement learning interactive environment; Step S2: Considering the task requirements and environmental constraints of controlling a multi-leader follower cluster, design state, action control, and reward functions for deep reinforcement learning: Step S3: Based on the state designed in Step S2, at each time step, a dynamic topology graph is constructed according to the relative positions of the UAVs. Simultaneously, a two-layer attention encoding of the self-observation state and the communication observation state is performed. Based on this, an evaluation network model and a policy network model are designed, wherein: Evaluate the network in global state and joint actions As input, after two-layer graph attention encoding, the output is a state-behavior evaluation value; Policy network observes state As input, the acceleration control quantity is output after being encoded by a two-layer graph attention. Step S4: Store leader experience separately in the role-separated replay pool. and follower experience ; Step S5: Update network parameters using a deterministic strategy optimization method that combines centralized training with distributed execution; Step S6, the multi-leader follower cluster control process based on the multi-layer graph attention mechanism includes the following steps: Step S6.1: Using the dynamic topology graph designed in step S3, process the observation input of the agent, generate unified graph structure parameters, and inject hierarchical identification information; Step S6.2: Reconstruct the observation state of the agent using the state construction method of step S2 and the graph structure parameters generated in step S6.1; Step S6.3: Input the observed state of the agent reconstructed in step S6.2 into the policy network model constructed in step S3 to obtain the acceleration control behavior of the agent; Step S6.4: Input the agent behavior obtained in step S6.3 into the agent mass motion model in step S1 to complete the motion control execution of the agent.

2. The method according to claim 1, characterized in that, Step S1 includes: Step S1.1: Establish a motion model of the intelligent agent's mass point: Treat each intelligent agent as having a radius A circular entity, described by a second-order differential system. The state of motion of the center of the circle: in, , and Representing intelligent agents respectively Position, velocity, and acceleration vectors and They represent and The differential, Indicates the total number of agents; Step S1.2: Establish the following obstacle detection model: Based on the agent's detection range, the agent The set of obstacles observed in the environment can be represented as: in, It is the collection of all obstacles in the current environment. It is an obstacle Location, The confidence radius of the detection sensors fitted to the intelligent agent, within which the detected obstacle motion information is reliable, wherein... Indicates the current position of the agent; Step S1.3: Based on the hierarchical collaborative control architecture, construct the jurisdictional relationship between leaders and followers and the intra-group perception mechanism; Jurisdiction definition: The agents are divided into One leader and One follower, one leader Governing a fixed-size subset of followers ,satisfy: Full coverage ( No duplicate jurisdiction Equal-size grouping Intra-group perception mechanism: Leaders accept virtual leaders and other leaders at the same level ( ) observation information, followers The scope of their perception is limited to their leader. Within its jurisdiction, followers Leaders available Its precise location and speed can only detect other followers within the same group. The state.

3. The method according to claim 1, characterized in that, Step S2 includes: Design State Space: Leader Agent By establishing communication with neighboring leader agents, the relative positions and velocities of all agents observed by the neighboring leader agents can be obtained. Based on the relative positions and velocities of the virtual leader observed by itself and its own flag, a leader agent can be constructed. The internal observation state of is represented as: , ,…, , , ,…, ] in, Represents the neighbor leader intelligent agent Compared to the leader intelligent agent The relative motion state, Represents follower intelligent agents Compared to the leader intelligent agent The relative motion state, This indicates that the virtual leader is relative to the leader intelligent agent. The relative motion states are represented as follows: in, Represented as the magnitude of the agent's maximum speed. Represented as the normalized parameter of the relative distance between agents. Indicates the position of the agent. Indicates the speed of the intelligent agent. This represents the agent's maximum speed; Similarly, follower agents By establishing communication with the leader and other followers within the group, obtaining relative position and velocity information, and combining this with its own flags, a follower intelligent agent can be constructed. The internal observation state of is represented as: , , ,…, ] in, The relative motion state between the follower agent and its leader agent; Therefore, the system state of a multi-agent cluster is defined as the joint state: Designing the motion space: Based on the intelligent agent's point motion model in step S1, motion control of the intelligent agent can be achieved by simply providing acceleration control. Therefore, the actions of the intelligent agent are designed to... Defined as acceleration control quantity: ; Design a reward function: Use the system control error of the multi-agent cooperative control system as the agent's reward. The control error includes the control error of each agent. , represented as ,include: Step 1. Leaders track punishment items Based on potential field theory, a tracking penalty term is designed to drive the leader's movement toward the virtual leader via the potential energy gradient: in, Let be the potential energy function, with parameters The potential energy field intensity coefficient, yes The Euclidean distance between the virtual leader and the current agent at any given moment. Minimum target tracking penalty A constant representing a negative number; Step 2. Follower tracking penalty Similarly, a tracking penalty term is designed based on potential field theory, which drives the followers to move toward the leader through the potential energy gradient; in, yes The European distance between a constant follower and their leader. A constant representing a negative number; Step 3. Cluster control distance penalty term Clustering requires all agents to maintain a desired distance while moving together. Each agent needs to design a cluster control distance penalty term with other agents connected to it. in, It is the expected distance between any two agents in the cluster, where Represents intelligent agents Location, Represents intelligent agents Location, A constant representing a negative number Let be the potential energy function; Step 4. Acceleration mutation penalty term Suppress acceleration mutations in all agents: in A constant representing a positive number. This represents the current acceleration of the agent. Step 5. Obstacle Avoidance Penalty Intelligent agents need to avoid obstacles when they encounter them. in, It is a maximally negative constant. It is the distance between the center of the obstacle and the center of the agent. It is the obstacle avoidance limit distance. It is the radius of the obstacle avoidance buffer zone; Therefore, in a cluster environment, the reward function of a multi-agent system is designed as follows:

4. The method according to claim 1, characterized in that, Step S3 includes: Step S3.1, Dynamic Topology Graph Construction: The distance matrix is ​​obtained by Euclidean distance calculation of the agent position obtained in step S1.1 and the obstacle data obtained in step S1.

2. The intra-group perception mechanism in step S1.3 is applied for binarization processing. The value is assigned as 1 within the perception range and 0 otherwise, generating a real-time adjacency matrix. This can reflect the changes in dynamic formation and record key neighbor information. Step S3.2, Two-layer attention encoding: The first layer is ontology observation attention, which performs a linear transformation on the independent observation data of a single agent, calculates the weight distribution of its own state through a self-attention mechanism, and outputs an enhanced ontology feature vector. The second layer is communication observation attention, which filters neighboring drones within the communication range based on the adjacency matrix. Through a trainable attention weight calculation module, the features of neighboring nodes are weighted and aggregated. The weights are dynamically determined by the agent hierarchy, the relative distance between nodes, and the feature similarity. The output is a communication feature vector that integrates neighbor information. At the same time, a topology link break protection mechanism is adopted. When a drone loses all connections due to obstacles, the communication feature vector inherits the historical values ​​with a decay factor. Step S3.3: Evaluate the network model: System status and agent behavior In the input evaluation network model, the observation state is first extracted, and then a two-layer graph attention encoding is performed according to step S3.

2. Next, it is concatenated with the action feature encoding obtained by the fully connected layer to obtain all feature vectors, which are then sent to the multilayer perceptron to obtain the evaluation value. Step S3.4, Policy Network Model: System status In the input policy network model, the observed state is extracted and encoded using a two-layer graph attention method. The processed features are then fed into a multilayer perceptron to obtain the agent's behavior.

5. The method according to claim 1, characterized in that, Step S4 includes: By reducing the interference of experience through a hierarchical storage mechanism, a leader experience replay pool is designed. Follower experience replay pool ,in Indicates the state at the next moment. It signifies a reward.

6. The method according to claim 1, characterized in that, Step S5 includes: Network sharing mechanism design: In the multi-agent cooperative control framework, a parameter sharing strategy is adopted to achieve efficient training. Firstly, the policy network is shared, with all agents sharing the same policy network. Second, assess network sharing: adopt a dual-assessment network architecture. ; Evaluation of network parameter updates: Evaluating the network by minimizing the Bellman residual optimization. Indicates the state of the agent. Represents the actions of the intelligent agent; The target value calculation uses a double... Inhibition mechanism: Indicates the first Value network, This indicates the state of the agent at the next moment. This indicates the agent's observation at the next moment. Represents the target policy network; Policy network parameter update: Gradient calculation introduces an entropy regularization term: in, , For policy entropy, This is the entropy coefficient.

Images