Large-scale unmanned aerial vehicle cluster adaptive grouping method fusing responsibility evolution

Abstract

The invention belongs to the technical field of collaborative unmanned aerial vehicle cluster reinforcement learning, and relates to a large-scale unmanned aerial vehicle cluster adaptive grouping method fusing responsibility evolution. Firstly, a group state evaluation mechanism is introduced, so that each unmanned aerial vehicle can obtain personalized group updating suggestions according to the marginal value of the unmanned aerial vehicle to collective efficiency, a cluster system is supported to realize distributed and dynamic group reconstruction in a large-scale scene, and low efficiency and inadaptability of a static division strategy are avoided. And secondly, generating responsibility representation based on the latest group structure, and realizing continuous optimization and diversified modeling of group function roles. The mechanism supports the groups to flexibly adjust responsibility division in the task evolution process, and the decision-making ability and function matching performance of the system in heterogeneous and changeable environments are enhanced. And finally, proposing a responsibility-guided information fusion mechanism, and through intra-group aggregation and inter-group selective interaction, improving local information sharing efficiency, remarkably reducing communication cost, and avoiding system overhead caused by frequent global communication.

Description

Technical Field

[0001] The present invention belongs to the technical field of collaborative unmanned aerial vehicle (UAV) cluster reinforcement learning, and relates to a large-scale UAV cluster adaptive grouping method integrating responsibility evolution. Background Art

[0002] With the rapid development of science and technology, drone swarms, as a typical large-scale multi-agent system, are becoming a core subject of research in complex dynamic systems. These systems have been widely applied in highly dynamic mission scenarios such as wireless communications, power dispatching, robot collaboration, intelligent transportation, and military gaming, and have attracted widespread attention in disciplines such as computer science, artificial intelligence, and ecological control. Unlike traditional strategies that rely on expert experience, the recent development of reinforcement learning technology has enabled drone swarms to autonomously learn through environmental interaction, providing a new path for intelligent decision-making.

[0003] In large-scale mission environments, efficiently managing and coordinating numerous unmanned units becomes a key challenge for system design. As mission complexity increases and the number of drones expands, the system must not only be able to rapidly adapt to changing environments but also support highly scalable collaborative mechanisms. To this end, the centralized training and decentralized execution (CTDE) multi-agent reinforcement learning architecture is widely adopted. It achieves global coordination through centralized policy optimization, followed by independent execution using distributed policies, balancing system efficiency and scalability.

[0004] However, most existing reinforcement learning methods often assume that the cluster structure remains static during training and execution, ignoring the dynamic changes often faced by unmanned swarm systems in practical applications. For example, during a coordinated patrol mission involving multiple drones, some drones may withdraw mid-route due to battery exhaustion or environmental interference, requiring timely adjustments to the original coordination strategy to reallocate the monitoring area. Similarly, in a game-playing environment, the original target mission may be temporarily upgraded to a strike or interference mission, requiring the cluster to quickly reconfigure the mission without changing the overall architecture. If a fixed structure is still used for scheduling and communication, it is very likely to lead to imbalanced resource allocation, conflicting coordination paths, or delayed information transmission, severely weakening the system's robustness and task completion efficiency in dynamic environments.

[0005] To address these challenges, the "UAV swarm collaboration mechanism based on adaptive grouping" has become a research hotspot. This mechanism allows the swarm to dynamically divide functional groups based on environmental changes, member status, and mission requirements, and adjusts the division of responsibilities in real time, thereby achieving flexible and efficient collaborative operations. Compared to traditional static grouping methods, adaptive grouping can provide more fine-grained task coordination and more reasonable resource scheduling, and is particularly suitable for the long-term operation of large-scale swarms in non-stationary environments. Therefore, building a reinforcement learning framework that supports dynamic grouping and improving its flexibility and stability in task collaboration, information transmission, and policy updates has become a key direction for promoting the intelligent evolution of unmanned swarms. Summary of the Invention

[0006] In response to the limitations of current large-scale unmanned clusters in group division and collaboration, this paper proposes a cluster adaptive grouping method based on social organization evolution, aiming to build an efficient, flexible and dynamically adaptable collaborative framework, thereby improving the cluster's execution capabilities in complex task environments.

[0007] In human society, organizational departments gradually develop specific responsibilities over time to adapt to changes in the external environment. This division of responsibilities and evolutionary mechanisms help improve an organization's responsiveness to change, resource allocation efficiency, and information flow. Similarly, in large-scale drone swarm systems, the group structure should also be capable of dynamic evolution. The division of tasks within the swarm continuously adjusts as environmental conditions, mission requirements, and member composition change. For example, in adversarial game scenarios, such as red-blue confrontation exercises, drone swarms must continuously adjust responsibilities and information sharing mechanisms based on dynamic changes in enemy strategies. For example, a unit originally serving as a communication relay may be converted to a decoy jammer, while a reconnaissance unit may be deployed to support strike missions. The system must evaluate the marginal contribution of each unit in different roles in real time and restructure its groups and functions accordingly. This evolution of responsibilities from static homogeneous division of labor to dynamic heterogeneous collaboration significantly improves the swarm's adaptability and collaborative efficiency in complex environments.

[0008] Inspired by this, the present invention introduces a self-organizing grouping mechanism based on Shapley Group Value (SGV), which breaks through the limitations of traditional reliance on predefined groups and static responsibility allocation. Combined with the group responsibility evolution model, it guides the cluster to achieve dynamic and adaptive collaborative structure optimization during the execution process. Individuals in the cluster can independently choose to join the strategy based on their marginal contribution value in different groups, thereby forming the optimal group configuration and improving the overall collaborative efficiency. Group responsibility as the basis for division not only improves the rationality of grouping, but also provides structured support for task decomposition and coordinated control. By dynamically adjusting the group composition and function allocation, the cluster can be flexibly reorganized according to task requirements and environmental changes, thereby significantly improving the efficiency of task completion.

[0009] In real-world scenarios involving large numbers of members and diverse tasks, large-scale unmanned swarms require highly adaptable grouping and collaboration capabilities. Compared to the communication bottlenecks and response delays associated with fixed architectures, adaptive grouping mechanisms can effectively reduce system communication load, improve resource allocation efficiency, and enhance system stability and scalability. Therefore, the method proposed in this paper provides a practical solution for achieving efficient collaboration among large-scale unmanned swarms in dynamic environments.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A large-scale UAV swarm adaptive grouping method integrating responsibility evolution is proposed. The specific steps are as follows:

[0012] Step 1: Set up the drone reinforcement learning model.

[0013] The UAV swarm reinforcement learning task is represented by a tuple .in, The state space of the environment represents the state of the entire environment, including the position, speed, and state of all drones; Represents the drone's action space, including level flight, turning, climbing, diving, and holding; Represents the drone's observation space, including its own status (such as speed, heading, battery level), information about neighboring drones (such as distance), and environmental information (such as nearby obstacles and mission objectives). Rewards for the entire environment, reflecting the task completion or collaborative performance of the entire cluster in its current state; is the state transition function of the environment, which defines the state evolution law of the environment after executing the action; is the reward discount factor, which controls the degree of attenuation of future rewards; Given that each drone can only detect a small area around itself in reality, the present invention sets a partially observable multi-agent reinforcement learning environment, which increases the difficulty of drone learning.

[0014] Step 2: Interactive data collection training environment value evaluation network (SGVEnv).

[0015] The proposed environmental value evaluation network (SGVEnv) consists of an input feature encoding module, a relationship modeling module based on a multi-head self-attention mechanism, and a feed-forward reward prediction module. Its input consists of the current global state, the next global state, and the joint action of all drones. These three are concatenated along the feature dimension to form an overall input tensor. First, the input feature encoding module performs a high-dimensional mapping of the input using a fully connected layer and introduces a virtual positional encoding to enhance the perception of global structural information. Subsequently, the multi-head self-attention mechanism is used to model the interactions between drones and extract potential global collaborative features. Finally, the attention output is fed into the multi-layer perceptron of the feed-forward reward prediction module to generate reward estimates for each drone.

[0016] This paper uses the QMIX framework to complete specific tasks (such as a fixed-formation-size and fixed-model confrontation scenario). The specific steps are as follows:

[0017] (2.1) Randomly initialize the network parameters of each drone and clear the experience pool .

[0018] (2.2) In each round of interaction, Environmental status at any moment , get the current joint action , after execution, the environment state at the next moment is obtained and the corresponding global rewards .

[0019] (2.3) The quadruple , stored in the experience pool.

[0020] (2.4) When the end condition is reached (e.g., the total number of interaction steps reaches 20,000), all data are recorded and saved as .npy data files.

[0021] (2.5) Load the saved .npy data file and read all interaction samples from it, including the state , joint action , next state Corresponding global rewards .

[0022] (2.6) The In the feature dimension splicing, denoted as , as the input tensor of the SGVEnv network, As the supervision label, calculate the mean squared error loss between the predicted reward and the actual reward ,in Represents the total number of samples (ie, the total number of interaction steps):

[0023]

[0024] Step 3: Group adaptive selection update

[0025] The previous step has completed the training of SGVEnv. Next, we will implement group adaptive selection and GANet training based on this network.

[0026] GANe consists of an attention module and a linear layer, and its input is the local observation information of the current drone and the responsibility characteristics of the group. The responsibility characteristics are derived from the output of the group responsibility learning module ResNet at the previous moment. The group responsibility learning module (ResNet) dynamically generates a responsibility representation for each group. The network consists of two layers of linear modules: a state aggregation layer and a responsibility mapping layer. The input includes the local observation features and historical responsibility trajectories of the group members. First, the states of all group members are aggregated and the historical responsibility trajectory representation is generated based on the average of the past T moments in a sliding time window. Both are input into the state aggregation layer. Second, the mean and variance of the responsibility vector are generated through two parallel linear mappings in the responsibility mapping layer, and the group responsibility representation is obtained through Gaussian sampling.

[0027] The GANet network first uses an attention mechanism to evaluate the relevance between the target drone and each candidate group, extracting contextual information that represents the overall characteristics of the group. A feedforward linear layer then outputs the drone's priority score for joining each group, representing the estimated value of its participation in each group. Based on this output, the drone selects the group with the highest value as its next collaboration group.

[0028] The rationality of GANet's group selection recommendations is judged by the value benchmark provided by SGVEnv, that is, comparing the difference in group values ​​corresponding to the drones in the original group and the new recommended group.

[0029] The specific steps are as follows:

[0030] (3.1) In the initial state, each drone is defined as a group, that is, the initial group number is , numbered as ;

[0031] (3.2) Determine the influencing elements of the group self-organizing network GANet: First, the target drone Local observation of , O; second, the responsibilities and characteristics of all groups at the previous moment .

[0032] (3.3) 、 Mapped to the feature space of the same dimension to unify the representation dimension and facilitate attention modeling. As a query, As the key and value respectively. Through the attention mechanism, the semantic relevance between the target drone and each candidate group can be captured, and the contextual embedding representation of each group can be output.

[0033] (3.4) Then, the context information of each group is combined with the original local observation Splicing is performed, and the behavioral value estimation of the drone joining each candidate group is calculated through the feedforward network. The output dimension is Finally, the drone is output through the softmax layer. The probability of joining each group , select the group with the highest probability As its next step collaboration group.

[0034] (3.5) There will be three situations:

[0035] First, , drones The independent group is considered as a group, and its status information is defined as cur, and the original group status is defined as pre;

[0036] Second, The number of the group the drone currently belongs to will remove the drone The group state after is defined as pre, and the drone is retained The group status is defined as cur.

[0037] Third, The number of the group the drone currently belongs to, and the status of its original group and after joining the new group are defined as pre and cur respectively.

[0038] After getting the pre and cur status, pass them to SGVEnv to get the drone The two group states correspond to rewards and , and the difference between the two rewards As a reward for the GANet network.

[0039] (3.6) Count and accumulate the reward differences of all drone groups to guide the optimization of the GANet network:

[0040]

[0041] in, It's a drone The probability distribution of joining each group, Indicates the drone in its latest state The optimal group number.

[0042] Step 4: Evolution of group responsibilities after group status update

[0043] Step 3 has completed the adaptive update of the group. Next, the group responsibility learning module ResNet is used to complete the evolution of the new group responsibility information. The specific steps are as follows:

[0044] (4.1) Group Responsibility Learning

[0045] In order to characterize the core functions of a group under different tasks, the group responsibility vector is introduced Indicates the current Moment Group functional role.

[0046] (4.1.1) Assume Current group at the moment The local observations of all members in Input to the responsibility generation network ResNet to extract the group-level representation vector:

[0047]

[0048] in, Represents the state embedding representation of the current group, are the ResNet network parameters.

[0049] (4.1.2) To introduce the "historical consistency" mechanism in the evolution of responsibilities, the evolution trajectory of the group is constructed based on the state embedding vector. The sliding aggregation strategy within the time window is used to construct the group. The historical trajectory vector :

[0050]

[0051] (4.1.3) Embed the current state The input is fed into ResNet, which outputs the mean and variance of the responsibility distribution. The responsibility vector is obtained through a Gaussian sampling strategy. This responsibility vector represents the current functional role of the group and is continuously updated in subsequent cycles to support task collaboration, role scheduling, and organizational adaptability assessment.

[0052] (4.2) Optimization objective function design

[0053] To optimize , the present invention constructs the fitness function of responsibility evolution , whose goal is to make the responsibility vector better able to predict the future performance of the group and have the following two properties:

[0054] (4.2.1) Historical consistency: The determination of responsibilities should fully consider the historical trajectory and evolution trend of the group to ensure the integrity and interpretability of the responsibility encoding. To this end, the mutual information (MI) constraint between the historical information trajectory of the group and the current responsibility representation is introduced:

[0055]

[0056] in, is the mutual information function, is the historical trajectory set of all groups, is the set of observations for all groups.

[0057] because Difficult to calculate directly, variational inference is used to introduce a learnable approximate distribution Make a substitution and construct its lower bound (ELBO) based on Jensen inequality as the optimization objective:

[0058]

[0059] in, The training of the group history trajectory vector constructed by the sliding time window Guidance, the trajectory is constructed by averaging the group state embeddings at several past moments to ensure that the current responsibility representation is consistent and interpretable with the historical state evolution trend.

[0060] (4.2.2) Group differentiation: The responsibility vector should also have good differentiation, that is, different groups should correspond to different responsibility expressions. To this end, the group trajectory difference metric is introduced , and construct regular terms to encourage differences in responsibilities:

[0061]

[0062] in, The difference threshold set by experience can be set to 0.5; and are group numbers, and .

[0063] (4.2.3) Network optimization objective: Based on steps (4.2.1) and (4.2.2), a target optimization function for generating a responsibility network is formed:

[0064]

[0065] In order to make Closer to the true posterior distribution, during the training phase of reinforcement learning, by maximizing and The mutual information updates the parameters of the variational network , then ϕ remains unchanged throughout the evolution stage to ensure the stability of the optimization process.

[0066] (4.2.4) To improve training stability and avoid gradient interference, an alternating optimization strategy of the responsibility network and the variational network is adopted: in each training round, the responsibility network parameters are first fixed. , by minimizing the KL divergence Update variational network parameters ; Then fix the variational network and use the complete objective function Optimize responsibility network parameters.

[0067] Step 5: Information sharing mechanism after group status update

[0068] Next, the drone cluster will share information based on the grouping results of step 3. The present invention is designed as a full sharing within the group and selective sharing between groups. Full sharing within the group means that the local observations of each member in the group can be fully perceived by other intelligent agents; inspired by the fact that the communication needs and frequencies between departments with different responsibilities in the real world are different, inter-group sharing needs to be screened according to different responsibilities, and only new observations can be shared between strongly connected groups. In large-scale settings, compared with the full communication mode, this sharing mode can effectively reduce communication bandwidth and costs, and achieve efficient and economical information exchange. The specific implementation steps are as follows:

[0069] (5.1) Intra-group information fusion:

[0070] In each decision cycle, the group All members' observations are encoded by a multi-layer perceptron (MLP) to generate state representations .

[0071]

[0072] (5.2) Information fusion between groups: Calculate the gap between all group responsibility information and determine the intimacy between groups based on this value. Each group takes the first Close groups are used as information sharing targets. To ensure efficiency, a non-parametric hybrid method is used to fuse information between groups.

[0073]

[0074] Finally, for the aggregated inter-group information and information within the group Will be grouped All members in the system can sense each other, realize organized and targeted communication, and then complete the learning of drone strategies.

[0075] Beneficial effects of the present invention:

[0076] In large-scale mission environments, efficiently managing and coordinating numerous drones has become a core challenge in swarm system design. To this end, this paper proposes a dynamic, adaptive grouping method that significantly improves the system's adaptability and learning efficiency in complex environments. First, the system introduces a group status assessment mechanism, enabling each drone to receive personalized group update recommendations based on its marginal contribution to overall collaborative efficiency. This mechanism supports distributed, dynamic group reconfiguration in large-scale scenarios, effectively overcoming the inefficiencies and limitations of traditional static grouping strategies. Second, the system dynamically generates responsibility representations based on the latest group structure, enabling continuous optimization and flexible modeling of group functional roles. This mechanism supports dynamic adjustment of group responsibilities during mission evolution, significantly improving the system's decision-making capabilities and mission adaptability in heterogeneous and unstable environments. Finally, the system introduces a responsibility-based guided information fusion mechanism. Through efficient intra-group aggregation and selective inter-group interaction, it enables rapid sharing of local information, significantly reducing communication costs and avoiding the system burden caused by frequent global communication, thereby promoting the collaborative evolution of swarm strategies.

[0077] In summary, the present invention realizes cluster collaboration capabilities with scalable structure, adaptable responsibilities, and efficient communication under the multi-agent reinforcement learning framework, providing key support for building a highly collaborative intelligent system. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a framework flow chart of the present invention.

[0079] Figure 2 This is the multi-agent reinforcement learning model used in the present invention.

[0080] Figure 3 Schematic diagram of group dynamic selection designed by the present invention.

[0081] Figure 4 Schematic diagram of the group responsibility learning model designed for the present invention.

[0082] Figure 5 Schematic diagram of the group adaptive selection and communication process of the present invention. DETAILED DESCRIPTION

[0083] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0084] The method flow of the present invention is as follows Figure 1 As shown in the figure, the QMIX network architecture is used to build a UAV cluster system, and its reinforcement learning model is as follows: Figure 2 The method of the present invention is roughly divided into three submodules:

[0085] (1) Dynamic group self-organization module: This module consists of two core sub-networks: the environmental value evaluation network (SGVEnv) and the group dynamic selection network (GANet), which are used for the marginal contribution evaluation of the intelligent agent (the intelligent agent of the present invention is a drone) and the optimization of group selection decision. First, the environmental value evaluation network (SGVEnv) is a reward modeling network used in drone cluster systems, which aims to evaluate the overall task value under different group configurations. The training process of SGVEnv relies on the state, observation, action and reward data collected in the specified task scenario to learn the impact of each group configuration on the overall task value. After training, the network can output the reward prediction value for the current group configuration based on the current group state and joint action input, that is, the group value estimate. By comparing the value differences under group configurations, the system can evaluate the contribution of individuals joining a certain group to the overall performance, thereby guiding drones to autonomously select the optimal collaborative group to improve the overall task execution efficiency and collaborative ability of the cluster. Second, the group dynamic selection network (GANet) is used to guide the intelligent agent to autonomously choose to retain the original group or switch to a more suitable group, thereby realizing self-organizing evolution within the cluster.

[0086] The rationality of GANet's group selection recommendations is evaluated using a value benchmark provided by SGVEnv. This benchmark compares the difference in group value between the agents in the original group and the newly recommended group. This value difference not only reflects the impact of individual reorganization on the overall effectiveness of the cluster but also serves as a learning signal for GANet, driving the network to continuously optimize its group recommendation strategy. By continuously perceiving, judging, and adjusting its own group affiliation in a multi-agent environment, GANet enables the system to possess dynamic self-organizing capabilities, thereby continuously converging towards a more optimal collaborative structure and greater cluster adaptability during task execution.

[0087] (2) Group Responsibility Learning Module:

[0088] In large-scale unmanned swarm collaboration tasks, groups need to assume differentiated functions to adapt to the dynamic changes in complex environments and the diverse needs of task decomposition. To achieve continuous modeling and guidance of group functions, the present invention designs a group responsibility learning module, ResNet, which is used to dynamically generate a responsibility vector for each group that represents its current functional characteristics, providing structured semantic support for subsequent group self-organization decision-making and strategy learning processes. This responsibility vector is periodically updated as the task is executed, and can continuously reflect the core functions undertaken by the group in the collaborative task, thereby enhancing the system's responsiveness and organizational adaptability when facing task switching or sudden changes in the environment.

[0089] (3) Group information sharing module: The present invention divides communication into two categories: intra-group and inter-group. A fully connected communication mode is adopted within the group, and a commander is randomly designated in each decision cycle to be responsible for summarizing the observation information of group members and generating a group fusion representation, while also assuming the responsibility of external communication. Inter-group communication determines whether to establish a connection based on the semantic difference of the responsibility vector. The system selects the top 10% of the closest groups in each group as communication objects. After the commanders complete the information fusion, they distribute it to their respective group members, realizing efficient and low-overhead local information sharing.

[0090] The present invention adopts the traditional deep reinforcement learning algorithm framework (QMIX) to solve the strategy of a single agent in a multi-agent environment, enhances the collaboration of drone clusters through adaptive grouping, and accelerates the task convergence speed.

[0091] This example, based on a "12v12 confrontation game," demonstrates the present invention's strategy training process for a Red drone swarm in a large-scale drone swarm system. The Red and Blue teams each deploy 12 drones, engaging in a confrontational game in unobstructed airspace. The goal is to maximize the destruction of enemy units while maintaining the survival of their own team. The Blue team's drones use a fixed strategy, while the Red team's strategy is trained and optimized by this system.

[0092] (1) Initialization process: Initialize the policy network and hybrid network parameters of each red drone and clear the experience pool.

[0093] (2) Data collection and training environment evaluation module: Through multiple rounds of red-blue confrontation data collection, the environment evaluation module SGVEnv is constructed and trained to output the reward signal of the current red team group structure state.

[0094] (3) Main loop process:

[0095] (3.1) Observation Acquisition: Each Red Team UAV obtains local observation information such as its own state (such as coordinates, speed, and health points) and the position and number of enemy and friendly units within its field of view at each time step.

[0096] (3.2) Group self-organization: Figure 3 As shown in the figure, the system calls the GANet module to generate the group selection probability of each UAV based on the current observation input and group responsibility information, and forms the adaptive group division structure of the current red team cluster.

[0097] (3.3) Responsibility learning and evolution: Figure 4 As shown in Figure 2, based on the current group division, the system calls the responsibility network ResNet to output the functional responsibility representation of each group, and combines the historical responsibility trajectory to form a stable and differentiated responsibility embedding to facilitate strategic division of labor.

[0098] (3.4) Intra-group and inter-group communication: The command drone aggregates member observations within the group to generate a representation of the group's fusion state; inter-group communication is established based on the semantic similarity of responsibilities, and information transmission is only established between a few high-intimacy groups to avoid redundant communication.

[0099] (3.5) Strategy feedback and action execution: The fusion information input corresponds to the red team's strategy network, outputs the current action and executes it uniformly. The system then receives the reward and the next state.

[0100] (3.6) Experience Update and Strategy Optimization: The interaction information of each round is stored in the experience pool. The hybrid network is used to calculate the joint value function and perform parameter optimization to improve the overall strategic performance of the red team.

[0101] After the training reaches the preset rounds, the system outputs the final red team strategy model, which has adaptive grouping capabilities, dynamic division of responsibilities and efficient information fusion mechanism.

[0102] like Figure 5 As shown, this example is set up for a 12v12 drone-to-drone game, with the Red team serving as the training drone in the system. Each drone acquires its own state information through sensors and feeds it into the group dynamic selection network (GANet), which dynamically and adaptively groups the drones into five functional groups. UAVs within a group can fully communicate with each other, and the solid lines between groups indicate inter-group information fusion channels. The resulting group structure is further fed into the Responsibility Learning Module (ResNet) to exploit the differentiated division of labor within each group within the task, thereby enhancing the adaptability and collaborative efficiency of the grouping strategy.

Claims

1. A large-scale UAV swarm adaptive grouping method integrating responsibility evolution, characterized by: The specific steps are as follows: Step 1: Set up the drone reinforcement learning model; The UAV swarm reinforcement learning task is represented by a tuple ;in, The state space of the environment represents the state of the entire environment, including the position, speed, and state of all drones; Represents the drone's action space, including level flight, turning, climbing, diving, and holding; Represents the drone's observation space, including its own status, neighboring drone information, and environmental information; Rewards for the entire environment, reflecting the task completion or collaborative performance of the entire cluster in its current state; is the state transition function of the environment, which defines the state evolution law of the environment after executing the action; is the reward discount factor, which controls the degree of attenuation of future rewards; is the number of drones in the environment; Step 2: Interactive data collection and training of the environment value evaluation network SGVEnv; The environmental value assessment network SGVEnv consists of an input feature encoding module, a relationship modeling module based on a multi-head self-attention mechanism, and a feedforward reward prediction module. Its input consists of the current global state, the next global state, and the joint action of all drones, which are spliced ​​in the feature dimension to form an overall input tensor. First, the input feature encoding module performs high-dimensional mapping on the input through a fully connected layer and introduces a virtual position encoding to enhance the perception of global structural information. Then, the multi-head self-attention mechanism is used to model the interaction relationship between drones and extract potential global collaborative features. Finally, the attention output is fed into the multi-layer perceptron of the feedforward reward prediction module to generate the reward estimate corresponding to each drone. The specific steps are as follows: (2.1) Randomly initialize the network parameters of each drone and clear the experience pool ; (2.2) In each round of interaction, Environmental status at any moment , get the current joint action , after execution, the environment state at the next moment is obtained and the corresponding global rewards ; (2.3) The quadruple , stored in the experience pool; (2.4) When the end condition is reached, record all data and save them as .npy data files; (2.5) Load the saved .npy data file and read all interaction samples from it, including the state , joint action , next state Corresponding global rewards ; (2.6) The In the feature dimension splicing, denoted as , as the input tensor of the SGVEnv network, As the supervision label, calculate the mean squared error loss between the predicted reward and the actual reward ,in Representative total sample size: ， Step 3: Group adaptive selection update Next, we will implement adaptive group selection and GANet training based on the SGVEnv network; GANe consists of an attention module and a linear layer, with inputs including the local observation information of the current drone and the responsibility characteristics of the group. The responsibility characteristics are derived from the output of the group responsibility learning module ResNet at the previous moment. The group responsibility learning module ResNet dynamically generates responsibility representations for each group. The network consists of two layers of linear modules: a state aggregation layer and a responsibility mapping layer. The inputs include the local observation features and historical responsibility trajectories of group members. First, the states of all group members are aggregated and the historical responsibility trajectory representation is generated based on the average of the past T moments in a sliding time window. Both are input into the state aggregation layer. Second, the mean and variance of the responsibility vector are generated through two parallel linear mappings in the responsibility mapping layer, and the group responsibility representation is obtained through Gaussian sampling. The GANet network first uses an attention mechanism to evaluate the relevance between the target drone and each candidate group and extracts contextual information representing the overall characteristics of the group. Then, a feedforward linear layer outputs the drone's priority score for joining each group, i.e., the estimated value of its participation in each group. Based on this output, the drone selects the group with the highest value as the next collaboration point. The rationality of GANet's group selection recommendations is judged by the value benchmark provided by SGVEnv, which compares the difference in group value between the drones in the original group and the new recommended group. The specific steps are as follows: (3.1) In the initial state, each drone is defined as a group, that is, the initial group number is , numbered as ; (3.2) Determine the influencing elements of the group self-organizing network GANet: First, the target drone Local observation , O; second, the responsibilities and characteristics of all groups at the previous moment ; (3.3) 、 Mapped to the feature space of the same dimension to unify the representation dimension and facilitate attention modeling; As a query, As key and value respectively; through the attention mechanism, the semantic relevance between the target drone and each candidate group is captured, and the contextual embedding representation of each group is output; (3.4) Then, the context information of each group is combined with the original local observation Splicing is performed, and the behavioral value estimation of the drone joining each candidate group is calculated through the feedforward network. The output dimension is ; Finally, the drone is output through the softmax layer The probability of joining each group , select the group with the highest probability As its next step collaborative group; (3.5) There will be three situations: First, , drones The independent group is considered as a group, and its status information is defined as cur, and the original group status is defined as pre; Second, The number of the group the drone currently belongs to will remove the drone The group state after is defined as pre, and the drone is retained The group status is defined as cur; Third, The number of the group the drone currently belongs to, and the status of its original group and after joining the new group are defined as pre and cur respectively; After getting the pre and cur status, pass them to SGVEnv to get the drone The two group states correspond to rewards and , and the difference between the two rewards As a reward for the GANet network; (3.6) Count and accumulate the reward differences of all drone groups to guide the optimization of the GANet network: ， in, It's a drone The probability distribution of joining each group, Indicates the drone in its latest state The optimal group number; Step 4: Evolution of group responsibilities after group status update The group responsibility learning module ResNet is used to complete the evolution of the new group responsibility information. The specific steps are as follows: (4.1) Group Responsibility Learning In order to characterize the core functions of a group under different tasks, the group responsibility vector is introduced Indicates the current Moment Group functional roles; (4.1.1) Assume Current group at the moment The local observations of all members in ; Input to the responsibility generation network ResNet to extract the group-level representation vector: ， in, Represents the state embedding representation of the current group, is the ResNet network parameter; (4.1.2) To introduce the "historical consistency" mechanism in the evolution of responsibilities, the evolution trajectory of the group is constructed based on the state embedding vector; the sliding aggregation strategy within the time window is used to construct the group The historical trajectory vector : ， (4.1.3) Embed the current state The input is fed into ResNet, which outputs the mean and variance of the responsibility distribution. A Gaussian sampling strategy is used to obtain a responsibility vector, which represents the current functional role of the group and is continuously updated in subsequent cycles to support task collaboration, role scheduling, and organizational adaptability assessment. (4.2) Optimization objective function design To optimize , construct the fitness function of responsibility evolution , whose goal is to make the responsibility vector better able to predict the future performance of the group and have the following two properties: (4.2.1) Historical consistency: Introduce the mutual information constraint between the group's historical information trajectory and the current responsibility representation: ， in, is the mutual information function, is the historical trajectory set of all groups, is the set of observations for all groups; Introducing learnable approximate distributions using variational inference Make a substitution and construct its lower bound as the optimization objective based on Jensen inequality: ， (4.2.2) Group Distinction: Introducing Group Trajectory Difference Metrics , and construct regular terms to encourage differences in responsibilities: ， in, the difference threshold set for experience; and are group numbers, and ; (4.2.3) Network optimization objective: Based on steps (4.2.1) and (4.2.2), a target optimization function for generating a responsibility network is formed: ， (4.2.4) Adopt the alternating optimization strategy of the responsibility network and the variational network: in each training round, first fix the responsibility network parameters , by minimizing the KL divergence Update variational network parameters ; Then fix the variational network and use the complete objective function Optimize responsibility network parameters; Step 5: Information sharing mechanism after group status update The drone cluster will share information based on the grouping results of step 3, including full sharing within the group and selective sharing between groups. The specific implementation steps are as follows: (5.1) Intra-group information fusion: In each decision cycle, the group All members observe and generate state representation through multi-layer perceptron MLP encoding ; ， (5.2) Information fusion between groups: Calculate the gap between all group responsibility information and determine the intimacy between groups based on this value; each group takes the first Close groups are used as information sharing targets; to ensure efficiency, non-parametric mixing is used to fuse information between groups; ， Finally, for the aggregated inter-group information and information within the group Will be grouped All members in the system perceive and communicate with each other, thereby completing the learning of drone strategies.

Images