Multi-agent collaborative framework based on dark tactics under intermittent communication

Abstract

The invention relates to a multi-agent collaborative framework based on dark tactics under intermittent communication, and belongs to the technical field of multi-agent reinforcement learning. In order to solve the problem of multi-agent collaborative instability caused by intermittent communication, the invention provides a new'dark tactical 'collaborative mechanism, and designs an ACA three-layer collaborative architecture (Aggregate-Commander-Actor). The architecture comprises: 1) an aggregator module, which compensates information loss caused by communication loss by fusing local observation information obtained by intermittent communication, and enhances the situation awareness ability of an intelligent agent; 2) a commander module which integrates local observation and aggregation information to generate a dark tactical strategy, and optimizes group cooperation efficiency through dynamic role allocation; and 3) an actuator module which generates adaptive actions based on role allocation and local observation, and improves the robustness of the system in a communication limited environment. Experiments show that in an intermittent communication scene, the cooperative stability performance of the multi-agent system is remarkably improved, and compared with a current optimal baseline, the cooperative efficiency is improved by 11.1% at most.

Description

Technical Field

[0001] The present invention relates to a multi-agent collaborative framework based on dark tactics under intermittent communication, belonging to the technical field of multi-agent reinforcement learning. Background Art

[0002] In recent years, multi-agent reinforcement learning (MARL) has become an important method for solving complex decision-making problems and has been widely used in fields such as robotic control, autonomous driving, game AI, and social networks. The core goal of multi-agent collaboration is to enable multiple agents to work together efficiently in complex environments to achieve overall optimization. However, in most real-world applications, agents typically only have access to local observations and lack direct access to the global state. Therefore, information sharing between agents is crucial. This demand has driven the research and development of communication-based multi-agent reinforcement learning (Comm-MARL).

[0003] Existing Comm-MARL methods typically rely on advanced technologies such as Graph Neural Networks (GNNs) and attention mechanisms (Transformers) to process information from different teammates at each decision time step. These methods can effectively improve the collaborative ability of agents in an environment with stable communication, but in real-world scenarios, communication is often intermittent, that is, agents may not be able to stably receive information from teammates at certain time steps. This intermittent communication problem is prevalent in many practical applications, such as the coordinated control of drone swarms in complex electromagnetic environments, the interactive decision-making of autonomous driving fleets in areas with limited wireless signals, and the information transmission of multi-robot systems in remote operations. The core challenge of intermittent communication lies in the uncertainty and discontinuity of information exchange, which significantly weakens the collaborative effectiveness of multi-agent systems, mainly reflected in the following aspects:

[0004] (1) Information loss and incompleteness: Due to factors such as bandwidth limitations, signal interference, and network delays, the agent may not be able to obtain any information about teammates at certain time steps, or may only receive partial information. This incomplete information directly affects the agent's understanding of the global state, thereby weakening its decision-making ability. Figure 1The figure shows experimental results for QMIX, a widely used multi-agent reinforcement learning method, under intermittent communication conditions. As can be seen, when agents are completely unable to communicate, their collaborative performance, while limited, still maintains a certain degree of stability. However, in an environment with intermittent communication, QMIX's collaborative effectiveness decreases, even falling below that of a complete absence of communication. This phenomenon suggests that in an intermittent communication environment, agents may mistakenly rely on information received through intermittent communication, leading to confused decisions and seriously compromising overall collaborative capabilities.

[0005] (2) Difficulty in information fusion: Existing Comm-MARL methods typically assume that agents can stably receive and process information from teammates, and therefore rely on fixed-structure data input for information fusion. However, in intermittent communication scenarios, the amount and structure of information obtained by agents changes dynamically, making existing methods difficult to adapt. This leads to a decrease in the reliability of information fusion, thereby weakening the ability of agents to build stable collaborative strategies. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-agent collaborative framework based on dark tactics under intermittent communication to solve the problem of reduced overall collaborative ability of agents due to incomplete information loss caused by intermittent communication.

[0007] In order to solve the above technical problems, the present invention provides a multi-agent collaborative framework based on dark tactics under intermittent communication. The collaborative framework is equipped with an aggregator module, a commander module and an actuator module based on the QMIX framework.

[0008] The aggregator module is used to perform feature aggregation on the local observation information obtained by each agent under intermittent communication conditions to obtain aggregated information of each agent;

[0009] The commander module is used to perform tactical learning based on the aggregated information of each agent output by the aggregator module, and assign roles to each agent based on the learning results;

[0010] The actuator module is used to determine the action of each agent based on the role of each agent and its local observation information.

[0011] Furthermore, the aggregator module uses a multi-head attention mechanism to perform feature aggregation on the local observation information obtained by each agent. The local observation information obtained by the agent includes the local observation information of the agent itself and the local observation information received by the agent from other agents. The process is as follows:

[0012] Each agent's own local observation information is linearly projected through the transformation matrix to generate a query vector; the local observation information received by each agent from other agents is mapped into a key vector and a value vector, and an attention weight is obtained in each attention head based on the key vector and the value vector. The attention weight is used to describe the attention weight of the current agent to the information of other agents;

[0013] The outputs of the attention heads are concatenated and processed through a multi-layer perceptron to obtain the final aggregated result of the current agent.

[0014] Furthermore, the aggregated information of each agent obtained by the aggregator module is:

[0015]

[0016] where e i is the aggregate information of agent i, o -i is the local observation information received by agent i from other agents, o i is the local observation information of agent i itself, is the attention weight corresponding to agent i, is a value vector, and MLP() is processed by a multi-layer perceptron.

[0017] Furthermore, the commander module uses MLP and GRU to process the aggregated information obtained by the aggregator module. The process is as follows:

[0018] Use MLP to process the aggregated information obtained by the aggregator module, and use GRU to fuse the output generated by MLP with the historical hidden state;

[0019] The fusion results obtained by GRU are used to calculate the role probability distribution, and the role probability distribution is used to assign roles to each agent.

[0020] Furthermore, the calculation formula used for the role probability distribution is:

[0021]

[0022] in is the fusion result obtained by GRU, MLP() is processed by multi-layer perceptron, W ρ It is a trainable parameter, and softmax() indicates processing through the activation function layer.

[0023] Furthermore, the actuator module uses MLP and GRU to determine the action of the agent, and the process includes:

[0024] The role of the agent is mapped to a continuous space to obtain the role representation of the agent, the role representation of the agent is concatenated with the local observation information of the agent itself, and the MLP is used to extract features from the concatenated information;

[0025] GRU is used to process the features extracted by MLP and the historical hidden state to obtain the current hidden state, and the current hidden state is mapped to a vector consistent with the dimension of the action space, where each element corresponds to the Q value of the action that the agent can perform. The action of the agent is determined based on the Q value of the action that the agent can perform.

[0026] Furthermore, the multi-agent collaborative framework adopts an alternating training strategy during the training process, which means keeping the parameters of the actuator module fixed while training the commander module and the aggregator module.

[0027] Furthermore, the multi-agent collaborative framework uses mean squared error loss during training to align the outputs of the aggregator and commander modules to the outputs of their corresponding aggregator* and commander* modules under stable communication:

[0028]

[0029] e i and Aggregate information generated by the aggregator module and aggregator* modules respectively, and Role assignment probability distributions calculated for the Commander module and Commander* module, where the Aggregator* module and Commander* module refer to the modules with the same structures as the Aggregator module and Commander module used under stable communication conditions, respectively.

[0030] The beneficial effects of the present invention are as follows: as an improved invention, the present invention sets an aggregator module, a commander module and an actuator module on the basis of the QMIX framework; the aggregator module is used to perform feature aggregation on the local observation information obtained by each agent under intermittent communication conditions to obtain the aggregated information of each agent, thereby alleviating the impact of communication loss and improving the integrity of the information; the commander module is used to perform tactical learning on the aggregated information, and a role is assigned to each agent according to the learning results, thereby enhancing the tactical coordination of the agents; the actuator module is used to determine the actions of each agent according to the role of each agent and its local observation information, thereby improving the adaptability and robustness of the system. Therefore, the present invention can effectively utilize existing information to improve the collaborative ability of agents under intermittent communication conditions, reduce the impact of communication instability on decision-making, and improve the overall collaborative ability of agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1This is a schematic diagram of the experimental results corresponding to multi-agent reinforcement learning using the QMIX framework under intermittent communication conditions;

[0032] Figure 2 It is a schematic diagram of the multi-agent collaborative framework based on dark tactics under intermittent communication of the present invention. DETAILED DESCRIPTION

[0033] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0034] Based on the QMIX framework, this paper establishes an aggregator module, a commander module, and an actuator module. The aggregator module aggregates the features of local observation information obtained by each agent under intermittent communication conditions. The commander module assigns roles to each agent based on local observations and aggregated global information. The actuator module determines each agent's action based on its role and its local observation information. By introducing role representation learning and information aggregation mechanisms, this paper effectively alleviates decision-making instability caused by information loss and improves the collaborative efficiency and robustness of multi-agent systems.

[0035] Implementation of a multi-agent collaborative framework based on dark tactics under intermittent communication

[0036] The present invention is based on the QMIX framework, such as Figure 2 As shown in the figure, three core modules are introduced: the Aggregator module, the Commander module, and the Actor module to optimize the collaborative capabilities of intelligent agents under intermittent communication conditions. The Aggregator module is used to mitigate the impact of communication loss and improve information integrity through feature extraction and fusion mechanisms. The Commander module combines local observations and global information to assign roles based on learnable dark tactics, thereby enhancing the tactical coordination of intelligent agents. The Actor module uses role information to guide decision-making and optimizes the collaboration between intelligent agents through reinforcement learning strategies, ultimately improving the adaptability and robustness of the system.

[0037] Before describing the various modules of the collaborative framework in detail, we will now describe the multi-agent collaborative task under intermittent communication conditions. This paper models the multi-agent collaborative task under intermittent communication conditions as a decentralized partially observable Markov decision process (Dec-POMDP), which is defined by the following tuple:

[0038] G=<N,S,A,P,R,Ω,O,n,γ>

[0039] Among them, N is a set of n agents, S represents the global state of the environment, γ is the discount factor, and at each time step t, each agent i∈{1,2,…,n} obtains its own local observation information o according to the observation function Ω(s,i) i ∈O, and choose the corresponding action a accordingly i ∈A, therefore, the joint action of all agents is expressed as:

[0040]

[0041] Subsequently, the environment state is updated according to the state transition function P(s′|s,a) and transferred to the new state s′. Finally, all agents receive rewards from the environment together:

[0042] r=R(s,a)

[0043] 1. Use the aggregator module to perform feature aggregation on the local observation information obtained by each agent under intermittent communication conditions.

[0044] The aggregator module of the present invention utilizes a multi-head attention mechanism to perform feature aggregation on the local observation information obtained by each agent. The local observation information obtained by the agent includes the local observation information of the agent itself and the local observation information received by the agent from other agents.

[0045] At each time step t, due to unstable communication and information loss, agent i may not be able to receive all the information from other teammates (agents). Assume that ω% of the information is not received. The present invention uses a binary vector Indicates this situation, where the element value is 1 indicates that the information is successfully received, and the element value is 0 indicates that the information is lost. Therefore, the information obtained by agent i at each time step can be expressed as:

[0046]

[0047] Among them, all is the information sent by all agents to agent i. To this end, the aggregator module of the present invention introduces the Multi-Head Attention (MHA) mechanism to aggregate information features to effectively integrate the communication information obtained by agents under intermittent communication conditions. Specifically, each agent i’s own local observation o i First, through a learnable transformation matrix Perform linear projection to generate a query vector; at the same time, the local observation o received by the agent from other agents -i Through the learnable matrix and Mapped into key and value vectors. In each attention head, the query vector and the key vector are inner-producted to obtain the attention weight This weight measures the current agent's attention to the information of other agents and is used to weight the information received. The specific attention weight calculation formula is as follows:

[0048]

[0049] Among them, d K is the dimension of the key, which plays a scaling role to avoid vanishing gradients. In this way, the local observations of each agent i are matched with the local observations from other agents to generate corresponding attention weights. Next, the outputs of all attention heads are spliced ​​and further processed through an MLP (multi-layer perceptron) to obtain the final aggregated result e for the current agent. i , the formula is:

[0050]

[0051] 2. Use the commander module to assign appropriate roles to each agent to achieve coordinated control among agents.

[0052] The commander module of the present invention uses MLP and GRU to process the aggregated results e i Processing is performed to generate stable dark tactics. The specific process includes:

[0053] First, use MLP to process the aggregated results e i After processing, the result is MLP(e i ), and then uses the MLP processing results and historical hidden states as the input of the GRU (Gated Recurrent Unit). Through GRU processing, the current information and historical data are effectively integrated. This processing method captures temporal dependencies to improve the stability and relevance of the dark strategy. The calculation formula is as follows:

[0054]

[0055] Use the above calculation results to calculate the role probability distribution and assign roles to each agent:

[0056]

[0057] Among them, W ρ It is a trainable parameter. During the training phase, the ε-greedy strategy is used to dynamically allocate the roles of the agents, which is used to balance exploration and exploitation:

[0058]

[0059] Among them, Ψ represents the role, Indicates selecting the character with the highest probability from the character set. During the testing phase, it is generally desirable for the agent to minimize or completely eliminate exploration. Setting ε = 0 ensures that the agent always selects the character with the highest estimated probability based on the learned strategy. To distinguish the behavioral models represented by different characters, the present invention maps discrete characters to a continuous space. This embodiment is implemented using a function Z = f(Ψ), which can be composed of a fully connected layer and a ReLU activation function.

[0060] 3. Use the actuator module to determine the agent's actions based on the agent's local observation information and assigned role.

[0061] Action a of agent i i By its local observation information o i With the role of z i ∈Z decision, in this embodiment, the actuator module uses MLP and GRU to determine the action of the agent. The process includes:

[0062] First, the role of agent i is represented by z i Instead of local observation o i Splice and merge the input [o i ,z i ] is fed into the MLP for feature extraction. Subsequently, the output of the MLP is further processed by the GRU to generate the hidden state:

[0063]

[0064] in, Represents the hidden state of GRU at the previous time step. Mapped to a vector consistent with the dimension of the action space, where each element corresponds to the Q value of the action that the agent can perform, which is used to guide decision selection:

[0065]

[0066] In the same way as selecting the role, the ε-greedy strategy is used to determine the action of the agent based on the Q value.

[0067] It can be seen that the aggregator module, commander module and actor module introduced in the present invention can alleviate the impact of communication loss and improve the integrity of information. They can also perform dark tactical learning on the aggregated information and assign roles to each intelligent agent based on the learning results, thereby enhancing the tactical coordination of the intelligent agents, and can improve the adaptability and robustness of the system, and improve the collaborative performance of each intelligent agent under intermittent communication.

[0068] Since the present invention is based on the existing QMIX framework, which is a reinforcement learning framework for multi-agent collaboration, the core idea is to achieve efficient training by decomposing the joint action value function. Based on the QMIX framework, the present invention introduces role information and attention mechanism to optimize the collaborative decision-making ability of the agents, such as Figure 2 As shown. QMIX nonlinearly combines individual Q values ​​into a global Q value Q through a mixing network. tot The global state s is used to dynamically adjust the network parameters to ensure that the monotonicity constraint is satisfied. Since it is difficult to characterize the collaborative relationship between roles by relying solely on the global state, it is necessary to integrate role representations in the hybrid network so that the global state and role information are combined in the weight generation process to achieve optimized policy coordination.

[0069] On this basis, the present invention introduces the attention mechanism to enable the global state to focus on different role modes, thereby improving the expression ability of value decomposition. First, use GRU to analyze the historical global state (s 0 ,s 1 ,…,s t ) is encoded to obtain time series embedding Then, using s as the query, the role embedding As key and value, calculate the attention weight:

[0070]

[0071] ξ=Concat(ξ 1 ,ξ 2 ,…,ξ L′ )W o

[0072] Here, l′ (l′∈1,2,...,L′) represents the output of each attention head, The final attention output is combined with s to dynamically adjust the hybrid network parameter β through a hypernetwork, allowing the system to adaptively adjust the Q value calculation according to different role configurations and enhance coordination.

[0073] The training process of the present invention includes three stages, wherein the first stage is the optimization of the actuator module; the second stage is the optimization of the aggregator* module and the commander* module, wherein the aggregator* module and the commander* module refer to modules with the same structure as the aggregator module and the commander module used under stable communication conditions; the third stage only trains the aggregator module and the commander module, using the loss functions L1 and L2 for training, so that the aggregator module and the commander module fit the aggregator* module and the commander* module respectively.

[0074] During the training process, in order to optimize the actuator module to make it learn the optimal strategy, the present invention adopts the temporal difference (TD) loss function:

[0075]

[0076] Here, η is the parameter of the actuator module, is the target network, s′ and a′ are the state and joint action at the next time step. The expectation is estimated using uniform samples in the same replay buffer D.

[0077] After the commander module determines the role ρ of the agent, the corresponding role representation z is retrieved from the predefined representation space and input to the actuator module. Since the role selection is a non-differentiable discrete process, the gradient cannot be directly returned to the commander module for optimization during model training. rl (η, β) cannot perform gradient updates on the commander module. Inspired by the training method of generative adversarial networks (GANs), this paper adopts an alternating training strategy during the training process, that is, keeping the parameters of the actuator module fixed while optimizing the commander module and the aggregator module. The corresponding loss function is as follows:

[0078]

[0079] To enhance the system's robustness under intermittent communication conditions, a mean squared error (MSE) loss is used to align the outputs of the Aggregator and Commander modules with those of the Aggregator* and Commander* modules under stable communication. Aggregator* and Commander* modules are modules with the same structure as those used under stable communication. By minimizing the difference in their outputs, the Aggregator module under intermittent communication learns the information representation aggregated by the Aggregator* module under stable communication, while the Commander module draws on the Commander* module's precise role allocation strategy. This narrows the performance gap between intermittent and stable communication scenarios, improving the system's collaborative consistency and adaptability.

[0080]

[0081] Among them, e i and Aggregate information generated by the aggregator module and aggregator* modules respectively, and The role assignment probability distributions computed by the Commander and Commander* modules are used to minimize these losses, ensuring that the agents can still collaborate well under communication constraints.

[0082] In order to reduce the fluctuations caused by intermittent communication and improve the model's adaptability to discontinuous data streams, this paper introduces a supervisory signal based on stable communication: During the training phase, considering that the input of stable communication may contain noise interference, this paper introduces a learnable Gaussian noise perturbation at the input layer of the aggregator* module. The formula is as follows:

[0083]

[0084] Among them, μ(e i ) and σ(e i ) are respectively composed of e i The calculated mean and standard deviation vectors, is a noise vector sampled from a standard normal distribution, and ⊙ represents element-wise multiplication. This processing reduces fluctuations caused by intermittent information and improves model stability.

[0085] During the testing phase, the fused information e generated by the aggregator module and the role representation Z generated by the commander module are directly used as the input of the actuator module, enabling the agent to make efficient decisions under different communication conditions and enhancing the robustness and adaptability of the system.

[0086] In response to the problem of multi-agent collaborative instability caused by intermittent communication, the present invention proposes a new "dark tactics" collaborative mechanism and designs an ACA three-layer collaborative architecture (Aggregator-Commander-Actor). The architecture includes an aggregator module, a commander module and an actuator module. The aggregator module compensates for the information loss caused by the lack of communication by fusing the local observation information obtained by intermittent communication, thereby enhancing the situational awareness capability of the intelligent agent; the commander module is used to integrate local observations and aggregated information to generate dark tactical strategies, and optimize the group collaborative effectiveness through dynamic role allocation; the actuator module is used to generate adaptive actions based on assigned roles and local observations, thereby improving the robustness of the system in a communication-restricted environment. Experiments show that in intermittent communication scenarios, the present invention significantly improves the collaborative stability performance of the multi-agent system, and the collaborative efficiency is increased by up to 11.1% compared to the current optimal baseline.

Claims

1. A multi-agent collaborative framework based on dark tactics under intermittent communication, characterized by: The collaborative framework sets up aggregator module, commander module and executor module based on the QMIX framework; The aggregator module is used to perform feature aggregation on the local observation information obtained by each agent under intermittent communication conditions to obtain aggregated information of each agent; The commander module is used to perform tactical learning based on the aggregated information of each agent output by the aggregator module, and assign roles to each agent based on the learning results; The actuator module is used to determine the action of each agent based on the role of each agent and its local observation information.

2. The multi-agent collaborative framework based on dark tactics under intermittent communication according to claim 1 is characterized in that The aggregator module uses a multi-head attention mechanism to perform feature aggregation on the local observation information obtained by each agent. The local observation information obtained by the agent includes the local observation information of the agent itself and the local observation information received by the agent from other agents. The process is as follows: Each agent's own local observation information is linearly projected through the transformation matrix to generate a query vector; the local observation information received by each agent from other agents is mapped into a key vector and a value vector, and an attention weight is obtained in each attention head based on the key vector and the value vector. The attention weight is used to describe the attention weight of the current agent to the information of other agents; The outputs of the attention heads are concatenated and processed through a multi-layer perceptron to obtain the final aggregated result of the current agent.

3. The multi-agent collaborative framework based on dark tactics under intermittent communication according to claim 2 is characterized in that The aggregate information of each agent obtained by the aggregator module is: where e i is the aggregate information of agent i, o -i is the local observation information received by agent i from other agents, o i Agent i’s own local observation information, is the attention weight corresponding to agent i, is a value vector, and MLP() is processed by a multi-layer perceptron.

4. The multi-agent collaborative framework based on dark tactics under intermittent communication according to claim 1 is characterized in that The commander module uses MLP and GRU to process the aggregated information obtained by the aggregator module. The process is as follows: Use MLP to process the aggregated information obtained by the aggregator module, and use GRU to fuse the output generated by MLP with the historical hidden state; The fusion results obtained by GRU are used to calculate the role probability distribution, and the role probability distribution is used to assign roles to each agent.

5. The multi-agent collaborative framework based on dark tactics under intermittent communication according to claim 4 is characterized in that: The calculation formula used for the role probability distribution is: in is the fusion result obtained by GRU, MLP() is processed by multi-layer perceptron, W ρ It is a trainable parameter, and softmax() indicates processing through the activation function layer.

6. The multi-agent collaborative framework based on dark tactics under intermittent communication according to claim 1 is characterized in that The actuator module uses MLP and GRU to determine the action of the agent. The process includes: The role of the agent is mapped to a continuous space to obtain the role representation of the agent, the role representation of the agent is concatenated with the local observation information of the agent itself, and the MLP is used to extract features from the concatenated information; GRU is used to process the features extracted by MLP and the historical hidden state to obtain the current hidden state, and the current hidden state is mapped to a vector consistent with the dimension of the action space, where each element corresponds to the Q value of the action that the agent can perform. The action of the agent is determined based on the Q value of the action that the agent can perform.

7. The multi-agent collaborative framework based on dark tactics under intermittent communication according to any one of claims 1 to 6, characterized in that: This multi-agent collaborative framework adopts an alternating training strategy during training, which means keeping the parameters of the actuator module fixed while training the commander module and the aggregator module.

8. The multi-agent collaborative framework based on dark tactics under intermittent communication according to claim 7 is characterized in that: The multi-agent collaborative framework uses mean squared error loss during training to align the outputs of the aggregator and commander modules to the outputs of their corresponding aggregator* and commander* modules under stable communication: e i and Aggregate information generated by the aggregator module and aggregator* modules respectively, and Role assignment probability distributions calculated for the Commander module and Commander* module, where the Aggregator* module and Commander* module refer to the modules with the same structures as the Aggregator module and Commander module used under stable communication conditions, respectively.

Images