Multi-agent global consensus cooperation method based on progressive communication

Abstract

The invention discloses a multi-agent global consensus cooperation method based on progressive communication, and relates to the technical field of multi-agent deep reinforcement learning. The method comprises the following steps: updating self-cognition # imgabs0 # of an intelligent agent i through a coding module; generating a weight vector alpha i and communication information # imgabs1 # through an attention module; grouping the intelligent agents according to the similarity of the weight vector alpha i, and generating communication masks MASK1 and MASK2; the communication module takes the information # imgabs2 # and the mask MASK1 as the input of a local visual converter to generate a local consensus hL in the first stage, and takes the local consensus hL and the mask MASK2 as the input of a global visual converter to promote formation of a global consensus hG in the second stage; the Q-value module takes the self-cognition # imgabs3 #, the local consensus # imgabs4 # and the global consensus # imgabs5 # of the agent i as input to generate a Q-value function # imgabs6 #, and a KL divergence module and a reconstruction module are established to promote alignment of the global consensus among multiple agents. The method has the advantage that a progressive multi-agent communication method is provided to gradually promote the achievement of the global consensus.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent deep reinforcement learning, and specifically provides a multi-agent global consensus collaboration method based on progressive communication. Background Art

[0002] Multi-agent reinforcement learning technology is a technology applicable to the coordination and cooperation between multiple agents to improve the overall efficiency, which achieves common goals through effective communication and coordination among agents.

[0003] In scenarios with multiple agents, complex environments, and high-dimensional state spaces, such as collaborative tasks in large-scale multiplayer online games like StarCraft, how to ensure the consistency and accuracy of the global consensus of all agents on the global state has become a key issue in multi-agent collaboration.

[0004] In the field of multi-agent reinforcement learning (MARL), although many efficient multi-agent communication methods have been proposed, these methods mainly focus on solving the problems of partial observability and non-stationarity of the environment. However, how to align the understanding and strategies of agents through effective communication in a multi-agent system to ensure the formation of global consensus is still a less studied and solved problem. Although the current multi-agent communication methods have achieved certain results in local environmental information sharing and policy coordination, they often cannot efficiently achieve global consensus among agents when facing challenges such as a large number of agents, complex environments, and high-dimensional tasks. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides a multi-agent global consensus collaboration method based on progressive communication, which solves the problems mentioned in the above background art.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: A multi-agent global consensus collaboration method based on progressive communication, comprising the following steps:

[0007] S1. The encoding module uses the observation of agent i at the current time step t and the action at the previous time step as inputs to update the self-awareness of agent i

[0008] S2. Based on the attention mechanism, the attention module generates an attention weight vector α and communication information according to the self-awareness of the agent i and the observation

[0009] S3, according to the attention weight vector α between agents i The selection module groups the agents with high attention weight similarity into the same group and generates corresponding communication masks MASK1 and MASK2 according to the grouping information;

[0010] S4, the communication module transmits the communication information in the first stage and the communication mask MASK1 as the input of the local visual converter to generate the local consensus h L In the second stage, the local consensus h L and the communication mask MASK2 as the input of the global visual transformer to promote the formation of a global consensus h G ;

[0011] S5, Q-value module converts the self-cognition of agent i Local consensus and global consensus As input, generate the current Q value of the agent, that is

[0012] S6. Establish KL divergence module and reconstruction module.

[0013] Preferably, the specific implementation process of S1 includes:

[0014] The encoding module is used to process the local observation information of each agent to update the internal cognition of the agent, which is expressed as Its composition includes multilayer perceptrons and gated recurrent units;

[0015] The multilayer perceptron is used to transform the observation of agent i at the current time step t and the action at the previous time step As input, to obtain its potential features

[0016]

[0017] The gated recurrent unit is used to transform the latent features and the cognition of agent i at the previous time step As input, to obtain the agent's cognition at the current time step

[0018]

[0019] Preferably, the specific implementation process of S2 includes:

[0020] The attention weight vector α i and communication information Generated by the attention module, which includes a query q, a key k, and a value v, and is used to selectively extract key information from the observations ;

[0021] The observation of agent i at the current time step t is divided into four parts: the characteristics of the agent itself the characteristics of teammates the characteristics of opponents and the environmental characteristics The attention module uses the agent's own cognition as the query q h , uses the observation as the key k j and the value v j , that is:

[0022]

[0023] wherein, represents the j-th part of the observation of agent i at time step t, and then calculates the query attention weight α : ij :

[0024]

[0025] The communication information of agent i at the current time step t is obtained by weighted summation of the value v j and the attention weight α ij , that is:

[0026]

[0027] Preferably, the S3 includes:

[0028] The attention weight α ij is used to record the degree of attention to the j-th part of the observation of agent i at the current time step t, where if different agents have similar attention weights, they are more likely to communicate and reach a consensus;

[0029] The selection module is used to focus on the dimension with the largest attention weight, find the agents with the largest attention weight in the same dimension (argmax(α i ) = argmax(α j )) and group them into the same group; specifically: for agent i, if in its attention weight vector, for the observation the k-th part, the attention weight α ik ​has the highest value, and for agent j, among its attention weight vectors for the observation the k * part has the highest attention weight value. If i≠j and k = k * , then agents i and j are grouped into the same group k;

[0030] The communication mask MASK1 is a matrix that identifies the agents communicating in the first stage. The element in the i-th row and j-th column is 1, indicating that agent i can communicate with agent j in the first-stage communication; if the element in the i-th row and j-th column is 0, it means that agent i cannot communicate with agent j in the first-stage communication. At the same time, according to the fact that agents in the same group can communicate with each other in the first-stage communication, the required communication mask MASK1 is obtained;

[0031] Here, all agents are divided into K groups, denoted as:

[0032] g = {g1, g2,..., g K}},

[0033] where g k = {i|argmax(α i ) = k, i ∈ {1, 2,..., n}}, that is, the attention weights of agents in each group are the same; according to g = {g1, g2,..., g k}, the communication mask MASK1 is generated;

[0034] The communication mask MASK2 is used to confirm the agents that need to communicate between groups during the second-stage communication process;

[0035] In each group g k , the agent with the highest attention weight is selected for global communication. The set of agents for global communication is denoted as:

[0036] G = {G1, G2,..., G k}},

[0037] where G k = {argmax(α ik ), i ∈ g i , k ∈ {1, 2,..., K}};

[0038] Specifically: if in group g k , for agent G k the attention weight α for the k-th part in the observation ik is the largest, then for group g k , the agent that needs to communicate with agents in other groups during the second-stage communication is agent Gk ; The communication mask MASK2 is a matrix that identifies the communication agents in the second stage. If the element in the i-th row and j-th column is 1, it means that agent i can communicate with agent j in the second-stage communication; if the element in the i-th row and j-th column is 0, it means that agent i cannot communicate with agent j in the second-stage communication. In the second-stage communication, agent i can communicate with agents in a different group G k Therefore, the element in the i-th row and the G k column of the communication mask MASK2 is 1. According to this principle, the required communication mask MASK2 can be obtained.

[0039] Preferably, the specific implementation process of S4 includes:

[0040] The local vision transformer is used to exchange information m i within the same group to generate a local consensus h L . The local vision transformer takes the communication information [m1, m2,..., m n and the communication mask MASK1 as inputs to obtain a local consensus vector representation h L , that is:

[0041]

[0042] Generate the local consensus among agents within the same group represents the local consensus of the agent; where, represents the identity coding information of each agent i;

[0043] The global vision transformer is responsible for aggregating the local consensus h L in each group to form a global consensus h G ; The global vision transformer outputs a global consensus vector That is:

[0044]

[0045] The identity coding information of the corresponding agent is added to each local consensus vector

[0046] Preferably, the specific implementation process of S5 includes:

[0047] The input of the Q-value module includes the self-awareness τ i of agent i, the local consensus and the global consensus The specific implementation is: ​

[0048]

[0049] Among them, the multi-layer perceptron maps τ i , and to the same dimension as the agent's action space, denotes element-wise addition, and finally, the Q-values of each agent are aggregated through the mixing network in the QMIX architecture to generate a joint Q-value function for global decision-making in multi-agent reinforcement learning.

[0050] Preferably, the S6 includes:

[0051] Align the local consensus through the KL divergence loss, which is specifically implemented as: minimizing the difference between the distributions obtained by converting the respective local consensuses of different agents in the same group through the softmax function and to align the local consensus:

[0052]

[0053] where |g k | represents the number of agents in group k, and i, j ∈ g k , i ≠ j;

[0054] Align the global consensus through the KL divergence loss, which is specifically implemented as: minimizing the difference between the distributions obtained by converting the respective global consensuses of different agents through the softmax function and to align the global consensus:

[0055]

[0056] where N represents the total number of agents, and i ≠ j.

[0057] Preferably, the reconstruction module is used to ensure the accuracy of the global consensus reached by the agents, and reconstruct the global state s through the latent global consensus h G ;

[0058] The reconstruction module is implemented using a variational autoencoder model, which includes an encoder and a decoder. The encoder generates a latent global consensus embedding vector Z based on the global state s G , and the decoder reconstructs the global state s based on Z G using a deep neural network;

[0059] The embedding vector Z of the global consensus G is obtained through the variational posterior distribution q θ (ZG |s) is generated by sampling.

[0060] Preferably, the global state s is represented by a joint observation The joint observation is considered equivalent to the global state s, thus obtaining an approximate relationship Here, given the global state s, there is a true posterior distribution p(Z G |s), which is calculated by the following formula according to Bayes' theorem:

[0061]

[0062] The posterior distribution p(Z G |s) is approximated by learning the variational posterior distribution q θ (Z G |s), that is, by minimizing the following KL divergence:

[0063] min{q θ (Z G |s)||p(Z G |s)};

[0064] Through the above approximation, the variational autoencoder (VAE) can accurately capture the core features of the global state;

[0065] The formula for minimizing the KL divergence can be derived as follows:

[0066]

[0067] Among them, since logp(s), that is, the log-likelihood of the state s, is a constant, and the KL divergence is non-negative, maximizing the variational lower bound is equivalent to minimizing the KL divergence formula. Therefore, further derivation of the variational lower bound formula gives:

[0068]

[0069] Beneficial effects

[0070] The present invention provides a multi-agent global consensus collaboration method based on progressive communication. Compared with the prior art, it has the following beneficial effects:

[0071] (1) In the present invention, through progressive communication, agents can continuously improve their understanding and strategies, thereby more efficiently achieving global consensus. The idea of this method stems from the process by which humans gradually reach a consensus through multiple interactions when solving complex tasks, so it is particularly effective in complex environments.

[0072] (2) In the present invention, a new similarity measurement method is proposed based on the similarity of attention weights to dynamically allocate communication targets, ensuring that agents only communicate with the other agents that are most relevant to them. This not only improves communication efficiency but also avoids the transmission of redundant information, reducing computational and communication overhead.

[0073] (3) In the present invention, the KL divergence constraint and the variational autoencoder (VAE) model are used to align local and global consensus, ensuring the consistency and accuracy of the global consensus. This enables agents to form more consistent strategies and decisions in complex multi-agent environments.

[0074] (4) In the present invention, it can significantly improve performance in complex tasks. Extensive experiments are conducted in SMAC scenarios with multiple difficulty levels. The method of the present invention exhibits superior performance in various tasks. Especially in tasks that require complex coordination, it is significantly better than the existing state-of-the-art methods. Especially in extremely difficult scenarios, it can still maintain a high winning rate, while the existing benchmark methods perform poorly. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a model architecture diagram of the multi-agent global consensus collaboration method with progressive communication provided by the present invention

[0076] Figure 2 It is an implementation flowchart of the multi-agent global consensus collaboration method with progressive communication provided by the present invention

[0077] Figure 3 It is a specific explanatory diagram of the selection module provided by the present invention

[0078] Figure 4 It is a comparison diagram of the cooperation effects of the method provided by the present invention and other multi-agent reinforcement learning methods in different StarCraft scenarios

[0079] Figure 5 It is an effect diagram of the ablation study of the model architecture provided by the present invention

[0080] Figure 6 It is a performance comparison diagram of calculating communication groups every step and every five steps using the selection module provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0081] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0082] Embodiment 1

[0083] The present invention provides a method for multi-agent global consensus collaboration based on progressive communication, which includes the following seven modules:

[0084] An encoding module, an attention module, a selection module, a communication module, a Q-value module, a KL divergence module, and a reconstruction module (as Figure 1 shown), where: The encoding module takes the observation of agent i at the current time step t and the action at the previous time step as inputs to update the self-awareness of agent i Based on the attention mechanism, the attention module generates an attention weight vector α and communication information according to the self-awareness of the agent i and the observation According to the similarity of the attention weight vectors α i between agents, the selection module divides the agents with higher attention weight similarity into the same group and generates communication masks MASK1 and MASK2 to clarify the communication targets of the agents; The communication module takes the communication information and the communication mask MASK1 as inputs to a local vision transformer (LocalViT) to generate a local consensus h L , and takes the local consensus h L and the communication mask MASK2 as inputs to a global vision transformer (GlobalViT) in the second stage to promote the formation of a global consensus h G ; The Q-value module takes the self-awareness of agent i local consensus and global consensus as inputs to generate the current Q-value of the agent, that is, To ensure the consistency and accuracy of the local and global consensus, the present invention establishes a KL divergence module and a reconstruction module, and promotes the alignment of the global consensus among multiple agents by introducing the KL divergence loss and the variational autoencoder (VAE) model.

[0085] Embodiment 2

[0086] The method for multi-agent global consensus collaboration based on progressive communication includes:

[0087] S110: The encoding module takes the observation of agent i at the current time step t and the action at the previous time step as inputs to update the self-awareness of agent i

[0088] A specific description of this step is as follows:

[0089] The encoding module processes the local observation information of each agent to update the cognitive representation inside the agent, and it consists of a multi-layer perceptron (MLP) and a gated recurrent unit (GRU);

[0090] The multi-layer perceptron (MLP) takes the observation of agent i at the current time step t and the action at the previous time step as inputs to obtain its latent features

[0091]

[0092] The gated recurrent unit (GRU) takes the latent features and the cognition of agent i at the previous time step as inputs to obtain the cognition of the agent at the next time step

[0093]

[0094] S120: The attention module generates an attention weight vector α and communication information based on the agent's own cognition i and observation

[0095] A detailed description of this step is as follows:

[0096] The attention weight vector α i and communication information are generated by the attention module, which includes a query q, a key k, and a value v. This module selectively extracts key information from the observation to effectively avoid data redundancy; In the present invention, the observation of agent i at the current time step t is divided into four parts: the agent's own features teammate features opponent features environment features The attention module takes the agent's own cognition as the query q j , takes the observation as the key k j and the value v j , that is:

[0097]

[0098] Among them, and represents the observation of agent i at time step t the j-th part in; then calculate the query attention weight α ij :

[0099]

[0100] the communication information of agent i at the current time step t is obtained by weighted summation of the value v j and the attention weight α ij That is:

[0101]

[0102] S130: The selection module divides agents with relatively high attention weight similarity into the same group and generates communication masks MASK1 and MASK2 to clarify the communication targets of the agents;

[0103] A specific description of this step is as follows:

[0104] The attention weight α ij records the observation of agent i at the current time step t of the j-th part, where[[ID=3l]] if different agents have similar attention weights, they are more likely to communicate and reach a consensus;

[0105] The selection module only focuses on the dimension with the largest attention weight, finds the agents with the largest attention weight in the same dimension (argmax(α i ) = argmax(α j )) and divides them into the same group;

[0106] Specifically, for agent i, if the attention weight α of the k-th part of its observation ik is the highest, and for agent j, if the attention weight of the k * -th part of its observation is the highest, and if i ≠ j and k = k * , then agent i and agent j are divided into the same group k;

[0107] The communication mask MASK1 is a matrix identifying the communicating agents in the first stage. The element in the i-th row and j-th column is 1, indicating that agent i can communicate with agent j in the first stage of communication. If the element in the i-th row and j-th column is 0, it means that agent i cannot communicate with agent j in the first stage of communication. According to the principle that agents in the same group can communicate with each other in the first stage of communication, the required communication mask MASK1 can be obtained;

[0108] All the agents are divided into K groups here, denoted as:

[0109] g = {g1, g2,..., g K},

[0110] where g k = {i | argmax(α i ) = k, i ∈ {1, 2,..., n}}, that is, the attention weights of the agents in each group are the same; according to g = {g1, g2,..., g k}, a communication mask MASK1 is generated to help the agents identify their communication objects in the subsequent first-stage communication;

[0111] The communication mask MASK2 is used to confirm the agents that need to communicate information between groups during the second-stage communication; the core idea is that the local consensus in the same group is similar, so during the second-stage inter-group communication, each agent only needs to communicate with one agent in other groups to achieve global consensus, which reduces data overhead and avoids information redundancy;

[0112] In each group g k , the agent with the largest attention weight is selected for global communication, and the set of agents for global communication is denoted as:

[0113] G = {G1, G2,..., G k},

[0114] where, G k = {argmax(α ik ), i ∈ g i , k ∈ {1, 2,..., K}};

[0115] Specifically, that is, if in group g k , for the agent G k , the attention weight α for the k-th part of the observation is the largest, then for group g ik , the agent that needs to communicate with agents in other groups during the second-stage communication is the agent G k ; the communication mask MASK2 is used as a matrix to identify the agents in the second-stage communication. If the element in the i-th row and j-th column is 1, it means that agent i can communicate with agent j during the second-stage communication. If the element in the i-th row and j-th column is 0, it means that agent i cannot communicate with agent j during the second-stage communication. And during the second-stage communication, agent i can communicate with the non-group agent G k . So, the element in the i-th row and the G k -th column of the communication mask MASK2 is 1, and the element in the i-th row and the j-th column of the communication mask MASK2 is 0 when j ≠ G kis 1; according to this principle, the required communication mask MASK2 can be obtained;

[0116] The communication agent G in each group selected by the above method k , will become the main participant in the second-stage communication, and the communication mask MASK2 generated based on G = {G1, G2,..., G k} is used to identify the agents for information communication, so as to guide the transmission of information in subsequent communication steps.

[0117] As Figure 3 shown, it explains the process of the selection module for agent grouping, identifying the communication agents in the first and second stages, so as to generate the communication masks MASK1 and MASK2. In Figure 3 , there are a total of 4 agents, and their attention weight vectors are [α1, α2, α3, α4]. First, perform the argmax() operation on each α i , that is, the result is:

[0118] [argmax(α1), argmax(α1), argmax(α1), argmax(α1)] = [1, 1, 2, 2],

[0119] This indicates that agent 1 and agent 2 have the same maximum attention weight, corresponding to agent 3 and agent 4 have the same maximum attention weight, corresponding to Therefore, we get:

[0120] g = {g1, g2},

[0121] where g1 = {1, 2}, g2 = {3, 4}; agents belonging to the same group share the same mask in the first communication step. Specifically, agent 1 and agent 2 from group g1 have the same communication mask MASK1[1] = MASK1[2] = [1, 1, 0, 0], while agent 3 and agent 4 from group g2 have the same communication mask MASK1[3] = MASK1[4] = [0, 0, 1, 1]; finally, the communication mask MASK1 of the first stage can be obtained as MASK1 = [MASK1[1], MASK1[2], MASK1[3], MASK1[4]].

[0122] For the generation of MASK2, as Figure 3 shown: The selection module selects the agent G with the largest attention weight α k in each group g i as the communication target in the second-stage communication. For g1, since argmax([α k , α 11 , α 12) = 1, so G1 = g1[1] = 1; for g2, since argmax([α 32 , α 42 ) = 1, so G2 = g2[1] = 3; thus, the communication target set G = {G1, G2} = {1, 3}, which means that in the second - stage communication process, each agent only needs to communicate with agent 1 and agent 3 in other groups. Specifically, for agent 1, the communication is limited to communicating with agent 3, and the corresponding communication mask is MASK2[1] = [1, 0, 1, 0]; for agent 2, agent 1 is in the same group with it, and its communication is limited to communicating with agent 3, and the corresponding communication mask is MASK2[2] = [0, 1, 1, 0]; for agent 3, the communication is limited to communicating with agent 1, and the corresponding communication mask is MASK2[3] = [1, 0, 1, 0]; for agent 4, agent 3 is in the same group with it, and its communication is limited to communicating with agent 1, and the corresponding mask is MASK2[4] = [1, 0, 0, 1]; finally, the communication mask MASK2 of the second stage can be obtained as MASK2 = [MASK2[1], MASK2[2], MASK2[3], MASK2[4]].

[0123] S140: The communication module uses the communication information and the communication mask MASK1 as the input of the Local Vision Transformer (LocalViT) to generate a local consensus and uses the local consensus and the communication mask MASK2 as the input of the Global Vision Transformer (GlobalViT) to facilitate the formation of a global consensus

[0124] A specific description of this step is as follows, including:

[0125] The communication module consists of a Local Vision Transformer (LocalViT) and a Global Vision Transformer (GlobalViT), where the Local Vision Transformer (LocalViT) is used for information communication between agents in the same group, and the Global Vision Transformer (GlobalViT) is used for information communication between all agents.

[0126] The Local Vision Transformer (LocalViT) exchanges information m i within the same group to generate a local consensus h L , to ensure that each agent only receives messages from its own group, the Local Vision Transformer (LocalViT) uses the communication information [m1, m2,..., m n and the communication mask MASK1 as inputs to obtain the local consensus vector representation h L , that is:

[0127]

[0128] Generate local consensus among agents within the same group Represent the local consensus of the agents; among them, Represents the identity coding information of each agent i to improve the distinguishability of messages;

[0129] The Global Vision Transformer (GlobalViT) is responsible for aggregating the local consensus h within each group L to form a global consensus h G ; Its input is the local consensus vector h of each group L and the mask MASK2, the latter of which is used to screen or weight information during integration; the output of the Global Vision Transformer (GlobalViT) is the global consensus vector reflecting all agents' understanding of the global information:

[0130]

[0131] Each local consensus vector is added with the corresponding agent identity coding information thus further enhancing the uniqueness and distinguishability of the information.

[0132] S150: The Q - value module takes the agent i's self - cognition local consensus and global consensus as inputs to generate the agent's current Q - value, that is

[0133] A specific description of this step is as follows, including:

[0134] The inputs of the Q - value module include the agent i's self - cognition τ i , local consensus and global consensus The specific implementation is:

[0135]

[0136] Among them, the multi - layer perceptron (MLP) maps τ i , and to the same dimension as the agent's action space, denotes element - wise addition, and finally, through the mixing network in the QMIX architecture, the Q - value of each agent Aggregation to generate a combined Q-value function for global decision-making in multi-agent reinforcement learning.

[0137] S160: To ensure the consistency and accuracy of local and global consensus, a KL divergence module and a reconstruction module are established, and the KL divergence loss and the variational autoencoder (VAE) model are introduced to promote the alignment of global consensus among multi-agents.

[0138] A specific description of this step is as follows, including:[[]]

[0139] Align the local consensus through the KL divergence loss. The specific implementation idea is to minimize the difference between the distributions obtained by different agents in the same group after converting their respective local consensus through the softmax function and to align the local consensus:

[0140]

[0141] where |g k | represents the number of agents in group k, and i, j ∈ g k , i ≠ j;

[0142] Align the global consensus through the KL divergence loss. The specific implementation idea is to minimize the difference between the distributions obtained by different agents after converting their respective global consensus through the softmax function and to align the global consensus:

[0143]

[0144] where N represents the total number of agents, and i ≠ j;

[0145] The reconstruction module aims to ensure the accuracy of the global consensus reached by the agents. The idea is that if the global state s can be reconstructed based on the potential global consensus in the global state through this potential global consensus then the accuracy of the global consensus can be ensured.

[0146] The reconstruction module is specifically implemented using the variational autoencoder (VAE) model. The variational autoencoder (VAE) consists of an encoder and a decoder. The encoder generates a potential global consensus embedding vector Z based on the global state s G , and the decoder reconstructs the global state s using a deep neural network based on Z G ;

[0147] The specific generation of the embedding vector Z of the global consensus G is through the variational posterior distribution q θ (ZG is obtained by sampling, and its mean and variance are provided by the global consensus and can be regarded as a low-dimensional representation of

[0148] The global state s is represented by the joint observation and the joint observation is considered equivalent to the global state s, thus obtaining the approximate relationship Here, given the global state s, there is a true posterior distribution p(Z G |s), which is calculated by the following formula according to Bayes' theorem:

[0149]

[0150] The posterior distribution p(Z G |s) is approximated by learning the variational posterior distribution q θ (Z G |s), that is, by minimizing the following KL divergence:

[0151] min{q θ (Z G |s)||p(Z G |s)},

[0152] Through the above approximation, the variational autoencoder (VAE) can accurately capture the core features of the global state;

[0153] The formula for minimizing the KL divergence can be derived as follows:

[0154]

[0155]

[0156] Among them, since logp(s), that is, the log-likelihood of the state s, is a constant, and the KL divergence is non-negative, maximizing the variational lower bound (ELBO) is equivalent to minimizing the KL divergence formula. Therefore, further derivation of the variational lower bound (ELBO) formula gives:

[0157]

[0158] Specifically, the first term of the variational lower bound (ELBO) represents the reconstruction likelihood of the VAE decoder, and the second term is the regularization of the variational posterior distribution q θ (Z G |s) to align it with the true prior distribution p(Z G ); The reconstruction module ensures that the global consensus accurately captures the core information of the environmental state s by maximizing the variational lower bound ELBO.

[0159] Example 3

[0160] Experimental Simulation and Analysis

[0161] Benchmark methods: The communication-free method QMIX and communication-based methods such as TarMac, DGN, CommFormer, and the Neighborhood Cognitive Consistency method (NCC) were selected as comparison benchmarks. The comprehensive selection of these benchmark methods can more thoroughly evaluate the effectiveness of the proposed method.

[0162] Experimental simulation environment settings: The StarCraft Multi-Agent Challenge (SMAC) is built based on StarCraftII and focuses on micro-management tasks. In this, agents independently control allied units under partially observable conditions, while rule-based bots control the enemies. Agents can observe attributes such as distance, position, health, shield, and type, and the goal is to defeat enemy units.

[0163] As shown in Table 1, the specific settings of the corresponding hyperparameters in the embodiments of the present invention are as follows: The optimizer uses the RMSProp optimizer, the learning rate is set to 1×10 -4 , the number of attention heads set in the Vision Transformer (ViT) is 2, the number of neurons in the hidden layer of the neural network is set to 32 or 64, the activation function uses the ReLU activation function, the buffer size is set to 5000, the sample batch size is set to 32, the discount factor is set to 0.99, and both α and β are set to 0.1.

[0164] The method STAGE of the present invention was evaluated in scenarios of three difficulty levels: simple scenarios (such as MMM and 8m maps), difficult scenarios (such as 5m vs 6m and 10m vs 11m maps), and super-difficult scenarios (such as MMM2 and 6h vs 8z maps) to comprehensively evaluate the performance of the proposed method.

[0165] Analysis of Experimental Results

[0166] As Figure 4 shown, the performance of different methods in the SMAC scenario was evaluated. The results show that the multi-agent global consensus cooperation method based on progressive communication outperforms the existing state-of-the-art benchmark methods in all environments. This clearly demonstrates the excellent adaptability and outstanding performance of the method proposed in the present invention in a wide range of SMAC tasks, especially in tasks of different difficulty levels and scales.

[0167] In addition, as the scene difficulty increases, the performance of the method proposed by the present invention has a greater improvement compared to the existing benchmark methods. It is worth noting that in the 6h vs 8z scenario, all benchmark methods failed to successfully complete the task, while the winning rate of STAGE still exceeded 60%. The excellent performance of this multi-agent global consensus collaboration method based on progressive communication stems from its ability to effectively extract accurate global consensus through progressive communication and consensus alignment, thus significantly enhancing the collaboration results.

[0168] Ablation analysis

[0169] As Figure 5 shown, an ablation study was conducted to quantify the specific contributions of each component in the multi-agent global consensus based on progressive communication (STAGE). Figure 5 In it, "STAGE w / o KL", "STAGE w / o VAE", and "STAGE w / oselection" respectively represent variants of STAGE after removing the KL module, VAE module, and selection module. "STAGE one-step-comm" is a variant of STAGE that replaces the two-step communication process with a one-step communication process. It can be observed that the performance of all STAGE variants is worse than that of STAGE itself. This indicates that the KL module, VAE module, selection module, and progressive communication technology all play important roles in the effectiveness of the method.

[0170] In the experiment, the performance of the selection module for calculating communication groups was further compared in the cases of every step and every five steps. The results are as Figure 6 shown. It can be observed that on the 5m vs 6m map, the performance of the two methods is almost the same. However, on the MMM2 map, the method of calculating communication groups using the selection module at every step is significantly better than the method of calculating communication groups every five steps. This result clearly shows that on extremely difficult maps, due to the complex and ever-changing corresponding situations, it is necessary to dynamically adjust the communication groups and corresponding strategies of agents in order to better improve the cooperation efficiency of multi-agents.

[0171] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0172] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or in one block or multiple blocks.

[0173] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or in one block or multiple blocks.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or in one block or multiple blocks.

[0175] Meanwhile, the content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

[0176] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device.

[0177] Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-agent global consensus collaboration method based on progressive communication, characterized in that: It includes the following steps: S1. The encoding module uses the observation of agent i at the current time step t and the action at the previous time step as inputs to update the self - cognition of agent i S2. Based on the attention mechanism, the attention module generates an attention weight vector α and observations according to the agent's own cognition i and communication information S3. According to the similarity of the attention weight vectors α between agents i the selection module groups the agents with high attention weight similarity into the same group, and generates corresponding communication masks MASK1 and MASK2 according to the grouping information; S4. The communication module uses the communication information and the communication mask MASK1 as the input of the local vision transformer to generate the local consensus h L . In the second stage, the local consensus h L and the communication mask MASK2 are used as the input of the global vision transformer to facilitate the formation of the global consensus h G ; S5. The Q-value module takes the agent i's own cognition Local consensus and global consensus as inputs to generate the current Q-value of the agent, i.e., S6. Establish a KL divergence module and a reconstruction module.

2. The multi-agent global consensus collaboration method based on progressive communication according to claim 1, wherein: The specific implementation process of S1 includes: The encoding module is used to process the local observation information of each agent to update the cognition inside the agent, expressed as It consists of a multi-layer perceptron and a gated recurrent unit; The multi-layer perceptron is used to take the observation of agent i at the current time step t and the action at the previous time step as inputs to obtain its latent features The gated recurrent unit is used to take the latent features and the cognition of agent i at the previous time step as inputs to obtain the cognition of the agent at the current time step 3. A multi-agent global consensus collaboration method based on progressive communication according to claim 1, characterized in that: The specific implementation process of S2 includes: The attention weight vector α i and communication information are generated by an attention module, which includes a query q, a key k, and a value v, and is used to selectively extract key information from the observation ; Observation of agent i at the current time step t It is divided into four parts: the characteristics of the agent itself The characteristics of teammates The characteristics of opponents And the characteristics of the environment The attention module takes the agent's own cognition As the query q j , and takes the observation As the key k j And the value v j That is: Among them, represents the j-th part of the observation of agent i at time step t, and then calculates the query attention weight α in, and then calculate the query attention weight α ij : The communication information of the agent i at the current time step t is calculated by weighted summation of the value v j and the attention weight α ij That is:

4. A multi-agent global consensus collaboration method based on progressive communication according to claim 1, characterized in that: S3 includes: The attention weight α ij is used to record the observation of agent i at the current time step t and the degree of attention to the j-th part, where The selection module is used to focus on the dimension with the maximum attention weight and find the maximum attention weight in the same dimension (argmax(α i )=argmax(α j )) agents and divide them into the same group; Specifically: if for agent i, in its attention weight vector for observation The attention weight α of the kth part ik The value of is the highest, and for agent j, in its attention weight vector for observation kth * Partial attention weight The value is the highest if i≠j and k=k * , then agents i and j are divided into the same group k; The communication mask MASK1 is used as a matrix to identify the communication agents in the first stage. The element in the i-th row and j-th column being 1 indicates that agent i can communicate with agent j in the first-stage communication; if the element in the i-th row and j-th column is 0, it means that agent i cannot communicate with agent j in the first-stage communication. At the same time, according to the fact that agents in the same group can communicate with each other in the first-stage communication, the required communication mask MASK1 is obtained. Here, all agents are divided into K groups, denoted as: g = {g1, g2,..., g K}, where g k = {i | argmax(α i ) = k, i ∈ {1, 2,..., n}}, that is, the attention weights of the agents in each group are the same; generate a communication mask MASK1 according to g = {g1, g2,..., g k}; The communication mask MASK2 is used to confirm the agents that need to communicate information between groups during the second-stage communication. Select the agent with the largest attention weight in each group g k for global communication. The set of agents for global communication is denoted as: G = {G1, G2,..., G k}, where G k = {argmax(α ik ), i ∈ g i , k ∈ {1, 2,..., K}}; Specifically: If in group g k the agent G k has the largest attention weight α for the k-th part in the observation ik then for group g k the agent that needs to communicate with other group agents during the second-stage communication is agent G k ; the communication mask MASK2 is used as a matrix to identify the second-stage communication agents. If the element in the i-th row and j-th column is 1, it means that agent i can communicate with agent j during the second-stage communication; if the element in the i-th row and j-th column is 0, it means that agent i cannot communicate with agent j during the second-stage communication; and during the second-stage communication, agent i can communicate with a non-group agent G k so the element in the i-th row and the G k -th column of the communication mask MASK2 is 1; according to this principle, the required communication mask MASK2 is obtained.

5. A multi-agent global consensus collaboration method based on progressive communication according to claim 1, characterized in that: The specific implementation process of S4 includes: The local vision transformer is used to exchange information m within the same group i to generate a local consensus h L , and the local vision transformer is used to take the communication information [m1, m2,..., m n and the communication mask MASK1 as inputs to obtain the local consensus vector representation h L , that is: Generate local consensus among agents within the same group represent the local consensus of the agents; where, represent the identity coding information of each agent i; The global vision transformer is responsible for aggregating the local consensus h within each group L to form a global consensus h G ; the output of the global vision transformer is the global consensus vector That is: The said each local consensus vector is added with the corresponding agent identity coding information 6. A multi-agent global consensus collaboration method based on progressive communication according to claim 1, characterized in that: The specific implementation process of S5 includes: The input of the Q-value module includes the self-cognition τ of agent i i , local consensus and global consensus The specific implementation is as follows: Among them, the multi-layer perceptron maps τ i , and to the same dimension as the agent's action space. ⊕ represents element-wise addition. Finally, the Q-values of each agent are aggregated through the mixing network in the QMIX architecture to generate the joint Q-value function for global decision-making in multi-agent reinforcement learning. ​ 7. A multi-agent global consensus collaboration method based on progressive communication according to claim 1, characterized in that: S6 includes: Align the local consensus through the KL divergence loss, which is specifically implemented as: minimizing the difference between the distributions obtained by different agents in the same group after converting their respective local consensus through the softmax function to align the local consensus: and the difference between the distributions obtained by different agents in the same group after converting their respective local consensus through the softmax function to align the local consensus: where |g k | represents the number of agents in group k, and i, j ∈ g k , i ≠ j; Align the global consensus through the KL divergence loss, which is specifically implemented as: minimizing the difference between the distributions obtained by different agents after converting their respective global consensuses through the softmax function to align the global consensus: and the difference between the resulting distributions to align the global consensus: Where N represents the total number of agents, and i ≠ j.

8. A multi-agent global consensus collaboration method based on progressive communication according to claim 7, characterized in that: The reconstruction module is used to ensure the accuracy of the global consensus reached by the agent, through this potential global consensus h G Reconstruct the global state s; The reconstruction module is implemented using a variational auto - encoder model. The variational auto - encoder consists of two parts: an encoder and a decoder. The encoder generates a latent global consensus embedding vector Z based on the global state s G , and the decoder reconstructs the global state s using a deep neural network based on Z G ; The embedding vector Z of the global consensus G is generated by sampling through the variational posterior distribution q θ (Z G | s).

9. A multi-agent global consensus collaboration method based on progressive communication according to claim 8, characterized in that: The global state s is represented by the joint observation which is considered equivalent to the global state s, resulting in the approximation relationship Here, given the global state s, there is a true posterior distribution p(Z |s), which is calculated by the following formula according to Bayes' theorem: G |s), according to Bayes' theorem, it is calculated by the following formula: The posterior distribution p(Z G |s) is approximated by learning the variational posterior distribution q θ (Z G |s), that is, by minimizing the following KL divergence: min{q θ (Z G |s)||p(Z G |s)}; Through the above approximation, the variational autoencoder (VAE) can accurately capture the core features of the global state. The minimized KL divergence formula is derived as: Where, since logp(s), i.e., the log-likelihood of state s, is a constant, and the KL divergence is non-negative, maximizing the variational lower bound is equivalent to minimizing the KL divergence formula. Therefore, the variational lower bound formula is further derived as:

Images