Multi-agent communication method based on role learning

Abstract

The invention relates to a multi-agent communication method based on role learning, and the method comprises the steps: dividing roles according to action attributes, calculating the characterization of the roles, and determining the roles of each agent in some continuous time; then, an attention mechanism is introduced, communication weight transmission information is distributed to different roles, and message transmission between the roles is achieved; meanwhile, sending objects and message contents of messages sent by the intelligent agents are determined, and communication between different intelligent agents under the same role is achieved; in the design, an information prediction model is established by utilizing mutual information, the effectiveness and pertinence of messages are ensured, meanwhile, communication contents are filtered by utilizing an attention mechanism, the conciseness of the messages is ensured, the problems of information redundancy, lack of pertinence and low expansibility in multi-agent communication are solved, the efficiency of multi-agent strategy learning is improved, and the method is suitable for popularization and application. And moreover, the negative influence of information shortage on strategy model convergence is inhibited, so that the strategy model training performance of cooperative multi-agent reinforcement learning is finally improved, and efficient communication of multiple agents is realized.

Description

Technical Field

[0001] The invention relates to a multi-agent communication method based on role learning, and belongs to the technical field of multi-agent communication. Background Art

[0002] Multi-agent reinforcement learning (MARL) is a technology that involves multiple agents interacting and learning in a shared environment. It aims to achieve the optimal solution for the entire system through collaboration or competition between agents. Each agent has the ability to perceive the environment, choose actions, and obtain rewards. MARL has a wide range of applications in the real world, including collaborative robot systems, traffic management, smart grid scheduling, network routing optimization, and other fields. Through MARL technology, collaborative learning and optimization between multiple agents can be achieved to improve the overall performance and efficiency of the system.

[0003] In multi-agent systems, problems such as environmental non-stationarity, trust distribution, and partial observability are common. In order to avoid the negative impact of these factors, a centralized training and decentralized execution method is usually adopted. Centralized training can use global information to guide the learning process, while decentralized execution can improve the efficiency and adaptability of the system. However, in the fully decentralized execution process, the agent makes decisions based only on its own observations and information. Due to the lack of global knowledge of the entire system state, it may fall into the problem of local optimal solutions or sub-optimization, which further aggravates partial observability. The increase in partial observability makes the agent lack sufficient information, which in turn increases the uncertainty and complexity of the agent in the learning and decision-making process, thereby hindering its learning of the optimal strategy. In a partially observable environment, the agents are independent individuals. Each agent can only access its own local observations, and can neither obtain the global state of the environment nor access relevant information of other agents.

[0004] The content transmitted by existing communication algorithms is the encoding and reward of the local observation of the agent. These communication paradigms can reduce the uncertainty of individual decision-making, but simply increasing the number of messages will lead to information redundancy and expand the local strategy space of the agent. In addition, since not all communication messages are meaningful to the agent, the communication algorithm needs to take into account the quality and quantity of the communication content and screen the information. In terms of the selection of communication objects, broadcast communication requires a lot of bandwidth; local communication uses distance as the criterion to find communication objects. Communication can alleviate some considerable problems, but most of the current communication mechanisms use distance as the criterion for optimizing communication objects. This selection may ignore some agents that are far away but have valid information. In the real world, complex tasks usually need to be solved in groups, which derives the important concept of group, but previous communication work ignores the existence of groups and generally regards all agents as a whole to design communication mechanisms, which has problems such as information redundancy, lack of specificity and low scalability.

[0005] In recent years, role-based multi-agent reinforcement learning has become a research focus. Role-based reinforcement learning is a dynamic grouping mechanism that uses roles to decompose complex tasks. However, the current role-based multi-agent reinforcement learning methods ignore some observable issues in the multi-agent system, which makes the agents lack sufficient environmental information and easily fall into local optimal solutions. In the multi-agent system, some observable issues make the agents lack information and lack global perception of the environment, which makes it easy to fall into local optimal solutions or sub-optimization problems. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a multi-agent communication method based on role learning, improve the problems of information redundancy, lack of specificity and low scalability in the multi-agent communication method, improve the efficiency of multi-agent strategy learning, and suppress the negative impact of information scarcity on the convergence of strategy models, thereby ultimately improving the communication performance of cooperative multi-agents.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solution: The present invention designs a multi-agent communication method based on role learning, and the following communication method is executed periodically according to a preset cycle length c:

[0008] Step A. Apply the trained action encoder to process the current actions of all agents respectively, obtain the current action representations of all agents, and then perform clustering on the current action representations, define the obtained clusters to correspond to each role one by one, and constitute each role under the current periodic communication method. Each cluster constitutes the action space of the corresponding role, and obtain the representation of each role, and then enter step B;

[0009] Step B. All agents input their current local observation and previous action into the observation encoding network to obtain the current historical information of the agent, and then proceed to step C;

[0010] Step C. The role selector is connected with the role allocation evaluation result under the previous cycle communication method, and according to the current historical information of each agent, the allocation of each role under the current cycle communication method is realized for all agents, and the roles corresponding to all agents are obtained, and then step D is entered;

[0011] Step D: realize the communication between the roles under the current periodic communication method, and realize the communication between the agents under each role respectively.

[0012] As a preferred technical solution of the present invention: in step A, the trained action encoder is applied For the current actions of all agents respectively implement Get the current action representation of each agent Among them, 1≤i≤I, I represents the number of all agents, represents the current action of the i-th agent among all agents, θ a Representation action encoder Medium parameters, Represents the current action representation of the i-th agent among all agents;

[0013] Based on the DBSCAN clustering process for each current action representation, it is defined that each cluster corresponds to each character one by one, and each cluster constitutes the action space of the corresponding character. For each character, the following formula is used:

[0014]

[0015] Obtain role representation; where 1≤l≤L, L represents the number of roles obtained, A ι represents the action space corresponding to the lth character, |A ι | indicates A ι The number of current action representations in z ι Represents the role representation of the lth role.

[0016] As a preferred technical solution of the present invention: the action encoder in step A According to the preset first cycle, the cycle is obtained by executing the following training method;

[0017] For motion encoders Corresponding to the current action is the input, represented by the current action As output, based on all agents, the current action representation of the agent Current local observation Current joint actions with other agents Substitute the following reward prediction encoder Observation Predictive Encoder to process;

[0018]

[0019] Get the agent's current prediction reward and the next predicted local observation Among them, 1≤i≤I, I represents the number of all intelligent agents;

[0020] Based on The current index of the agent obtained According to the following formula:

[0021]

[0022] Constructing the loss function And according to the loss function, the action encoder is realized After training, the action encoder is obtained in, represents the current reward of the agent, represents the next local observation of the agent, λ a represents the preset ruler factor, represents the square of the L2 norm, θ a Representation action encoder The middle parameter, θ r Represents the reward prediction encoder The parameter θ o Represents the observation predictive encoder Parameters, represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Represents the averaging of the sampled data of each agent in the experience replay pool.

[0023] As a preferred technical solution of the present invention: in the step D, each role respectively performs the following steps D1-1 to D1-4 to realize the communication between the roles under the current periodic communication method;

[0024] Step D1-1. Take the overall current historical information of each agent corresponding to the role As input, the historical encoding network is used to process the current historical information representation corresponding to the role Then proceed to step D1-1; where 1≤l≤L, L represents the number of roles, Represents the overall current historical information of each agent corresponding to the lth role, Indicates the current historical information representation corresponding to the lth role;

[0025] Step D1-2. Apply the message generator to represent the current historical information corresponding to the role Process and obtain the current original channel message corresponding to the role Then proceed to step D1-3; wherein, Indicates the current original channel message corresponding to the lth role;

[0026] Step D1-3. Apply the attention mechanism module to represent the current historical information Current raw channel message According to the following formula:

[0027]

[0028] Get the current weight corresponding to the role Then proceed to step D1-4; wherein, Indicates the current weight corresponding to the lth role, b l represents the set of serial numbers of other roles that communicate with the lth role, and exp() represents the exponential function;

[0029] Step D1-4. The role is based on its current weight The corresponding current original channel message The multiplication result between them constitutes a message, which is broadcast by the role to other roles with which it communicates.

[0030] As a preferred technical solution of the present invention: in the step D, based on the implementation of communication between roles under the current periodic communication method, it also includes executing the following steps D1-a to D1-c to implement role allocation evaluation;

[0031] Step D1-a. Representation of current historical information of each character After being processed by the role utility network, the role's Q value Q is obtained l , and combined with the messages broadcast by other roles received by the role l ,according to Get the communication bias Q value corresponding to the role Then proceed to step D1-b; wherein, Q l represents the Q value of the lth character, Υ l Indicates the message received by the lth role from other roles. Indicates the communication bias Q value corresponding to the lth role;

[0032] Step D1-b. The hybrid network biases the Q value of each role's communication Processing is performed to generate the Q value Q corresponding to all characters as a whole tot (τ,ρ), and according to the following formula:

[0033]

[0034] Get the TD target y for role learning tot , and then proceed to step D1-c; where τ represents the overall historical information of all agents, ρ represents the role combination of all roles of all agents under the current periodic communication method; r δ represents the sum of the rewards of all roles and all agents at the δth moment under the current periodic communication method, represents the sum of the rewards of all roles and all agents from the triggering moment to the end moment under the current periodic communication method, γ represents the preset discount factor, τ t+c represents the prediction history information of all agents under the next cycle communication method, ρ t+c represents the predicted role combination under the next cycle communication method, Q tot (τ t+c ,ρ t+c ) represents the Q value corresponding to the predicted role combination under the next cycle communication method, Indicates the maximum value of the Q value corresponding to each predicted role combination under the next cycle communication method;

[0035] Step D1-c. The Q value Q corresponding to all roles under the current cycle communication method tot (τ t ,ρ t ) and the role learning TD target y tot The difference between them is calculated according to the following loss function:

[0036]

[0037] Get loss results Constitute the role allocation evaluation result; where θ ρ represents the parameters in the role utility network, λ represents the preset penalty coefficient, represents the square of the L2 norm, represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Indicates averaging based on the experience replay pool.

[0038] As a preferred technical solution of the present invention: in the step C, the role selector performs the following steps C1 to C3 for all the agents respectively;

[0039] Step C1. Get the current historical information of the agent The corresponding current historical information representation And enter step C2; where 1≤i≤I, I represents the number of all agents, represents the current historical information of the i-th agent, Represents the current historical information representation of the i-th agent;

[0040] Step C2: Represent the agent’s current historical information Perform point multiplication with the matrix constructed by the role representation of each role under the current periodic communication method, and then obtain the probability of the agent corresponding to each role under the current periodic communication method through the softmax function layer, and then enter step C3;

[0041] Step C3. Determine whether the evaluation result of the role allocation under the communication method in the previous cycle is qualified. If it is qualified, the role with the maximum probability corresponds to the allocation of the agent; otherwise, the Q value Q of each role in the process is obtained according to the probability of the agent corresponding to each role and the evaluation result of the role allocation under the communication method in the previous cycle. l , apply the ε-greedy strategy to obtain the role assigned by the agent; where Q l Represents the Q value of the lth character.

[0042] As a preferred technical solution of the present invention: in the step D, for each agent in each role, the agent is used as the agent to be analyzed, and the following steps D2-1 to D2-4 are performed to realize communication between the agents under the role;

[0043] Step D2-1. For each other agent that communicates with the agent to be analyzed under the role to which the agent to be analyzed belongs, the current historical information of the agent to be analyzed and the serial numbers k of other agents j , apply the trained information prediction model to obtain the current action representation of the intelligent agent to be analyzed to other intelligent agents, and constitute the original communication content of the intelligent agent to be analyzed to other intelligent agents Then proceed to step D2-2; wherein, represents the current historical information of the jth agent to be analyzed in the role, k j represents the sequence number of the kth other agent among the other agents that communicate with the jth agent to be analyzed under the role, represents the jth agent to be analyzed in the role for the kth agent communicating with it jThe original communication content of other intelligent agents;

[0044] Step D2-2. Apply the attention mechanism module according to the following formula:

[0045]

[0046] Get the communication weight of the agent to be analyzed to other agents Then proceed to step D2-3; wherein k j represents the total number of other agents in the role that communicate with the jth agent to be analyzed, represents the kth agent in the role that communicates with the jth agent to be analyzed j The communication weight of other agents, λ represents the temperature factor used to scale the input, and exp() represents the exponential function;

[0047] Step D2-3. According to the following entropy regularization loss function:

[0048]

[0049] Get loss results And judge If it is qualified, go to step D2-4; otherwise, adjust the temperature factor λ according to the preset increment and return to step D2-2;

[0050] Step D2-4. For each other agent that communicates with the agent to be analyzed under the role to which the agent to be analyzed belongs, press Get the messages sent by the agent to be analyzed to other agents The agent to be analyzed broadcasts the information to other agents. Indicates that the jth agent to be analyzed in the role sends a signal to the kth agent communicating with it. j Send messages to other agents.

[0051] As a preferred technical solution of the present invention: the information prediction model in step D2-1 is obtained by periodically executing the following training method;

[0052] First, for the agent, the encoder is constructed with historical information and the serial numbers of other agents that communicate with the agent as input and the action representation of the agent to the other agents as output. That is, the information prediction model;

[0053] Then follow the formula:

[0054] I(N;A j |Ω)=H(N|Ω)-H(N|A j ,Ω)

[0055] Construct the encoder output prediction model parameters and calculate the conditional mutual information I(N; A) between the action representation and the actual action of the other agent j |Ω) and maximize the conditional mutual information, where N represents the information prediction model, A j represents the action space of the currently selected role of the jth agent to be analyzed, Ω represents the conditional distribution formed by the historical information of the jth agent to be analyzed and the serial numbers of other agents communicating with it, I(N; A j |Ω) represents N and A under the Ω conditional distribution j The conditional mutual information of N, H(N|Ω) is the conditional entropy of N under the conditional distribution of Ω, and H(N|A j ,Ω) is the conditional distribution under Ω and A j Conditional entropy under N;

[0056] Finally, according to the mutual information of the conditions, the following formula is used:

[0057]

[0058] Constructing the loss function And according to the loss function, the jth agent to be analyzed is connected to the kth agent communicating with it. j The information prediction model of other intelligent agents is trained to obtain the trained information prediction model; among them, represents the jth agent to be analyzed under the Ω conditional distribution and the kth agent to communicate with it j The distribution of information prediction models of other agents, Indicates that under the Ω conditional distribution and A j The next j-th agent to be analyzed communicates with the k-th agent j The variational distribution of other agents, ξ represents the variational distribution Parameters; represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Represents the averaging process for each agent in the experience replay pool.

[0059] As a preferred technical solution of the present invention: in the step D, based on the intelligent agent to be analyzed, the message is sent Sending to other agents also includes executing the following steps D2-a to D2-c;

[0060] Step D2-a. Apply the historical encoding network to obtain the current historical information of the agent to be analyzed The corresponding current historical information representation Then apply the agent utility network processing to obtain the Q value Q of the agent to be analyzed j, and combined with the message v broadcasted by other agents received by the agent to be analyzed j ,according to Get the communication bias Q value corresponding to the agent to be analyzed Then proceed to step D2-b; wherein, Q j represents the Q value of the jth agent to be analyzed under the role, v j represents the message broadcasted by other agents received by the jth agent to be analyzed under the role, represents the communication bias Q value corresponding to the jth agent to be analyzed under the role;

[0061] Step D2-b. The hybrid network's communication bias Q value for each agent to be analyzed under the role Processing is performed to generate the Q value Q corresponding to all the intelligent agents to be analyzed under all roles tota (τ, a), τ represents the overall historical information of all agents under all roles, a represents the actions of all agents under all roles, and the loss function is as follows:

[0062]

[0063] Get loss results Constitute the evaluation result of the agent's action selection, where θ TD represents the parameters of the agent utility network, r represents the sum of the rewards of all agents at each moment under the periodic communication method, γ represents the preset discount factor, τ' represents the next predicted historical information of all agents as a whole, a' represents the next predicted action combination of all agents, Q tot (τ',a') represents the Q value corresponding to the next combination of all agent predicted actions, Indicates the maximum value of the Q value corresponding to the next combination of all agent predicted actions; represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Indicates averaging based on the experience playback pool; then enters step D2-c;

[0064] Step D2-c. Determine whether the evaluation result of the agent's action selection is qualified. If it is, it means that the action selection made by the agent to be analyzed is correct; otherwise, the Q value of each communication bias in the process of obtaining the evaluation result of the previous role assignment is obtained according to the agent. and role ρ ι Input to the action network, apply the ε-greedy strategy to obtain the action selected by the agent, and each agent will input the executed action, local observation, reward, and next local observation as an experience sample into the experience replay pool

[0065] As a preferred technical solution of the present invention: it also includes step E, and step E includes steps E1 to E4 as follows: after executing step D, enter step E1;

[0066] Step E1. According to the role division loss and role distribution loss, the formula is as follows:

[0067]

[0068] Constructing the role learning loss function Then proceed to step E2, where φ r Contains the action encoder parameters θ a , reward prediction encoder parameters θ r , observe the predictive encoder parameters θ o , and the role utility network parameters θ ρ , represents the loss function of the action encoder, reward prediction encoder, and observation prediction encoder; represents the loss function for training the role utility network;

[0069] Step E2. According to the information prediction loss and the message filtering loss, the formula is as follows:

[0070]

[0071] Constructing the communication loss function Then proceed to step E3, where φ c Contains the prediction model parameters θ pre , ξ represents the variational distribution The parameter θ com Represents the message filtering parameters in the attention mechanism module, represents the loss function of the training information prediction model, represents the loss function for optimizing the communication weights of the agent;

[0072] Step E3. According to the following formula:

[0073]

[0074] Constructing the loss function Then proceed to step E4; wherein φ represents all parameters of the multi-agent communication method, and α and β are hyperparameters for adjusting the role learning process and optimizing the communication process respectively;

[0075] Step E4. According to the loss function The action encoder parameters θ a , reward prediction encoder parameters θ r , observe the predictive encoder parameters θ o, role utility network parameters θ ρ , the parameter θ of the attention mechanism module com , the parameter ξ of the approximate distribution of the information prediction model, the parameter θ of the agent utility network TD , the parameters of the agent action network are adjusted and updated.

[0076] The multi-agent communication method based on role learning described in the present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0077] (1) The present invention designs a multi-agent communication method based on role learning, which divides roles according to action attributes and calculates role representations, and determines the role of each agent at certain continuous times; then introduces an attention mechanism to assign communication weights to different roles to transmit information, thereby realizing message transmission between roles; at the same time, the sending object and message content of the message sent by the agent are determined to realize communication between different agents under the same role; in the design, mutual information is used to establish an information prediction model to ensure the effectiveness and pertinence of the message, and the attention mechanism is used to filter the communication content to ensure the simplicity of the message, thereby improving the problems of information redundancy, lack of pertinence, and low scalability in multi-agent communication, improving the efficiency of multi-agent strategy learning, and suppressing the negative impact of information scarcity on the convergence of the strategy model, thereby ultimately improving the strategy model training performance of cooperative multi-agent reinforcement learning and realizing efficient communication of multi-agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 It is a schematic diagram of character representation in the design of the present invention;

[0079] Figure 2 It is a schematic diagram of role communication in the design of the present invention;

[0080] Figure 3 It is a schematic diagram of multi-agent communication in the design of the present invention. DETAILED DESCRIPTION

[0081] The specific implementation methods of the present invention are further described in detail below in conjunction with the drawings in the specification. The present invention designs a multi-agent communication method based on role learning. In actual application, the following communication method is executed periodically according to a preset cycle length c.

[0082] Step A. Apply the trained action encoder For the current actions of all agents respectively implement Get the current action representation of each agent Among them, 1≤i≤I, I represents the number of all agents, represents the current action of the i-th agent among all agents, θ a Representation action encoder Medium parameters, Represents the current action representation of the i-th agent among all agents.

[0083] Then, by performing DBSCAN clustering processing on each current action representation, it is defined that each cluster obtained corresponds to each role one by one, forming each role under the current periodic communication method. Each cluster constitutes the action space of the corresponding role, and for each role, the following formula is used:

[0084]

[0085] Get role representation ι , where 1≤l≤L, L represents the number of roles obtained, A ι represents the action space corresponding to the lth character, |A ι | indicates A ι The number of current action representations in z ι Represents the role representation of the lth role; then proceed to step B.

[0086] The action encoder involved in the design of the above step A In actual application, Figure 1 As shown, according to the preset first cycle, the cycle executes the following training method to obtain.

[0087] For motion encoders Corresponding to the current action is the input, represented by the current action As output, based on all agents, the current action representation of the agent Current local observation Current joint actions with other agents Substitute the following reward prediction encoder Observation Predictive Encoder to process;

[0088]

[0089] Get the agent's current prediction reward and the next predicted local observation Among them, 1≤i≤I, I represents the number of all intelligent agents.

[0090] Based on The current index of the agent obtained According to the following formula:

[0091]

[0092] Constructing the loss function And according to the loss function, the action encoder is realized After training, the action encoder is obtained in, represents the current reward of the agent, represents the next local observation of the agent, λ a represents the preset ruler factor, represents the square of the L2 norm, θ a Representation action encoder The middle parameter, θ r Represents the reward prediction encoder The parameter θ o Represents the observation predictive encoder Parameters, represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Represents the averaging of the sampled data of each agent in the experience replay pool.

[0093] Step B: All agents respectively observe their current local Previous Action Input into the observation encoding network to obtain the current historical information of the agent Then proceed to step C.

[0094] Step C. The role selector connects the role allocation evaluation results under the previous cycle communication method, and based on the current historical information of each intelligent agent, realizes the allocation of each role to each intelligent agent under the current cycle communication method, obtains the roles corresponding to all intelligent agents, and then enters step D.

[0095] In actual application, the above step C is performed by the role selector for all intelligent agents respectively, and the following steps C1 to C3 are executed.

[0096] Step C1. Get the current historical information of the agent The corresponding current historical information representation And enter step C2; where 1≤i≤I, I represents the number of all agents, represents the current historical information of the i-th agent, Represents the current historical information representation of the i-th agent.

[0097] Step C2: Represent the agent’s current historical information Perform point multiplication with the matrix constructed by the role representation of each role under the current periodic communication method, and then obtain the probability of the agent corresponding to each role under the current periodic communication method through the softmax function layer, and then enter step C3.

[0098] Step C3. Determine whether the evaluation result of the role allocation under the communication method in the previous cycle is qualified. If it is qualified, the role with the maximum probability corresponds to the allocation of the agent; otherwise, the Q value Q of each role in the process is obtained according to the probability of the agent corresponding to each role and the evaluation result of the role allocation under the communication method in the previous cycle. l , apply the ε-greedy strategy to obtain the role assigned by the agent; where Q l Represents the Q value of the lth character.

[0099] Step D: realize the communication between the roles under the current periodic communication method, and realize the communication between the agents under each role respectively.

[0100] In practical applications, such as Figure 2 As shown, the above step D designs each role to execute the following steps D1-1 to D1-4 respectively to realize the communication between the roles under the current periodic communication method.

[0101] Step D1-1. Take the overall current historical information of each agent corresponding to the role As input, the historical encoding network is used to process the current historical information representation corresponding to the role Then proceed to step D1-1; where 1≤l≤L, L represents the number of roles, Represents the overall current historical information of each agent corresponding to the lth role, Represents the current historical information representation corresponding to the lth role.

[0102] Step D1-2. Apply the message generator to represent the current historical information corresponding to the role Process and obtain the current original channel message corresponding to the role Then proceed to step D1-3; wherein, Indicates the current original channel message corresponding to the lth role.

[0103] Step D1-3. Apply the attention mechanism module to represent the current historical information Current raw channel message According to the following formula:

[0104]

[0105] Get the current weight corresponding to the role Then proceed to step D1-4; wherein, Indicates the current weight corresponding to the lth role, b l It represents the set of serial numbers of other roles that communicate with the lth role, and exp() represents the exponential function.

[0106] Step D1-4. The role is based on its current weight The corresponding current original channel message The multiplication result between them constitutes a message, which is broadcast by the role to other roles with which it communicates.

[0107] In practical applications, based on the implementation of communication between roles under the above-mentioned design of the current cycle communication method, the following steps D1-a to D1-c are further designed and executed to realize role allocation evaluation.

[0108] Step D1-a. Representation of current historical information of each character After being processed by the role utility network, the role's Q value Q is obtained l , and combined with the messages broadcast by other roles received by the role l ,according to Get the communication bias Q value corresponding to the role Then proceed to step D1-b; wherein, Q l represents the Q value of the lth character, Υ l Indicates the message received by the lth role from other roles. Represents the communication bias Q value corresponding to the lth role.

[0109] Step D1-b. The hybrid network biases the Q value of each role's communication Processing is performed to generate the Q value Q corresponding to all characters as a whole tot (τ,ρ), and according to the following formula:

[0110]

[0111] Get the TD target y for role learning tot , and then proceed to step D1-c; where τ represents the overall historical information of all agents, ρ represents the role combination of all roles of all agents under the current periodic communication method; r δ represents the sum of the rewards of all roles and all agents at the δth moment under the current periodic communication method, represents the sum of the rewards of all roles and all agents from the triggering moment to the end moment under the current periodic communication method, γ represents the preset discount factor, τ t+c represents the prediction history information of all agents under the next cycle communication method, ρ t+c represents the predicted role combination under the next cycle communication method, Q tot (τt+c ,ρ t+c ) represents the Q value corresponding to the predicted role combination under the next cycle communication method, It represents the maximum value of the Q value corresponding to each predicted role combination under the next cycle communication method.

[0112] Step D1-c. The Q value Q corresponding to all roles under the current cycle communication method tot (τ t ,ρ t ) and the role learning TD target y tot The difference between them is calculated according to the following loss function:

[0113]

[0114] Get loss results Constitute the role allocation evaluation result; where θ ρ represents the parameters in the role utility network, λ represents the preset penalty coefficient, represents the square of the L2 norm, represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Indicates averaging based on the experience replay pool.

[0115] The design of step D not only needs to realize the communication between the roles under the current periodic communication method, but also needs to realize the communication between the agents under each role. In practical applications, each agent in each role is used as the agent to be analyzed, such as Figure 3 As shown, execute the following steps D2-1 to D2-4 to achieve communication between the agents under the role.

[0116] Step D2-1. For each other agent that communicates with the agent to be analyzed under the role to which the agent to be analyzed belongs, the current historical information of the agent to be analyzed and the serial numbers k of other agents j , apply the trained information prediction model to obtain the current action representation of the intelligent agent to be analyzed to other intelligent agents, and constitute the original communication content of the intelligent agent to be analyzed to other intelligent agents Then proceed to step D2-2; wherein, represents the current historical information of the jth agent to be analyzed in the role, k j represents the sequence number of the kth other agent among the other agents that communicate with the jth agent to be analyzed under the role, represents the jth agent to be analyzed in the role for the kth agent communicating with it j The original communication content of other intelligent agents.

[0117] Step D2-2. Apply the attention mechanism module according to the following formula:

[0118]

[0119] Get the communication weight of the agent to be analyzed to other agents Then proceed to step D2-3; wherein, K j represents the total number of other agents in the role that communicate with the jth agent to be analyzed, represents the kth agent in the role that communicates with the jth agent to be analyzed j The communication weight of other agents, λ represents the temperature factor, and exp() represents the exponential function.

[0120] Step D2-3. According to the following entropy regularization loss function:

[0121]

[0122] Get loss results And judge If it is qualified, go to step D2-4; otherwise, adjust the temperature factor λ according to the preset increment and return to step D2-2.

[0123] Step D2-4. For each other agent that communicates with the agent to be analyzed under the role to which the agent to be analyzed belongs, press Get the messages sent by the agent to be analyzed to other agents The agent to be analyzed broadcasts the information to other agents. Indicates that the jth agent to be analyzed in the role sends a signal to the kth agent communicating with it. j Send messages to other agents.

[0124] The information prediction model involved in the design and execution of step D2-1 is obtained by periodically executing the following training method.

[0125] First, for the agent, the encoder is constructed with historical information and the serial numbers of other agents that communicate with the agent as input and the action representation of the agent to the other agents as output. Output information predicts the parameters of the model.

[0126] Then follow the formula:

[0127] I(N;A j |Ω)=H(N|Ω)-H(N|A j ,Ω)

[0128] Construct the encoder output prediction model parameters and calculate the conditional mutual information I(N; A) between the action representation and the actual action of the other agent j |Ω) and maximize the conditional mutual information, where N represents the information prediction model, A j represents the action space of the currently selected role of the jth agent to be analyzed, Ω represents the conditional distribution formed by the historical information of the jth agent to be analyzed and the serial numbers of other agents communicating with it, I(N; A j |Ω) represents N and A under the Ω conditional distribution j The conditional mutual information of Ω, H(N|Ω) is the conditional entropy of N under the conditional distribution of Ω, H(N|A j ,Ω) is the conditional distribution under Ω and A j The conditional entropy of N.

[0129] Finally, according to the mutual information of the conditions, the following formula is used:

[0130]

[0131] Constructing the loss function And according to the loss function, the jth agent to be analyzed is connected to the kth agent communicating with it. j The information prediction model of other intelligent agents is trained to obtain the trained information prediction model; among them, represents the jth agent to be analyzed under the Ω conditional distribution and the kth agent to communicate with it j The distribution of information prediction models of other agents, Indicates that under the Ω conditional distribution and A j The next j-th agent to be analyzed communicates with the k-th agent j The variational distribution of other agents, ξ represents the variational distribution Parameters; represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Represents the averaging process for each agent in the experience replay pool.

[0132] Execute the above steps D2-1 to D2-4 to realize the communication between the agents under the role, that is, the agent to be analyzed will send a message Send to other agents, and then further design and execute the following steps D2-a to D2-c.

[0133] Step D2-a. Apply the historical encoding network to obtain the current historical information of the agent to be analyzed The corresponding current historical information representation Then apply the agent utility network processing to obtain the Q value Q of the agent to be analyzed j, and combined with the message v broadcasted by other agents received by the agent to be analyzed j ,according to Get the communication bias Q value corresponding to the agent to be analyzed Then proceed to step D2-b; wherein, Q j represents the Q value of the jth agent to be analyzed under the role, v j represents the message broadcasted by other agents received by the jth agent to be analyzed under the role, It represents the communication bias Q value corresponding to the jth agent to be analyzed under the role.

[0134] Step D2-b. The hybrid network's communication bias Q value for each agent to be analyzed under the role Processing is performed to generate the Q value Q corresponding to all the intelligent agents to be analyzed under all roles tot (τ, a), τ represents the overall historical information of all agents under all roles, a represents the actions of all agents under all roles, and the loss function is as follows:

[0135]

[0136] Get loss results Constitute the evaluation result of the agent's action selection, where θ TD represents the parameters of the agent utility network, r represents the sum of the rewards of all agents at each moment under the periodic communication method, γ represents the preset discount factor, τ' represents the next predicted historical information of all agents as a whole, a' represents the next predicted action combination of all agents, Q tot (τ',a') represents the Q value corresponding to the next combination of all agent predicted actions, Indicates the maximum value of the Q value corresponding to the next combination of all agent predicted actions; represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Indicates averaging processing based on the experience replay pool; then enters step D2-c.

[0137] Step D2-c. Determine whether the evaluation result of the agent's action selection is qualified. If it is, it means that the action selection made by the agent to be analyzed is correct; otherwise, the Q value of each communication bias in the process of obtaining the evaluation result of the previous role assignment is obtained according to the agent. and role ρ ι Input to the action network, apply the ε-greedy strategy to obtain the action selected by the agent, and each agent will input the executed action, local observation, reward, and next local observation as an experience sample into the experience replay pool

[0138] According to the execution of the above steps A to D, that is, based on the division of all intelligent agents into different roles, the design implements the communication between the roles and the communication between the agents under each role. Step E is further designed to consider the training and updating of each network involved in the designed multi-agent communication method. The specific execution of steps E1 to E4 is as follows, that is, after executing step D, enter step E1.

[0139] Step E1. According to the role division loss and role distribution loss, the formula is as follows:

[0140]

[0141] Constructing the role learning loss function Then proceed to step E2, where φ r Contains the action encoder parameters θ a , reward prediction encoder parameters θ r , observe the predictive encoder parameters θ o , and the role utility network parameters θ ρ , represents the loss function of the action encoder, reward prediction encoder, and observation prediction encoder; represents the loss function for training the role utility network.

[0142] Step E2. According to the information prediction loss and the message filtering loss, the formula is as follows:

[0143]

[0144] Constructing the communication loss function Then proceed to step E3, where φ c Contains the prediction model parameters θ pre , ξ represents the variational distribution The parameter θ com Represents the message filtering parameters in the attention mechanism module, represents the loss function of the training information prediction model, Represents the loss function for optimizing the communication weights of the agents.

[0145] Step E3. According to the following formula:

[0146]

[0147] Constructing the loss function Then proceed to step E4; wherein φ represents all parameters of the multi-agent communication method, and α and β are hyperparameters for adjusting the role learning process and optimizing the communication process, respectively.

[0148] Step E4. According to the loss function The action encoder parameters θ a , reward prediction encoder parameters θ r , observe the predictive encoder parameters θ o , role utility network parameters θ ρ , the parameter θ of the attention mechanism module com , the parameter ξ of the approximate distribution of the information prediction model, the parameter θ of the agent utility network TD , the parameters of the agent action network are adjusted and updated.

[0149] In practical applications, the collaboration between various intelligent agents includes collaboration between various robots, collaboration between various terminals in traffic management, and collaboration between various power users in smart grid dispatching. Such applications include drone collaborative tasks and intelligent dispatching. In the drone collaborative reconnaissance mission, the drone is the intelligent agent, and it is divided into different roles such as reconnaissance drones, relay drones, and navigation drones based on actions. Intra-role communication and inter-role communication are used to improve the efficiency of task collaboration. The reconnaissance drone shares local reconnaissance information within the group through intra-role communication to ensure that the target area is covered without omission. The relay drone is responsible for exploring the mission breakpoints to guide the drone's next action selection. The reconnaissance drone sends the collected data to the command center of the navigation guidance drone to obtain the specific mission route. Such a communication mechanism not only reduces the redundant load of the communication network, but also ensures the flexible collaboration of the drone cluster in a complex environment.

[0150] In the assembly line tasks, robots are divided into different roles such as assembly robots, inspection robots and handling robots based on roles. Intra-role communication is used to achieve efficient collaboration within the group, and inter-role communication is used to complete task coordination between different workstations. Assembly robots share component status information within the group to ensure a smooth assembly process within the group, while inspection robots optimize the quality inspection process through intra-role communication. Inter-role communication is used to coordinate different functional groups. For example, after receiving the component completion status of the assembly group, the handling robot is quickly dispatched to deliver the finished product to the downstream inspection group. Intra-role communication improves the efficiency of tasks within the group, and inter-role communication ensures the overall coordination and unity of the assembly line. Each robot can efficiently complete complex and multi-stage task objectives.

[0151] The multi-agent communication method designed by the present invention divides roles according to action attributes and calculates role representations, and at the same time determines the role of each agent at certain continuous times; then introduces an attention mechanism to assign communication weights to different roles to transmit information, thereby realizing message transmission between roles; at the same time, determines the sending object and message content of the message sent by the agent, thereby realizing communication between different agents under the same role; in the design, mutual information is used to establish an information prediction model to ensure the effectiveness and pertinence of the message, and at the same time, the attention mechanism is used to filter the communication content to ensure the simplicity of the message, thereby improving the problems of information redundancy, lack of pertinence, and low scalability in multi-agent communication, improving the efficiency of multi-agent strategy learning, and suppressing the negative impact of information scarcity on the convergence of the strategy model, thereby ultimately improving the strategy model training performance of cooperative multi-agent reinforcement learning and realizing efficient communication of multi-agents.

[0152] In the design, role learning is used to guide the agent to select the communication object and generate specific and concise information content. On the one hand, it solves the problem of ignoring groups, information redundancy and lack of specificity in the current multi-agent communication algorithm; on the other hand, it alleviates some observable problems of the current multi-agent role learning algorithm.

[0153] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A multi-agent communication method based on role learning, characterized in that: According to the preset cycle length c, the cycle triggers the execution of the following communication method: Step A. Apply the trained action encoder to process the current actions of all agents respectively, obtain the current action representations of all agents, and then perform clustering on the current action representations, define the obtained clusters to correspond to each role one by one, and constitute each role under the current periodic communication method. Each cluster constitutes the action space of the corresponding role, and obtain the representation of each role, and then enter step B; Step B. All agents input their current local observation and previous action into the observation encoding network to obtain the current historical information of the agent, and then proceed to step C; Step C. The role selector is connected with the role allocation evaluation result under the previous cycle communication method, and according to the current historical information of each agent, the allocation of each role under the current cycle communication method is realized for all agents, and the roles corresponding to all agents are obtained, and then step D is entered; Step D: realize the communication between the roles under the current periodic communication method, and realize the communication between the agents under each role respectively.

2. The multi-agent communication method based on role learning according to claim 1, characterized in that: In step A, the trained action encoder is applied For the current actions of all agents respectively implement Get the current action representation of each agent Among them, 1≤i≤I, I represents the number of all agents, represents the current action of the i-th agent among all agents, θ a Representation action encoder Medium parameters, Represents the current action representation of the i-th agent among all agents; Based on the DBSCAN clustering process for each current action representation, it is defined that each cluster corresponds to each character one by one, and each cluster constitutes the action space of the corresponding character. For each character, the following formula is used: Obtain role representation; where 1≤l≤L, L represents the number of roles obtained, A l represents the action space corresponding to the lth character, |A l | indicates A l The number of current action representations in z l Represents the role representation of the lth role.

3. A multi-agent communication method based on role learning according to claim 1 or 2, characterized in that: The action encoder in step A According to the preset first cycle, the cycle is obtained by executing the following training method; For motion encoders Corresponding to the current action is the input, represented by the current action As output, based on all agents, the current action representation of the agent Current local observation Current joint actions with other agents Substitute the following reward prediction encoder Observation Predictive Encoder to process; Get the agent's current prediction reward and the next predicted local observation Among them, 1≤i≤I, I represents the number of all intelligent agents; Based on The current index of the agent obtained According to the following formula: Constructing the loss function And according to the loss function, the action encoder is realized After training, the action encoder is obtained in, represents the current reward of the agent, represents the next local observation of the agent, λ a represents the preset ruler factor, represents the square of the L2 norm, θ a Representation action encoder The middle parameter, θ r Represents the reward prediction encoder The parameter θ o Represents the observation predictive encoder Parameters, represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Represents the averaging of the sampled data of each agent in the experience replay pool.

4. The multi-agent communication method based on role learning according to claim 1, characterized in that: In step D, each role respectively performs the following steps D1-1 to D1-4 to achieve communication between the roles under the current periodic communication method; Step D1-1. Take the overall current historical information of each agent corresponding to the role As input, the historical encoding network is used to process the current historical information representation corresponding to the role Then proceed to step D1-1; where 1≤l≤L, L represents the number of roles, Represents the overall current historical information of each agent corresponding to the lth role, Indicates the current historical information representation corresponding to the lth role; Step D1-2. Apply the message generator to represent the current historical information corresponding to the role Process and obtain the current original channel message corresponding to the role Then proceed to step D1-3; wherein, Indicates the current original channel message corresponding to the lth role; Step D1-3. Apply the attention mechanism module to represent the current historical information Current raw channel message According to the following formula: Get the current weight corresponding to the role Then proceed to step D1-4; wherein, Indicates the current weight corresponding to the lth role, b l represents the set of serial numbers of other roles that communicate with the lth role, and exp() represents the exponential function; Step D1-4. The role is based on its corresponding current weight The corresponding current original channel message The multiplication result between them constitutes a message, which is broadcast by the role to other roles with which it communicates.

5. The multi-agent communication method based on role learning according to claim 4, characterized in that: In the step D, based on the implementation of communication between roles under the current periodic communication method, it also includes executing the following steps D1-a to D1-c to implement role allocation evaluation; Step D1-a. Representation of current historical information of each character After being processed by the role utility network, the role's Q value Q is obtained l , and combined with the messages received by the role from other roles broadcast γ l ,according to Get the communication bias Q value corresponding to the role Then proceed to step D1-b; wherein, Q l represents the Q value of the lth role, γ l Indicates the message received by the lth role from other roles. Indicates the communication bias Q value corresponding to the lth role; Step D1-b. The hybrid network biases the Q value of each role's communication Processing is performed to generate the Q value Q corresponding to all characters as a whole tot (τ,ρ), and according to the following formula: Get the TD target y for role learning tot , and then proceed to step D1-c; where τ represents the overall historical information of all agents, ρ represents the role combination of all roles of all agents under the current periodic communication method; r δ represents the sum of the rewards of all roles and all agents at the δth moment under the current periodic communication method, represents the sum of the rewards of all roles and all agents from the triggering moment to the end moment under the current periodic communication method, γ represents the preset discount factor, τ t+c represents the prediction history information of all agents under the next cycle communication method, ρ t+c represents the predicted role combination under the next cycle communication method, Q tot (τ t+c ,ρ t+c ) represents the Q value corresponding to the predicted role combination under the next cycle communication method, Indicates the maximum value of the Q value corresponding to each predicted role combination under the next cycle communication method; Step D1-c. The Q value Q corresponding to all roles under the current cycle communication method tot (τ t ,ρ t ) and the role learning TD target y tot The difference between them is calculated according to the following loss function: Obtain loss results Constitute the role allocation evaluation result; where θ ρ represents the parameters in the role utility network, λ represents the preset penalty coefficient, represents the square of the L2 norm, represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Indicates averaging based on the experience replay pool.

6. The multi-agent communication method based on role learning according to claim 5, characterized in that: In step C, the role selector performs the following steps C1 to C3 for all agents respectively; Step C1. Get the current historical information of the agent The corresponding current historical information representation And enter step C2; where 1≤i≤I, I represents the number of all intelligent agents, represents the current historical information of the i-th agent, Represents the current historical information representation of the i-th agent; Step C2: Represent the agent’s current historical information Perform point multiplication with the matrix constructed by the role representation of each role under the current periodic communication method, and then obtain the probability of the agent corresponding to each role under the current periodic communication method through the softmax function layer, and then enter step C3; Step C3. Determine whether the evaluation result of the role allocation under the communication method in the previous cycle is qualified. If it is qualified, the role with the maximum probability corresponds to the allocation of the agent; otherwise, the Q value Q of each role in the process is obtained according to the probability of the agent corresponding to each role and the evaluation result of the role allocation under the communication method in the previous cycle. l , apply the ε-greedy strategy to obtain the role assigned by the agent; where Q l Represents the Q value of the lth character.

7. The multi-agent communication method based on role learning according to claim 1, characterized in that: In the step D, for each agent in each role, the agent is used as the agent to be analyzed, and the following steps D2-1 to D2-4 are executed to realize communication between the agents under the role; Step D2-1. For each other agent that communicates with the agent to be analyzed under the role to which the agent to be analyzed belongs, the current historical information of the agent to be analyzed and the serial numbers k of other agents j , apply the trained information prediction model to obtain the current action representation of the intelligent agent to be analyzed to other intelligent agents, and constitute the original communication content of the intelligent agent to be analyzed to other intelligent agents Then proceed to step D2-2; wherein, represents the current historical information of the jth agent to be analyzed in the role, k j represents the sequence number of the kth other agent among the other agents that communicate with the jth agent to be analyzed under the role, represents the jth agent to be analyzed in the role for the kth agent communicating with it j The original communication content of other intelligent agents; Step D2-2. Apply the attention mechanism module according to the following formula: Get the communication weight of the agent to be analyzed to other agents Then proceed to step D2-3; wherein, K j represents the total number of other agents in the role that communicate with the jth agent to be analyzed, represents the kth agent in the role that communicates with the jth agent to be analyzed j The communication weights of other agents, λ represents the temperature factor, and exp( ) represents the exponential function; Step D2-3. According to the following entropy regularization loss function: Obtain loss results And judge If it is qualified, go to step D2-4; otherwise, adjust the temperature factor λ according to the preset increment and return to step D2-2; Step D2-4. For each other agent that communicates with the agent to be analyzed under the role to which the agent to be analyzed belongs, press Get the messages sent by the agent to be analyzed to other agents The agent to be analyzed broadcasts the information to other agents. Indicates that the jth agent to be analyzed in the role sends a signal to the kth agent communicating with it. j Send messages to other agents.

8. The multi-agent communication method based on role learning according to claim 7, characterized in that: The information prediction model in step D2-1 is obtained by periodically executing the following training method; First, for the agent, the encoder is constructed with historical information and the serial numbers of other agents that communicate with the agent as input and the action representation of the agent to the other agents as output. That is, the information prediction model; Then follow the formula: I(N;A j |Ω)=H(N|Ω)-H(N|A j ,Oh) Construct the encoder output prediction model parameters and calculate the conditional mutual information I(N; A) between the action representation and the actual action of the other agent j |Ω) and maximize the conditional mutual information, where B represents the information prediction model, A j represents the action space of the currently selected role of the jth agent to be analyzed, Ω represents the conditional distribution formed by the historical information of the jth agent to be analyzed and the serial numbers of other agents communicating with it, I(N; A j |Ω) represents N and A under the Ω conditional distribution j The conditional mutual information of N, H(N|Ω) is the conditional entropy of N under the conditional distribution of Ω, and H(N|A j ,Ω) is the conditional distribution under Ω and A j Conditional entropy under N; Finally, according to the mutual information of the conditions, the following formula is used: Constructing the loss function And according to the loss function, the jth agent to be analyzed is connected to the kth agent communicating with it. j The information prediction model of other intelligent agents is trained to obtain the trained information prediction model; among them, represents the jth agent to be analyzed under the Ω conditional distribution and the kth agent to communicate with it j The distribution of information prediction models of other agents, Indicates that under the Ω conditional distribution and A j The next j-th agent to be analyzed communicates with the k-th agent j The variational distribution of other agents, ξ represents the variational distribution Parameters; represents the experience replay pool that stores all the actions, rewards, local observations, and historical information of each agent. Represents the averaging process for each agent in the experience replay pool.

9. The multi-agent communication method based on role learning according to claim 7, characterized in that: In step D, based on the intelligent agent to be analyzed, the message Sending to other agents also includes executing the following steps D2-a to D2-c; Step D2-a. Apply the historical encoding network to obtain the current historical information of the agent to be analyzed The corresponding current historical information representation Then apply the agent utility network processing to obtain the Q value Q of the agent to be analyzed j , and combined with the message ν broadcast by other agents received by the agent to be analyzed j ,according to Get the communication bias Q value corresponding to the agent to be analyzed Then proceed to step D2-b; wherein, Q j represents the Q value of the jth agent to be analyzed under the role, ν j represents the message broadcasted by other agents received by the jth agent to be analyzed under the role, represents the communication bias Q value corresponding to the jth agent to be analyzed under the role; Step D2-b. The hybrid network's communication bias Q value for each agent to be analyzed under the role Processing is performed to generate the Q value Q corresponding to all the intelligent agents to be analyzed under all roles tot (τ, a), τ represents the overall historical information of all agents under all roles, a represents the actions of all agents under all roles, and the loss function is as follows: Obtain loss results Constitute the evaluation result of the agent's action selection, where θ TD represents the parameters of the agent utility network, r represents the sum of the rewards of all agents at each moment under the periodic communication method, γ represents the preset discount factor, τ' represents the next predicted historical information of all agents as a whole, a' represents the next predicted action combination of all agents, Q tot (τ',a') represents the Q value corresponding to the next combination of all agent predicted actions, represents the maximum value of the Q value corresponding to the next combination of all agent prediction actions; D represents the experience replay pool that stores the actions, rewards, local observations, and historical information of all agents. Indicates averaging based on the experience playback pool; then enters step D2-c; Step D2-c. Determine whether the evaluation result of the agent's action selection is qualified. If it is, it means that the action selection made by the agent to be analyzed is correct; otherwise, the Q value of each communication bias in the process of obtaining the evaluation result of the previous role assignment is obtained according to the agent. and role ρ l Input to the action network, apply the ε-greedy strategy to obtain the action selected by the agent, and each agent will input the executed action, local observation, reward, and next local observation as an experience sample into the experience replay pool 10. The multi-agent communication method based on role learning according to claim 1, characterized in that: It also includes step E, which includes steps E1 to E4 as follows: after executing step D, proceed to step E1; Step E1. According to the role division loss and role distribution loss, the formula is as follows: Constructing the role learning loss function Then proceed to step E2, where φ r Contains the action encoder parameters θ a , reward prediction encoder parameters θ r , observe the predictive encoder parameters θ o , and the role utility network parameters θ ρ , represents the loss function of the action encoder, reward prediction encoder, and observation prediction encoder; represents the loss function for training the role utility network; Step E2. According to the information prediction loss and the message filtering loss, the formula is as follows: Constructing the communication loss function Then proceed to step E3, where φ c Contains the prediction model parameters θ pre , ξ represents the variational distribution The parameter θ com Represents the message filtering parameters in the attention mechanism module, represents the loss function of the training information prediction model, represents the loss function for optimizing the communication weights of the agent; Step E3. According to the following formula: Constructing the loss function Then proceed to step E4; wherein φ represents all parameters of the multi-agent communication method, and α and β are hyperparameters for adjusting the role learning process and optimizing the communication process respectively; Step E4. According to the loss function The action encoder parameters θ a , reward prediction encoder parameters θ r , observe the predictive encoder parameters θ o , role utility network parameters θ ρ , the parameter θ of the attention mechanism module com , the parameter ξ of the approximate distribution of the information prediction model, the parameter θ of the agent utility network TD , the parameters of the agent action network are adjusted and updated.

Images