Multi-agent communication method, device and equipment based on communication graph learning

Abstract

The invention relates to the technical field of multi-agent reinforcement learning, and discloses a multi-agent communication method, device and equipment based on communication graph learning, and the method comprises the steps: obtaining a full communication graph structure and a communication neural network obtained through the last round of training when a plurality of agents start the current round of training; acquiring multiple groups of training data of the current round of training based on the communication neural network and the full communication graph structure obtained by the previous round of training; based on multiple groups of training data, updating the communication neural network obtained by the last round of training, obtaining the communication neural network of the current round of training, and completing the current round of training; repeating the training process, and performing multiple rounds of training to obtain a target communication neural network; and based on the coding network and the strategy network in the target communication neural network, performing communication among the multiple agents in a distributed manner. According to the method, efficient and autonomous distributed communication and cooperation are achieved, direct intervention of a central controller is not needed, the intelligent agent makes a decision according to individual observation and information transmitted in the network, and flexibility and robustness are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent reinforcement learning, and in particular to a multi-agent communication method, device and equipment based on communication graph learning. Background Art

[0002] In multi-agent reinforcement learning, communication technology refers to the methods and mechanisms for information exchange and cooperation between agents. Communication technology plays a vital role in multi-agent reinforcement learning. For example, in game application scenarios, it can help game agents learn and work together in collaborative tasks.

[0003] At present, in game application scenarios, usually a fixed set of neighbor nodes is predefined by the agent, and information is exchanged and collaborated through communication with the neighbor nodes. Alternatively, the agent broadcasts its own information to all other agents, and all agents receive the same information. However, due to the large number of agents in the game and their rapid changes, the above two technical solutions lack flexibility and adaptability, cannot cope with complex and dynamic game environments, and cannot adaptively learn communication strategies, and cannot continuously learn and improve their own capabilities according to the game task objectives, resulting in poor communication effects and poor robustness in different game tasks and game environments. Summary of the invention

[0004] In view of this, the present invention provides a multi-agent communication method, device and equipment based on communication graph learning to solve the problem that the existing multi-agents lack flexibility and adaptability in communication and cannot adapt to task requirements in complex and dynamic environments, resulting in poor communication effect and poor robustness.

[0005] In a first aspect, the present invention provides a multi-agent communication method based on communication graph learning, the method comprising:

[0006] When multiple agents start this round of training, the full communication graph structure and the communication neural network obtained in the previous round of training are obtained. The communication neural network includes the encoding network, the scheduling network, the strategy network and the evaluation network, which are used to realize the communication between multiple agents. The full communication graph structure indicates that multiple agents can communicate arbitrarily.

[0007] Based on the communication neural network and full communication graph structure obtained in the previous round of training, multiple sets of training data for this round of training are obtained;

[0008] Based on multiple sets of training data, the communication neural network obtained in the previous round of training is updated to obtain the communication neural network for this round of training, thereby completing this round of training;

[0009] Repeat the above training process for multiple rounds of training to obtain the target communication neural network;

[0010] Based on the encoding network and policy network in the target communication neural network, the communication between multiple agents is executed in a distributed manner.

[0011] The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention obtains the full communication graph structure and the communication neural network obtained in the previous round of training at the beginning of each round of training, so that the agent can further optimize the communication strategy based on previous learning experience, and then generates training data for this round of training through the training results of the previous round, and uses the training data to update the communication neural network obtained in the previous round of training to gradually approach the optimal communication strategy. After multiple rounds of such training iterations, a highly optimized target communication neural network is obtained, which can better cope with environmental changes and the diversity of task requirements. By using the encoding network and strategy network in the target communication neural network, the agent can achieve efficient and autonomous distributed communication and collaboration in actual tasks without the direct intervention of the central controller. Each agent makes decisions based on its individual observations and the information transmitted in the network, thereby enhancing the flexibility and robustness of multi-agent communication.

[0012] In an optional implementation, any set of training data includes a communication graph structure, observation information, actions, global state of the environment, rewards, action values, and predicted communication objects. Based on the communication neural network and the full communication graph structure obtained in the previous round of training, multiple sets of training data for this round of training are obtained, including:

[0013] In the first time step of this round of training, for any agent, the node features of the node corresponding to the agent in the first time step are obtained from the full communication graph structure. The node features represent the observation information of the agent.

[0014] The encoding network in the communication neural network obtained from the previous round of training is used to encode the node features of the agent to obtain the encoding vector;

[0015] The policy network in the communication neural network obtained from the previous round of training is used to determine the action of the agent in the first time step and the predicted communication object in the next time step based on the encoding vector;

[0016] The evaluation network in the communication neural network obtained from the previous round of training is used to make the agent perform the action of the first time step in the initial global state of the environment, and then determine the observation information, global state of the environment, reward and action value of the agent in the first time step;

[0017] Adopt the policy network in the communication neural network obtained in the previous round of training, update the full communication graph structure based on the observation information of the agent in the first time step and the predicted communication object in the next time step, and obtain the communication graph structure of the first time step;

[0018] The communication graph structure of the first time step is used as the communication graph structure for obtaining node features in the next time step, and the global state of the environment in the first time step is used as the initial environment state for the next time step. The above process of obtaining observation information, actions, global state of the environment, rewards, action values, predicted communication objects, and updating the communication graph structure is repeated until all time steps are completed to obtain multiple sets of training data for this round of training.

[0019] The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention is that each round of training is performed on the basis of the previous round, so that the communication strategy and understanding of the environment of the agent will continue to evolve as the training deepens, thereby improving the overall performance. The generation of training data takes into account the behavior of the agent at different time steps and the environmental feedback, so that multiple networks can learn dynamic interaction patterns that change over time, and enhance the adaptability in complex dynamic environments. By encoding and generating strategies for the specific context of each agent at a specific time step, the method promotes personalized decision-making, so that the agent can optimize its behavior according to the current communication environment and goals. The real-time update of the communication graph structure means that the network can learn the most effective communication path and communication object, which is conducive to efficient collaboration between agents. By predicting the communication object, the agent can plan future behavior and communication in advance, thereby improving the overall foresight and coordination efficiency of the system. When generating the global state of the environment, rewards and action values, the agent's understanding of the consequences of its actions is strengthened, which helps to form a more reasonable expected value estimate, thereby optimizing long-term goal-oriented behavior. By generating multiple sets of training data, support is provided for subsequent training.

[0020] In an optional implementation, the coding network in the communication neural network obtained in the previous round of training is used to encode the node features of the agent to obtain a coding vector, including:

[0021] Perform linear transformation on the node features of the agent to obtain the feature vector;

[0022] Determine the attention coefficient of the feature vector;

[0023] Normalize the attention coefficient to get the attention score;

[0024] Based on the attention score, update the node features of the agent to obtain the aggregated feature vector;

[0025] Add bias to the aggregated feature vector to get the encoded vector.

[0026] The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention improves the depth of the agent's understanding of the environment and the status of other agents by extracting more refined and representative feature representations, guides the agent to focus on the most important information through the attention mechanism, reduces unnecessary communication load, and improves the pertinence and effectiveness of communication. Normalization processing and the addition of deviation terms help to improve adaptability and robustness in different tasks and environments. The encoded feature vectors provide more accurate input for subsequent strategy formulation, allowing the agent to make more reasonable judgments in complex decision-making scenarios.

[0027] In an optional implementation, the policy network in the communication neural network obtained by the previous round of training is used to determine the action of the agent in the first time step and the predicted communication object in the next time step based on the encoding vector, including:

[0028] Using the first linear layer of the strategy network, linearly transform the encoding vector and multiple available action vectors preset in the strategy network to obtain the linearly transformed encoding vector and multiple available action vectors;

[0029] The recurrent neural network layer of the policy network is used to obtain the time series characteristics of the linearly transformed encoding vector and multiple available action vectors to obtain the hidden state;

[0030] The second linear layer of the policy network is used to determine the action of the first time step after generating the action values ​​of multiple available action vectors based on the hidden state;

[0031] The communication information generation layer of the policy network is used to determine the predicted communication object for the next time step based on the hidden state.

[0032] The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention effectively integrates the real-time information and historical behavior of the agents through the multi-layer deep learning structure of the policy network, which not only improves the decision-making quality of a single agent, but also promotes efficient collaboration and communication among multi-agent systems.

[0033] In an optional implementation, the evaluation network in the communication neural network obtained in the previous round of training is used to make the agent perform the action of the first time step in the initial environment global state, and then determine the action value of the agent in the first time step, including:

[0034] Get the initial environment global state and the action values ​​of multiple available action vectors;

[0035] Using a conditional super network of the evaluation network, based on an initial global state of the environment, a first weight and bias parameter and a second weight and bias parameter are generated;

[0036] Based on the first weight and bias parameter, linearly transform the action values ​​of the plurality of available action vectors and then apply nonlinear activation to obtain intermediate action values;

[0037] Based on the second weight and bias parameters, the intermediate action value is linearly transformed to obtain the action value of the agent at the first time step.

[0038] The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention significantly improves the learning efficiency and quality of the agent behavior strategy by adopting an evaluation network, so that it has good decision-making ability and adaptability in a complex and changing environment.

[0039] In an optional implementation, based on multiple sets of training data, updating the communication neural network obtained in the previous round of training to obtain the communication neural network in this round of training includes:

[0040] For any set of training data, the communication loss is obtained based on the difference between the training data before and after the communication graph structure is updated during the training process;

[0041] Determine the target action value of the training data;

[0042] Determine the action value loss based on the difference between the target action value and the action value of the training data;

[0043] Based on the communication loss and action value loss of each set of training data, the network parameters of the communication neural network obtained in the previous round of training are optimized to obtain the communication neural network in this round of training.

[0044] The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention significantly improves the efficiency of the communication neural network and the quality of collaboration between agents by determining the action value loss and the communication loss, and provides support for solving complex multi-agent tasks.

[0045] In an optional embodiment, based on the encoding network and the policy network in the target communication neural network, the communication between multiple agents is distributedly executed, including:

[0046] Based on the communication between multiple agents and multiple agents, an execution communication graph structure is generated;

[0047] Based on the encoding network in the target communication neural network, the observation information of multiple agents is encoded to obtain multiple target encoding vectors;

[0048] Based on the policy network in the target communication neural network and multiple target encoding vectors, the action of each agent at the current moment and the communication object at the next moment are obtained;

[0049] Make each agent perform the action at the current moment;

[0050] Based on the execution communication graph structure, each agent communicates with the communication object at the next moment, obtains the observation information received by each agent, returns to the encoding network based on the target communication neural network, encodes the observation information of multiple agents, and obtains the steps of multiple target encoding vectors until the observation information can no longer be obtained.

[0051] The multi-agent communication method based on communication graph learning provided in an embodiment of the present invention performs distributed communication by adopting a coding network and a strategy network in a target communication neural network, thereby significantly improving the collaborative efficiency, adaptability and ability to handle complex tasks of the multi-agent system, and providing support for the efficient operation of intelligent systems in a variety of application scenarios.

[0052] In a second aspect, the present invention provides a multi-agent communication device based on communication graph learning, the device comprising:

[0053] The first acquisition module is used to acquire the full communication graph structure and the communication neural network obtained in the previous round of training when multiple agents start this round of training. The communication neural network includes a coding network, a scheduling network, a strategy network and an evaluation network, which are used to realize the communication between multiple agents. The full communication graph structure indicates that multiple agents can communicate arbitrarily;

[0054] The second acquisition module is used to acquire multiple sets of training data for this round of training based on the communication neural network and the full communication graph structure obtained in the previous round of training;

[0055] An updating module is used to update the communication neural network obtained in the previous round of training based on multiple sets of training data, obtain the communication neural network of this round of training, and complete this round of training;

[0056] A training module is used to repeat the above training process, perform multiple rounds of training, and obtain a target communication neural network;

[0057] The communication module is used to perform distributed communication between multiple agents based on the encoding network and the policy network in the target communication neural network.

[0058] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the multi-agent communication method based on communication graph learning of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0059] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multi-agent communication method based on communication graph learning of the above-mentioned first aspect or any corresponding embodiment thereof.

[0060] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the multi-agent communication method based on communication graph learning of the above-mentioned first aspect or any corresponding embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 is a flowchart of a multi-agent communication method based on communication graph learning according to an embodiment of the present invention;

[0063] Figure 2 is a schematic diagram of a communication graph structure according to an embodiment of the present invention;

[0064] Figure 3 is a flowchart of obtaining training data according to an embodiment of the present invention;

[0065] Figure 4 is a flowchart of distributed execution according to an embodiment of the present invention;

[0066] Figure 5 is a structural block diagram of a multi-agent communication device based on communication graph learning according to an embodiment of the present invention;

[0067] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0069] In the game application scenario, multi-agent communication technology can help game agents learn and work together in collaborative tasks, mainly by pre-defining a fixed set of neighbor nodes by the agent, and exchanging information and collaborating through communication with neighbor nodes. Alternatively, the agent broadcasts its own information to all other agents. However, due to the large number of agents in the game and the rapid changes, the above scheme lacks flexibility and adaptability, cannot cope with complex and dynamic game environments, and cannot adaptively learn communication strategies, and cannot continuously learn and improve its own capabilities according to the game task objectives, resulting in poor communication effects and poor robustness in different game tasks and game environments. The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention can better cope with environmental changes and the diversity of task requirements, and can achieve efficient and autonomous distributed communication and collaboration in actual tasks without the direct intervention of the central controller. Each agent makes decisions based on its individual observations and the information transmitted in the network, which enhances the flexibility and robustness of multi-agent communication.

[0070] According to an embodiment of the present invention, an embodiment of a multi-agent communication method based on communication graph learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0071] In this embodiment, a multi-agent communication method based on communication graph learning is provided, which can be used in electronic devices. Figure 1 is a flow chart of a multi-agent communication method based on communication graph learning according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0072] Step S101, when multiple agents start this round of training, obtain the full communication graph structure and the communication neural network obtained in the previous round of training. The communication neural network includes a coding network, a scheduling network, a strategy network and an evaluation network, which are used to realize the communication between multiple agents. The full communication graph structure indicates that multiple agents can communicate arbitrarily. Specifically, taking the game application scenario of 3marine vs 3marine as an example, the multi-agent communication method based on communication graph learning in the embodiment of the present invention is described in detail. This scenario involves a battle between two teams, each team consists of 3 agents, and a total of 6 agents participate in the game, of which 3 belong to one side, called "our side", and the other 3 belong to the other side, called "the enemy side". The main task of each agent is to help its team defeat the enemy team. This process involves multiple behaviors such as coordinated attack, defense, resource collection, etc. Therefore, multiple agents participating in the game can communicate through the communication neural network and the communication graph structure to realize the above-mentioned multiple behaviors. Agents need to learn how to make decisions in a complex game environment in order to maximize the survivability and attack efficiency of their own team. At the beginning of each game, multiple agents communicate during the game according to the full communication graph structure, and the communication graph structure after each time step is output through the strategy network of the communication neural network to reflect the communication process between agents. Figure 2 is a schematic diagram of a communication graph structure according to an embodiment of the present invention, such as Figure 2 As shown in the figure, taking four agents: agent A, agent B, agent C and agent D as examples, each agent is taken as a node of the communication graph structure, and the observation information of each agent is taken as the node feature, and a directed edge is constructed based on the communication between agents. For example, agent A sends information to agent B, then a directed edge is constructed between agent A and agent B. Each agent can only observe the observation information within its field of view, such as the position, attributes and terrain of its own and enemy units. The observation information of the agent is encoded through the encoding network in the communication neural network, and the action of the agent is selected through the strategy network, such as the direction of movement, attack target, use of skills, etc. The choice of the agent's action will directly affect the process and result of the game. When the agent successfully attacks the enemy unit or destroys the enemy building, the agent will receive a positive reward; when it is attacked by the enemy or its own unit dies, the agent will receive a negative reward. The above positive rewards or negative rewards are determined through the evaluation network, and the rewards are used as feedback signals in reinforcement learning to help the intelligent agent continuously optimize its strategy by obtaining the communication neural network obtained in the previous round of training, so as to optimize it based on the results of the previous round of training.

[0073] Step S102, based on the communication neural network and the full communication graph structure obtained in the previous round of training, multiple sets of training data for this round of training are obtained. Specifically, the above-mentioned multiple agents are made to communicate in the game application scenario of 3marine vs 3marine based on the full communication graph structure and the communication neural network obtained in the previous round of training to obtain multiple sets of training data for this round of training.

[0074] Step S103, based on the multiple sets of training data, the communication neural network obtained in the previous round of training is updated to obtain the communication neural network for this round of training, and the current round of training is completed. Specifically, based on the multiple sets of training data obtained above, the communication neural network obtained in the previous round of training is updated to improve the accuracy of the agent's decision-making and the overall coordination of the team.

[0075] Step S104, repeat the above training process, perform multiple rounds of training, and obtain the target communication neural network. Specifically, repeat the above steps S101 to S103, perform multiple rounds of training, each round of training will be optimized on the basis of the previous round of training, and finally obtain the target communication neural network, which can better cope with environmental changes and the diversity of task requirements, and effectively guide the intelligent agents to cooperate in complex game environments.

[0076] Step S105, based on the coding network and the strategy network in the target communication neural network, the communication between the multiple agents is executed in a distributed manner. Specifically, by using the coding network and the strategy network in the target communication neural network, the multiple agents can achieve efficient and accurate communication in a complex game environment, which is not limited to direct instruction transmission, but also achieves synchronization and adaptation at the strategy level, thereby enhancing the flexibility and robustness of the multiple agents in the face of complex environments.

[0077] The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention obtains the full communication graph structure and the communication neural network obtained in the previous round of training at the beginning of each round of training, so that the agent can further optimize the communication strategy based on previous learning experience, and then generates training data for this round of training through the training results of the previous round, and uses the training data to update the communication neural network obtained in the previous round of training to gradually approach the optimal communication strategy. After multiple rounds of such training iterations, a highly optimized target communication neural network is obtained, which can better cope with environmental changes and the diversity of task requirements. By using the encoding network and strategy network in the target communication neural network, the agent can achieve efficient and autonomous distributed communication and collaboration in actual tasks without the direct intervention of the central controller. Each agent makes decisions based on its individual observations and the information transmitted in the network, thereby enhancing the flexibility and robustness of multi-agent communication.

[0078] In this embodiment, a multi-agent communication method based on communication graph learning is provided, which can be used for the above-mentioned electronic device. The method specifically includes the following steps:

[0079] Step S201, when multiple agents start this round of training, obtain the full communication graph structure and the communication neural network obtained in the previous round of training. The communication neural network includes the encoding network, the scheduling network, the strategy network and the evaluation network, which are used to realize the communication between multiple agents. The full communication graph structure indicates that multiple agents can communicate with each other at will. For details, please refer to Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0080] Step S202, based on the communication neural network and the full communication graph structure obtained in the previous round of training, multiple groups of training data for this round of training are obtained.

[0081] Specifically, the above step S202 includes:

[0082] Step S2021, in the first time step of this round of training, for any agent, obtain the node features of the node corresponding to the agent in the first time step from the full communication graph structure, and the node features represent the observation information of the agent. Specifically, the communication graph structure describes the framework of how agents exchange information, and the full communication graph structure indicates that multiple agents can communicate arbitrarily. Each agent is a node in the communication graph structure, and the node features represent the observation information of the agent.

[0083] Step S2022, using the coding network in the communication neural network obtained in the previous round of training, encodes the node features of the intelligent agent to obtain a coding vector.

[0084] In some optional implementations, the above step S2022 includes:

[0085] Step a1, linearly transform the node features of the agent to obtain a feature vector. Specifically, the encoding network is encoded through the graph attention mechanism to obtain an encoding vector. First, linearly transforming the node features through a weight matrix helps to extract more abstract and task-related feature representations from the node features.

[0086] Step a2, determine the attention coefficient of the feature vector. Specifically, the feature vector is divided into multiple parts, each of which corresponds to a head. For each head, the weights learned from the encoding network are multiplied by the feature vector to obtain an intermediate representation for each node in each head. For each pair of nodes in each head, if the pair of nodes is in the same head, their intermediate representations are added. The sum of the intermediate representations of each pair of nodes is input into the LeakyReLU activation function for nonlinear transformation to obtain the attention coefficient.

[0087] Step a3: normalize the attention coefficient to obtain the attention score. Specifically, the softmax function is used to normalize the attention coefficient of each pair of nodes in the feature vector to obtain the attention score.

[0088] Step a4, based on the attention score, update the node features of the agent to obtain an aggregated feature vector. Specifically, for each node in the feature vector, perform a weighted sum based on the attention scores of the node pairs to which the node belongs, that is, aggregate the neighbor nodes of the node to obtain an aggregated feature vector.

[0089] Step a5, adding a bias to the aggregated feature vector to obtain a coding vector. Specifically, adding a bias term to the aggregated feature vector helps to learn the offset characteristics of the data, so that the coding vector better adapts to the subsequent processing requirements of the model.

[0090] Step S2023, using the policy network in the communication neural network obtained in the previous round of training, based on the encoding vector, determines the action of the agent in the first time step and the predicted communication object in the next time step.

[0091] In some optional implementations, the above step S2023 includes:

[0092] Step b1, using the first linear layer of the policy network, linearly transforms the coding vector and multiple available action vectors preset in the policy network to obtain the linearly transformed coding vector and multiple available action vectors. Specifically, the first linear layer is used to transform the coding vector and multiple available action vectors preset in the policy network into feature space to better adapt to subsequent processing.

[0093] Step b2, using the recurrent neural network layer of the policy network, obtains the time series characteristics of the linearly transformed encoding vector and multiple available action vectors to obtain the hidden state. Specifically, a recurrent neural network layer (RNN) is used to process the transformed vector obtained in the previous step to capture the time series characteristics in the data, which helps to predict future actions based on historical behaviors. The RNN layer traverses the transformed encoding vector and available action vectors, and updates its hidden state time by time step.

[0094] Step b3, using the second linear layer of the policy network, based on the hidden state, generates the action values ​​of multiple available action vectors and determines the action of the first time step. Specifically, based on the hidden state generated by the RNN layer, each available action vector is converted by the second linear layer to obtain a scalar value, which represents the expected benefit of the agent taking the action, that is, the action value. By comparing the values ​​of all available actions, the principle of maximizing expected benefits, such as a greedy strategy, can be used to determine the best action that the agent should take in the first time step.

[0095] Step b4, using the communication information generation layer of the policy network, based on the hidden state, determines the predicted communication object for the next time step. Specifically, the hidden state of the RNN layer is transformed by the linear layer in the communication information generation layer, so that features directly related to the communication decision are extracted from the complex hidden representation, and then the output of the linear layer is mapped to the (0,1) interval through the sigmoid activation function. An output close to 0 indicates a low communication demand, and an output close to 1 indicates a strong communication demand. The node closest to 1 is used as the predicted communication object for the next time step.

[0096] Step S2024, using the evaluation network in the communication neural network obtained in the previous round of training, after the agent performs the action of the first time step in the initial environment global state, determines the observation information, environment global state, reward and action value of the agent in the first time step.

[0097] In some optional implementations, the above step S2024 uses the evaluation network in the communication neural network obtained in the previous round of training to make the agent perform the action of the first time step in the initial environment global state, and then determine the action value of the agent in the first time step, including:

[0098] Step c1, obtaining the initial environment global state and the action values ​​of multiple available action vectors. Specifically, the above data is obtained and shaped to be suitable for input into the evaluation network, providing a data basis for subsequent determination of the action value of the current time step.

[0099] Step c2, using the conditional hypernetwork of the evaluation network, based on the initial environment global state, generates a first weight and bias parameter and a second weight and bias parameter. Specifically, the conditional hypernetwork includes two layers, which dynamically generate the first weight and bias parameter and the second weight and bias parameter according to the initial environment global state, respectively. The method of dynamically generating parameters can more flexibly adapt to different state environments, thereby improving the accuracy and generalization ability of the evaluation.

[0100] Step c3, based on the first weight and bias parameter, linearly transform the action values ​​of multiple available action vectors and then apply nonlinear activation to obtain intermediate action values. Specifically, based on the first weight and bias parameter, linearly transform the action values ​​of multiple available action vectors, and then introduce nonlinear activation functions such as ReLU, tanh, etc., so as to better fit complex decision boundaries.

[0101] Step c4, based on the second weight and bias parameter, the intermediate action value is linearly transformed to obtain the action value of the agent at the first time step. Specifically, the intermediate action value is linearly transformed again using the second weight and bias parameter to further refine it, ensuring that the final output action value can more accurately reflect the expected benefits of taking each action under the global state of the environment, and directly reflects the degree of excellence of the agent in executing each action.

[0102] In some optional embodiments, the evaluation network obtained from the previous round of training is used to enable the agent to perform the action of the first time step in the initial global state of the environment. The process of determining the observation information, the global state of the environment and the reward of the first time step is a prior art and will not be repeated here.

[0103] Step S2025, using the policy network in the communication neural network obtained in the previous round of training, based on the observation information of the agent in the first time step and the predicted communication object in the next time step, the full communication graph structure is updated to obtain the communication graph structure of the first time step. Specifically, since the training data includes the observation information and the predicted communication object, when the communication graph structure is updated after the training data is obtained, the node features of the full communication graph structure are updated based on the observation information of the first time step, and an edge is constructed between the agent and the predicted communication object, thereby realizing the update of the full communication graph structure and obtaining the communication graph structure of the first time step.

[0104] Step S2026, the communication graph structure of the first time step is used as the communication graph structure for obtaining node features in the next time step, and the global state of the environment in the first time step is used as the initial state of the environment in the next time step. The above process of obtaining observation information, actions, global state of the environment, rewards, action values, predicted communication objects, and updating the communication graph structure is repeated until all time steps are completed to obtain multiple sets of training data for this round of training. Specifically, for the first time step of each round of training, based on the communication neural network and the full communication graph structure of the previous round of training, the training data of the first time step is obtained through the above steps S2021 to S2025. For each round of training except the first time step, the training data of the corresponding time step is obtained through the above steps S2021 to S2025 based on the communication graph structure of the previous time step and the communication neural network of the previous round of training. That is, the second time step obtains the node features from the communication graph structure of the first time step, and continues to execute steps S2022 and S2025 to obtain the training data of the second time step. In which, when referring to step S2024, the global state of the environment obtained in the first time step is used as the initial environment state, the action of the second time step is performed, and the training data corresponding to the second time step is obtained. By repeating the above process, multiple sets of training data for this round of training are obtained to provide data support for subsequent training.

[0105] In some optional embodiments, Figure 3 is a flowchart of obtaining training data according to an embodiment of the present invention. Figure 3 As shown in the figure, for any time step except the first time step, the observation information of the agent is obtained from the communication graph structure updated in the previous time step, and input into the encoding network to obtain the encoding vector. Then, the encoding vector is input into the policy network to obtain the action of the agent in the current time step and the predicted communication object in the next time step. Then, the agent is made to perform the action in the global state of the environment in the previous time step, and the global state of the environment, reward and action value are obtained based on the evaluation network. The communication graph structure is updated through the predicted communication object of the next time step to obtain the communication graph structure of the current time step. Based on the updated communication graph structure, the next set of training data is obtained.

[0106] Step S203, based on the multiple sets of training data, the communication neural network obtained in the previous round of training is updated to obtain the communication neural network for this round of training, thereby completing this round of training.

[0107] Specifically, the above step S203 includes:

[0108] Step S2031, for any set of training data, based on the difference between the communication graph structure of the training data before and after the update during the training process, the communication loss is obtained. Specifically, if the set of training data is obtained at the first time step of any round of training, the difference between the full communication graph structure and the communication graph structure of the first time step is determined. If the set of training data is obtained at other time steps except the first time step, the difference between the communication graph structure of the current time step and the communication graph structure after the update of the previous time step is determined, that is, the difference between the two communication graph structures when executing steps S2021 and S2025 at each time step, which can reflect the changes in the communication situation between the agents, such as which agents have newly added communication connections, which connections have changed in strength, etc. The cross entropy loss is calculated based on the difference between the communication graph structures as the communication loss, which is used to measure the communication efficiency between the agents.

[0109] Step S2032, determining the target action value of the training data. Specifically, obtaining the reward of the set of training data, and obtaining the maximum action value from the action values ​​of all possible action vectors. Based on the reward, the maximum action value and a preset discount factor, determining the target action value of the training data.

[0110] Step S2033, based on the difference between the target action value and the action value of the training data, the action value loss is determined. Specifically, the mean square error loss or Huber loss between the target action value and the action value of the training data is calculated as the action value loss, which indicates the accuracy in evaluating different action values ​​and is the key to optimizing network parameters.

[0111] Step S2034, based on the communication loss and action value loss of each set of training data, the network parameters of the communication neural network obtained in the previous round of training are optimized to obtain the communication neural network for this round of training. Specifically, based on the communication loss and action value loss of each set of training data, the network parameters of the communication neural network are adjusted through the gradient descent algorithm so that the scheduling network generates the best communication graph, the encoding network effectively encodes the state of the intelligent agent, the strategy network generates the optimal action selection strategy and communication strategy, and the evaluation network optimizes the rewards and action values ​​between each intelligent agent. The above process is performed once based on each set of training data until the update is completed based on all training data to obtain the communication neural network for this round of training.

[0112] Step S204, repeat the above training process, perform multiple rounds of training, and obtain the target communication neural network. Specifically, the above steps S201 to S203 are a round of training process, and the target communication neural network is obtained by repeating the above process until the preset rounds of training are completed, so as to better cope with environmental changes and the diversity of task requirements.

[0113] Step S205, based on the encoding network and the strategy network in the target communication neural network, the communication between multiple agents is executed in a distributed manner.

[0114] Specifically, the above step S205 includes:

[0115] Step S2051, based on the communication between multiple agents and multiple agents, generate and execute a communication graph structure. Specifically, based on the communication between multiple agents, each node represents an agent, and the edge between nodes represents that two agents can exchange information, and a communication graph structure is generated, which depicts how the agents communicate with each other, determines the path and mode of information flow, and helps the subsequent communication process.

[0116] Step S2052, based on the coding network in the target communication neural network, the observation information of multiple agents is encoded to obtain multiple target coding vectors. Specifically, referring to the above step S2022, each agent collects its observation information and inputs it into the coding network, converting the observation information into a target coding vector to make it more suitable for further processing and communication.

[0117] Step S2053, based on the policy network in the target communication neural network and multiple target encoding vectors, obtain the action of each agent at the current moment and the communication object at the next moment. Specifically, referring to the above step S2023, the policy network generates the action that should be performed at the current moment and the communication object at the next moment for each agent based on the target encoding vector.

[0118] Step S2054, each agent is made to perform the action at the current moment. Specifically, according to the action determined by the policy network, each agent is made to perform the action calculated by it, such as movement, resource allocation, signal transmission, etc., which helps to obtain new observation information for each agent.

[0119] Step S2055, based on the execution communication graph structure, each agent communicates with the communication object at the next moment, obtains the observation information received by each agent, returns to the encoding network based on the target communication neural network, encodes the observation information of multiple agents, and obtains multiple target encoding vectors, until the observation information cannot be obtained. Specifically, the agent exchanges information with the communication object at the next moment specified by the strategy network, simulating the collaboration and information sharing among multiple agents. The agent sends the received new observation information to the encoding network again for processing, and generates an updated target encoding vector. By repeating the above process, the agent is allowed to continuously adapt to environmental changes, optimize its behavior strategy and communication strategy based on the observation information, without the direct intervention of the central controller, and each agent makes decisions based on its individual observations and the information transmitted in the network, which enhances the flexibility and robustness of multi-intelligence and realizes communication and collaboration among multiple agents.

[0120] In some optional embodiments, Figure 4 is a flowchart of distributed execution according to an embodiment of the present invention, such as Figure 4 As shown in the figure, it is assumed that there are agents A, B and C. The three agents input the observation information into the encoding network respectively, and the encoding network encodes and obtains three target encoding vectors. Then, the three target encoding vectors are input into the policy network to obtain the actions that the three agents should take and the communication objects at the next moment. Based on the communication objects of the three agents at the next moment, the execution communication graph structure is updated. At the next moment, the three agents continue to communicate based on the execution communication graph structure to obtain new observation information, and the above process is repeated until no observation information can be obtained.

[0121] The multi-agent communication method based on communication graph learning provided by the embodiment of the present invention obtains the full communication graph structure and the communication neural network obtained in the previous round of training at the beginning of each round of training, so that the agent can further optimize the communication strategy based on previous learning experience, and then generates training data for this round of training through the training results of the previous round, and uses the training data to update the communication neural network obtained in the previous round of training to gradually approach the optimal communication strategy. After multiple rounds of such training iterations, a highly optimized target communication neural network is obtained, which can better cope with environmental changes and the diversity of task requirements. By using the encoding network and strategy network in the target communication neural network, the agent can achieve efficient and autonomous distributed communication and collaboration in actual tasks without the direct intervention of the central controller. Each agent makes decisions based on its individual observations and the information transmitted in the network, thereby enhancing the flexibility and robustness of multi-agent communication.

[0122] In this embodiment, a multi-agent communication device based on communication graph learning is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0123] This embodiment provides a multi-agent communication device based on communication graph learning, such as Figure 5 As shown, including:

[0124] The first acquisition module 501 is used to obtain the full communication graph structure and the communication neural network obtained in the previous round of training when multiple intelligent agents start this round of training. The communication neural network includes a coding network, a scheduling network, a strategy network and an evaluation network, which are used to realize communication between multiple intelligent agents. The full communication graph structure indicates that multiple intelligent agents can communicate arbitrarily.

[0125] The second acquisition module 502 is used to acquire multiple sets of training data for this round of training based on the communication neural network and the full communication graph structure obtained in the previous round of training.

[0126] The updating module 503 is used to update the communication neural network obtained in the previous round of training based on multiple sets of training data, obtain the communication neural network of the current round of training, and complete the current round of training.

[0127] The training module 504 is used to repeat the above training process, perform multiple rounds of training, and obtain the target communication neural network.

[0128] The communication module 505 is used to perform distributed communication between multiple agents based on the encoding network and the strategy network in the target communication neural network.

[0129] In some optional implementations, any set of training data includes a communication graph structure, observation information, actions, global state of the environment, rewards, action values, and predicted communication objects, and the second acquisition module 502 includes:

[0130] The acquisition unit is used to obtain the node features of the node corresponding to the intelligent agent in the first time step for any intelligent agent from the full communication graph structure in the first time step of this round of training. The node features represent the observation information of the intelligent agent.

[0131] The first encoding unit is used to encode the node features of the intelligent agent using the encoding network in the communication neural network obtained by the previous round of training to obtain the encoding vector.

[0132] The first determination unit is used to adopt the strategy network in the communication neural network obtained by the previous round of training, and determine the action of the intelligent agent in the first time step and the predicted communication object in the next time step based on the encoding vector.

[0133] The second determination unit is used to use the evaluation network in the communication neural network obtained in the previous round of training to make the intelligent agent perform the action of the first time step in the initial environment global state, and then determine the observation information, environmental global state, reward and action value of the intelligent agent in the first time step.

[0134] The first updating unit is used to adopt the policy network in the communication neural network obtained in the previous round of training, based on the observation information of the intelligent agent in the first time step and the predicted communication object in the next time step, to update the full communication graph structure and obtain the communication graph structure of the first time step.

[0135] The third determination unit is used to use the communication graph structure of the first time step as the communication graph structure for obtaining node features in the next time step, and use the global state of the environment in the first time step as the initial environment state for the next time step, and repeat the above process of obtaining observation information, actions, global states of the environment, rewards, action values ​​and predicted communication objects, and updating the communication graph structure until all time steps are completed to obtain multiple sets of training data for this round of training.

[0136] In some optional implementations, the first encoding unit includes:

[0137] The first transformation subunit is used to perform linear transformation on the node features of the agent to obtain a feature vector.

[0138] The first determination subunit is used to determine the attention coefficient of the feature vector.

[0139] The normalization subunit is used to normalize the attention coefficient to obtain the attention score.

[0140] The updating subunit is used to update the node features of the agent based on the attention score to obtain the aggregated feature vector.

[0141] Add a subunit to add bias to the aggregated feature vector to obtain the encoded vector.

[0142] In some optional implementations, the first determining unit includes:

[0143] The second transformation subunit is used to use the first linear layer of the strategy network to linearly transform the coding vector and multiple available action vectors preset in the strategy network to obtain the linearly transformed coding vector and multiple available action vectors.

[0144] The first acquisition subunit is used to adopt the recurrent neural network layer of the policy network to obtain the time series characteristics of the linearly transformed encoding vector and multiple available action vectors to obtain the hidden state.

[0145] The first generating subunit is used to determine the action of the first time step after generating the action values ​​of multiple available action vectors based on the hidden state using the second linear layer of the policy network.

[0146] The fourth determination subunit is used to adopt the communication information generation layer of the strategy network to determine the predicted communication object at the next time step based on the hidden state.

[0147] In some optional implementations, the second determining unit includes:

[0148] The second acquisition subunit is used to obtain the initial environment global state and the action values ​​of multiple available action vectors.

[0149] The second generating subunit is used to generate a first weight and bias parameter and a second weight and bias parameter based on an initial environment global state by using a conditional super network of the evaluation network.

[0150] The third transformation subunit is used to perform linear transformation on the action values ​​of multiple available action vectors based on the first weight and bias parameter and then apply nonlinear activation to obtain intermediate action values.

[0151] The fourth transformation subunit is used to perform a linear transformation on the intermediate action value based on the second weight and bias parameter to obtain the action value of the agent at the first time step.

[0152] In some optional implementations, the updating module 503 includes:

[0153] The second updating unit is used to obtain the communication loss for any set of training data based on the difference between the communication graph structure of the training data before and after the training process.

[0154] The fourth determining unit is used to determine the target action value of the training data.

[0155] The fifth determining unit is used to determine the action value loss based on the difference between the target action value and the action value of the training data.

[0156] The optimization unit is used to optimize the network parameters of the communication neural network obtained in the previous round of training based on the communication loss and action value loss of each group of training data to obtain the communication neural network in this round of training.

[0157] In some optional implementations, the communication module 505 includes:

[0158] The generating unit is used to generate an execution communication graph structure based on the communication conditions between multiple agents and multiple agents.

[0159] The second encoding unit is used to encode the observation information of multiple agents based on the encoding network in the target communication neural network to obtain multiple target encoding vectors.

[0160] The sixth determination unit is used to obtain the action of each intelligent agent at the current moment and the communication object at the next moment based on the policy network in the target communication neural network and based on multiple target encoding vectors.

[0161] The execution unit is used to enable each agent to execute the action at the current moment.

[0162] The communication unit is used to enable each intelligent agent to communicate with the communication object at the next moment based on the execution communication graph structure, obtain the observation information received by each intelligent agent, return to the encoding network based on the target communication neural network, encode the observation information of multiple intelligent agents, and obtain multiple target encoding vectors until the observation information cannot be obtained.

[0163] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0164] The multi-agent communication device based on communication graph learning in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0165] The embodiment of the present invention also provides a computer device having the above Figure 5 A multi-agent communication device based on communication graph learning is shown.

[0166] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0167] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0168] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0169] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0171] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0172] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0173] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0174] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multi-agent communication method based on communication graph learning, characterized in that: The method comprises: When multiple agents start this round of training, a full communication graph structure and a communication neural network obtained in the previous round of training are obtained, wherein the communication neural network includes a coding network, a scheduling network, a strategy network, and an evaluation network, and is used to realize communication between the multiple agents. The full communication graph structure indicates that the multiple agents can communicate with each other at will; Based on the communication neural network obtained in the previous round of training and the full communication graph structure, obtaining multiple sets of training data for the current round of training; Based on the multiple sets of training data, the communication neural network obtained in the previous round of training is updated to obtain the communication neural network of the current round of training, thereby completing the current round of training; Repeat the above training process for multiple rounds of training to obtain the target communication neural network; Based on the encoding network and the strategy network in the target communication neural network, the communication between the multiple agents is performed in a distributed manner.

2. The method according to claim 1, characterized in that Any set of training data includes communication graph structure, observation information, action, global state of environment, reward, action value and predicted communication object. The communication neural network and the full communication graph structure obtained based on the previous round of training are used to obtain multiple sets of training data for this round of training, including: In the first time step of this round of training, for any agent, the node features of the node corresponding to the agent in the first time step are obtained from the full communication graph structure, and the node features represent the observation information of the agent; Using the coding network in the communication neural network obtained in the previous round of training, the node features of the agent are encoded to obtain a coding vector; Using the policy network in the communication neural network obtained by the previous round of training, based on the encoding vector, determine the action of the agent in the first time step and the predicted communication object in the next time step; Using the evaluation network in the communication neural network obtained in the previous round of training, after the agent performs the action of the first time step in the initial environment global state, the observation information, environment global state, reward and action value of the agent in the first time step are determined; Adopting the policy network in the communication neural network obtained in the previous round of training, based on the observation information of the agent in the first time step and the predicted communication object in the next time step, updating the full communication graph structure to obtain the communication graph structure of the first time step; The communication graph structure of the first time step is used as the communication graph structure for obtaining node features in the next time step, and the global state of the environment in the first time step is used as the initial state of the environment in the next time step. The above process of obtaining observation information, actions, global state of the environment, rewards, action values, predicted communication objects, and updating the communication graph structure is repeated until all time steps are completed to obtain multiple sets of training data for this round of training.

3. The method according to claim 2, characterized in that The coding network in the communication neural network obtained by the previous round of training is used to encode the node features of the agent to obtain the coding vector, including: Performing linear transformation on the node features of the agent to obtain a feature vector; determining an attention coefficient of the feature vector; Normalizing the attention coefficient to obtain an attention score; Based on the attention score, updating the node features of the agent to obtain an aggregated feature vector; A bias is added to the aggregated feature vector to obtain the encoding vector.

4. The method according to claim 2, characterized in that: The strategy network in the communication neural network obtained by the previous round of training is used to determine the action of the agent in the first time step and the predicted communication object in the next time step based on the encoding vector, including: Using the first linear layer of the strategy network, linearly transform the encoding vector and a plurality of available action vectors preset in the strategy network to obtain the linearly transformed encoding vector and a plurality of available action vectors; Using the recurrent neural network layer of the strategy network, the time series characteristics of the linearly transformed encoding vector and multiple available action vectors are obtained to obtain a hidden state; Determine an action for a first time step by using a second linear layer of the policy network to generate action values ​​of the plurality of available action vectors based on the hidden state; The communication information generation layer of the strategy network is used to determine the predicted communication object for the next time step based on the hidden state.

5. The method according to claim 4, characterized in that The method uses the evaluation network in the communication neural network obtained in the previous round of training to make the agent perform the action of the first time step in the initial environment global state, and then determines the action value of the agent in the first time step, including: Obtaining the initial environment global state and the action values ​​of the multiple available action vectors; Using the conditional super network of the evaluation network, based on the initial environment global state, generate a first weight and bias parameter and a second weight and bias parameter; Based on the first weight and bias parameter, linearly transform the action values ​​of the plurality of available action vectors and then apply nonlinear activation to obtain intermediate action values; Based on the second weight and bias parameter, the intermediate action value is linearly transformed to obtain the action value of the agent in the first time step.

6. The method according to claim 2, characterized in that The updating of the communication neural network obtained in the previous round of training based on the multiple sets of training data to obtain the communication neural network in the current round of training includes: For any set of training data, a communication loss is obtained based on the difference between the communication graph structure of the training data before and after the communication graph structure is updated during the training process; determining a target action value of the training data; determining an action value loss based on a difference between a target action value and an action value of the training data; Based on the communication loss and action value loss of each set of training data, the network parameters of the communication neural network obtained in the previous round of training are optimized to obtain the communication neural network in this round of training.

7. The method according to claim 1, characterized in that The distributed execution of communication between the plurality of agents based on the encoding network and the strategy network in the target communication neural network includes: Based on the communication conditions between the plurality of agents and the plurality of agents, generating an execution communication graph structure; Based on the encoding network in the target communication neural network, encoding the observation information of the multiple agents to obtain multiple target encoding vectors; Based on the policy network in the target communication neural network and the multiple target encoding vectors, obtaining the action of each agent at the current moment and the communication object at the next moment; Make each agent perform the action at the current moment; Based on the execution communication graph structure, each agent communicates with the communication object at the next moment, obtains the observation information received by each agent, returns to the encoding network based on the target communication neural network, encodes the observation information of the multiple agents, and obtains the steps of multiple target encoding vectors, until the observation information can no longer be obtained.

8. A multi-agent communication device based on communication graph learning, characterized in that: The device comprises: A first acquisition module is used to acquire a full communication graph structure and a communication neural network obtained in a previous round of training when multiple agents start this round of training. The communication neural network includes a coding network, a scheduling network, a strategy network, and an evaluation network, which are used to realize communication between the multiple agents. The full communication graph structure indicates that the multiple agents can communicate with each other at will; A second acquisition module is used to acquire multiple sets of training data for this round of training based on the communication neural network obtained in the previous round of training and the full communication graph structure; An updating module, used to update the communication neural network obtained in the previous round of training based on the multiple sets of training data, obtain the communication neural network of the current round of training, and complete the current round of training; A training module is used to repeat the above training process, perform multiple rounds of training, and obtain a target communication neural network; A communication module is used to distribute the communication between the multiple agents based on the encoding network and the strategy network in the target communication neural network.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-agent communication method based on communication graph learning as described in any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multi-agent communication method based on communication graph learning as described in any one of claims 1 to 7.

Images