Training system and method for teammate-aware multi-agent coordination communication strategy

Abstract

The application discloses a kind of training system and method of multi-agent cooperation communication strategy based on teammate perception, with the help of teammate modeling method, generate customized, sparse message for teammates in the process of interacting with teammates. Through multi-agent reinforcement learning communication method, the agent can learn efficient cooperation communication strategy that can resist interference under the condition of limited bandwidth, and the agent cooperation communication strategy obtained by training can enable the agent to select reasonable cooperation action according to the observation information obtained by itself, and send information to teammates at appropriate time to promote cooperation, or cooperate according to the teammate information received. The application realizes close and effective cooperation through communication between a small number of nodes in a limited bandwidth actual scene, and can ensure the anti-interference ability in the process of multi-agent cooperation communication.

Description

TECHNICAL FIELD

[0001] The present application relates to a teammate-aware multi-agent communication system and method, in particular to an efficient communication strategy training system and method in cooperative multi-agent control tasks. BACKGROUND

[0002] Multi-agent technology has made great progress in various aspects in recent years, and important achievements have been made in intelligent path planning, intelligent delivery, game playing, military confrontation, etc. Traditional multi-agent cooperation strategy mainly adopts automatic control method, formulates strategy through artificial modeling process, and relies on relevant field experts to specify operation rules and design communication protocols that meet the actual use scenarios. This kind of cooperative communication method is costly and lacks adaptability in complex dynamic environments. When facing different use scenarios, it often relies on field experts to redesign operation strategies and communication rules, which is costly.

[0003] In recent years, with the vigorous development of machine learning, especially deep learning, reinforcement learning technology provides a new solution for multi-agent cooperation. Reinforcement learning enables agents to learn in interaction with the environment. Agents obtain rewards or punishments through interaction with the environment, thereby adapting to the environment and learning strategies that can maximize rewards from the environment. In cooperative multi-agent reinforcement learning, the actions of multiple agents will produce a common reward of the environment, and effective and close multi-agent cooperation strategy can efficiently control the autonomous action of the agent. Multi-agent algorithms based on reinforcement learning have achieved more robust and higher sample efficiency than traditional methods in multi-agent path planning and other fields, providing hope for the development of general artificial intelligence.

[0004] Although multi-agent reinforcement learning algorithms have achieved some results, in many real application scenarios such as intelligent cooperation confrontation scenarios, the cooperation ability of the strategy based on reinforcement learning will be affected by the partial observability of its own information. In such scenarios, agents often only have local information of the environment and cannot accurately make optimal actions, thereby affecting the coordination ability of the agents. Communication can enhance the cooperation ability between agents to some extent, but in real environments, communication between agents is often disturbed by various factors, such as limited communication bandwidth and unguaranteed communication reachability, thereby greatly affecting the performance of the cooperation strategy obtained by reinforcement learning. SUMMARY

[0005] Invention purposes: based on the current problems of multi-agent communication, how to train a robust cooperative communication strategy in a bandwidth-limited environment, that is, the problem to be solved by the invention. The invention proposes a training system and method for multi-agent cooperative communication strategy based on teammate awareness, which realizes close and effective cooperation through communication between a small number of nodes in a bandwidth-limited actual scene, and can guarantee the anti-interference ability in the process of multi-agent cooperative communication. The method generates customized and sparse messages for teammates in the process of interacting with teammates with the help of teammate modeling method.

[0006] Technical scheme: a training system for multi-agent cooperative communication strategy based on teammate awareness, which enables agents to learn efficient cooperative communication strategies that can resist interference under bandwidth-limited conditions through multi-agent reinforcement learning communication method. The trained agent cooperative communication strategy can enable agents to select reasonable cooperative actions according to the observation information obtained by themselves, and send information to teammates at appropriate times to promote cooperation, or cooperate according to the received teammate information.

[0007] The system constructs an agent simulation control environment to train multi-agent cooperative communication strategy, reduces the cost of strategy learning and improves the learning efficiency. It includes three modules:

[0008] Individual value network module, used to judge the action value from the observation of itself;

[0009] Distributed teammate modeling module, used to model teammates from historical information;

[0010] Information generation module, used to generate sparse and targeted teammate information.

[0011] The three modules proposed can be trained simultaneously in an end-to-end manner through reinforcement learning method.

[0012] The agent simulation control environment E can be modeled as a distributed partially observable Markov decision process <S,A,P,R,Ω,O,γ>, any agent i in the scene receives observation information o i ∈Ω at each decision step, and selects executable action a i from the action space A to make decisions.

[0013] The individual value network module uses a fully connected neural network representation, and can obtain the optimal action and its maximum return under the single-step agent control process according to the given observation information. In the training stage, the module uses a hybrid network combining all the teammate individual value networks in the scene, and uses the experience replay method of the deep Q network to learn, and uses the historical data (τ, a, r, τ') in the interaction process between the agent and the simulation control environment in the training process to update the network parameters according to the Bellman equation, to obtain a better network model. The objective function of the reinforcement learning process of the module is as follows:

[0014]

[0015] In the formula, τ is the historical observation information received by all agents in the corresponding time step, a is the corresponding action taken by all agents, θ is the parameter of the individual value network of the agent and the hybrid network used in the training process, Q(τ, a; θ) is the maximum return prediction output by the hybrid network, y=r+γmax a′ Q(τ,a'; θ - ) is the expected target maximum return, θ - is the target network parameter used to improve the training efficiency, which is completely the same as the network to be trained, and the parameter will be updated periodically to the parameter of the network to be trained.

[0016] The distributed teammate modeling module models the corresponding teammate j as a multi-dimensional Gaussian distribution according to the historical observation information τ i and the teammate encoding information d j , and uses the vector z ij sampled from the Gaussian distribution as the teammate representation for information generation. In addition to the above end-to-end reinforcement learning objective function, the module additionally introduces an information-based mutual information objective I(z ij ; a j |τ i ,d j ) to ensure the accuracy of the teammate modeling.

[0017] The information generation module can generate targeted information for different agent teammates based on the teammate information obtained by the distributed teammate modeling module and the self-observation, so as to achieve efficient cooperation. The information generation module uses a self-attention network to integrate the self-observation information and the teammate modeling information, obtains different weights for different teammates, and selectively communicates with part of the teammates according to the weight size. In order to make the communication target more sparse, the information generation module additionally introduces an optimization target based on information entropy, so that the agent information generation module can mainly consider a few important targets during communication. The information generation module can generate information of the same dimension as the corresponding action, and the agent directly adds the available communication information to the output of the individual value network after obtaining the available communication information at each time step, avoiding the need for additional processing of the information by the individual value network.

[0018] A training method of a multi-agent cooperative communication strategy based on teammate awareness, comprising the following steps:

[0019] Step 1: interaction process of the agent with the simulated environment.

[0020] Step 2: using a cooperative multi-agent reinforcement learning method, updating the agent cooperative communication strategy according to the global reward in the interaction process with the simulated environment using a time difference method, the agent cooperative communication strategy including an individual value network, a distributed teammate modeling network and an information generation network, the strategy network finally giving an individual state-action value function, obtaining a global value function through a hybrid network, and updating the individual value network according to the Bellman equation.

[0021] Step 3: using a shallow neural network based on a recurrent neural network and a multilayer perceptron to construct the individual value network of the agent.

[0022] Step 4: using a distributed teammate modeling module to predict teammate information and obtain teammate modeling representation, the teammate modeling module using a multilayer perceptron to obtain the teammate information at the current time step from the trajectory information and representing the teammate information as a multidimensional Gaussian distribution.

[0023] Step 5: using an information generation module based on an attention mechanism to generate sparse information for different agent teammates from the trajectory information obtained from the individual value network and the teammate modeling information obtained from the distributed teammate modeling module, and to perform agent cooperative communication.

[0024] Step 6: after the strategy training converges, completing the training process.

[0025] A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the training method of the multi-agent cooperative communication strategy based on teammate awareness when executing the computer program.

[0026] A computer-readable storage medium storing a computer program for performing the training method of the teammate-aware multi-agent collaborative communication strategy as described above.

[0027] Beneficial effects: Compared with the prior art, the teammate-aware multi-agent collaborative communication strategy training system and method provided by the application can greatly reduce the trial and error cost and training efficiency by perceiving the cooperative teammates without manual modeling and rule making by field experts. The multi-agent collaborative communication strategy can learn reasonable cooperative strategies and communication methods by utilizing efficient distributed teammate modeling, realizes a sparse and robust agent communication mode, and can be applied to complex network scenes with limited bandwidth and other instabilities, and has a wide application prospect. The application verifies the effectiveness and rationality of the application in a typical multi-agent cooperative strike environment, and the experimental results show that the application can effectively improve the multi-agent collaboration performance. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 A schematic diagram of the multi-agent collaborative communication strategy interacting with the agent simulation environment in the embodiments of the application;

[0029] Figure 2 A schematic diagram of the multi-agent collaborative communication strategy training in the simulation environment through a hybrid network in the embodiments of the application;

[0030] Figure 3 A schematic diagram of the agent individual value network in the embodiments of the application;

[0031] Figure 4 A schematic diagram of the agent distributed teammate modeling module in the embodiments of the application;

[0032] Figure 5 A schematic diagram of the agent information generation module in the embodiments of the application;

[0033] Figure 6 A flowchart of the multi-agent collaborative communication strategy training in the embodiments of the application;

[0034] Figure 7 An instance diagram of the multi-agent cooperative strike simulation environment for verification in the embodiments of the application;

[0035] Figure 8 Verification results of the multi-agent collaborative communication strategy training in the above multi-agent cooperative strike simulation environment in the embodiments of the application. DETAILED DESCRIPTION

[0036] The application will be further illustrated by the following specific examples, which are only used to illustrate the application and not used to limit the scope of the application. After reading the application, those skilled in the art can make various equivalent modifications of the application, which fall within the scope defined by the appended claims.

[0037] The training system of the multi-agent cooperation communication strategy based on teammate awareness includes three modules.

[0038] The individual value network module is used for judging the action value from the self observation;

[0039] The distributed teammate modeling module is used for teammate modeling from the self historical information;

[0040] The information generation module is used for generating sparse and targeted teammate information.

[0041] The three modules proposed can be simultaneously trained in an end-to-end manner through the reinforcement learning method.

[0042] The agent simulation control environment E can be modeled as a distributed partially observable Markov decision process <S,A,P,R,Ω,O,γ>, and any agent i in the scene receives observation information o i ∈Ω at each decision step, and selects an executable action a i from the action space A to make a decision.

[0043] The individual value network module uses a fully connected neural network representation, and can obtain the optimal action and its maximum return under a single-step agent control process according to the given observation information. In the training phase, the module uses a hybrid network combining all teammate individual value networks in the scene, uses the experience replay method of the deep Q network for learning, uses the historical data (τ,a,r,τ′) in the interaction process with the simulator (agent simulation control environment) in the training process, updates the network parameters according to the Bellman equation, and obtains a better network model. The objective function of the reinforcement learning process of this module is as follows:

[0044]

[0045] In the formula, τ is the historical observation information received by all agents at the corresponding time step, a is the corresponding action taken by all agents, θ is the parameter of the individual value network of the agent and the hybrid network used in the training process, Q(τ,a;θ) is the maximum return prediction output by the hybrid network, y=r+γmax a′ Q(τ,a′;θ - ) is the expected target maximum return, and θ -The parameters of the target network, which is identical to the network to be trained, are periodically updated to the parameters of the network to be trained to improve training efficiency.

[0046] The distributed teammate modeling module models the corresponding teammate j as a multi-dimensional Gaussian distribution according to historical observation information τ i and teammate encoding information d j , and uses a vector z ij obtained by sampling the Gaussian distribution as a teammate representation for information generation. In addition to the above-mentioned end-to-end reinforcement learning objective function, the module simultaneously introduces an information mutual information-based objective I(z ij ; a j | τ i , d j ) to ensure the accuracy of teammate modeling, and the objective is trained using a loss function based on variational inference.

[0047] The information generation module can generate targeted information for different agent teammates according to the teammate information obtained by the distributed teammate modeling module and the agent's own observation, so as to achieve the purpose of efficient cooperation. The module uses a self-attention network to integrate the agent's own observation information and teammate modeling information to obtain different weights for different teammates, and selectively communicates with part of the teammates according to the weight size. In order to make the communication target more sparse, the information generation module additionally introduces an information entropy-based optimization objective, so that the agent information generation module can mainly consider a small number of important targets when communicating. The information generation module can generate information of the same dimension as the corresponding action, and the agent directly adds the available communication information to the output of the individual value network after obtaining it at each time step, avoiding the need for additional processing of the information by the individual value network.

[0048] The multi-agent cooperation communication method is verified on a cooperative strike simulation environment (see Figure 7 ). In this environment, we need to control 2 trackers and 1 striker, where the tracker's field of view is the shaded area, and the striker's striking range is the circle with the center at the striker. By moving to chase an enemy plane, the enemy plane is struck when it falls within the striker's striking range for 2 consecutive steps after being tracked by the tracker for more than 2 steps. The tracker and the striker can share information about the enemy plane's position and decision-making plan through appropriate communication. The environment contains random obstacles in the horizontal and vertical directions, and the tracking range of the tracker will be affected by the obstacles, while the striking range and communication are not affected by the obstacles. Figure 8The verification results of the multi-agent cooperative communication method of the application and other recent related multi-agent reinforcement learning algorithms VDN, QMIX and QPLEX in the simulation environment are shown, and the experimental results show that the method can achieve a better hit rate than the existing multi-agent reinforcement learning algorithm in the simulation environment. The training method of the multi-agent cooperative communication strategy based on the perception of teammates includes the following steps:

[0049] Step 1: Figure 1 The interaction process of the agent and the agent simulation control environment is shown in the middle. The agent simulation control environment E based on the real system is used for interaction, and the agent simulation control environment defines a distributed partially observable Markov decision process <S, A, P, R, Ω, O, γ> so that the agent can perform reinforcement learning in the simulation environment. The state space S is defined as the environment information and all agent information, the action space A is the set of actions that the agent can take, P is the state transition function of the simulator (agent simulation control environment), R is the reward function of the simulator, Ω is the set of observations that each agent can receive, O is the state observation function of the environment, and γ is the discount factor. Taking the cooperative attack environment verified as an example, the simulator used in the method generally needs to include the following steps to achieve:

[0050] Step 11: Initialize the simulator, initialize the initial state of the simulator, and add the information of each agent. In the cooperative attack simulator environment, initialize the environment map, select random initial positions for all agents and enemy aircraft within a certain range, and initialize each agent at the corresponding position.

[0051] Step 12: Calculate the observation information for each agent according to the state observation function. In the cooperative attack simulator environment, according to the global state information, the local observation of each agent within a certain field of view is given.

[0052] Step 13: After the agent makes an action selection, submit the decision to the simulator. In the cooperative attack simulator environment, the actions that the agent can take include forward, backward, left and right turning, and the striker can additionally take a strike action.

[0053] Step 14: The simulator obtains the actions selected by all agents and calculates the reward at this time step. In the cooperative attack simulator environment, the simulator calculates the reward at this step based on the current state and the joint action submitted by all agents, and the behavior of the agent, including the reward for the agent observing the enemy aircraft, the reward for the tracker finding the enemy aircraft, and the reward for the enemy aircraft entering the attack range of the striker.

[0054] Step 15: The simulator judges whether the hit success condition is met, and if so, the current scenario is ended. In the cooperative attack simulator environment, if the enemy aircraft is tracked by the tracker for more than 2 steps and falls within the range of the attacker for 2 consecutive steps, the enemy aircraft is successfully hit.

[0055] Step 2: Using the cooperative multi-agent reinforcement learning method, the global reward during the interaction with the simulation environment is used to update the agent cooperation communication strategy using the time difference method. The agent cooperation communication strategy includes individual value network, distributed teammate modeling network and information generation network. The strategy network finally gives the individual state action value function, and the global value function is obtained through the hybrid network. According to the Bellman equation, the individual value network is updated, Figure 2 The training process of the agent cooperation communication strategy is shown in the figure.

[0056] Step 21: Initialize the training system of the multi-agent cooperation communication strategy based on teammate perception, including individual value network module, distributed teammate modeling module and information generation module, and hybrid network for multi-agent reinforcement learning training. The agent strategy network can share network parameters according to actual conditions, and the network structure can refer to the following steps.

[0057] Step 22: Collect historical trajectory information during environment interaction to form a reinforcement learning heterogeneous policy update dataset. Use the historical trajectory information of each agent to calculate the individual value function Q i (τ i ,a i ), and input all individual value functions into the hybrid network to calculate the global value function Q(τ,a). The hybrid network for multi-agent reinforcement learning training can use widely used VDN, QMIX, QPLEX and other network structures.

[0058] Step 23: Use the current reward of each step in the historical trajectory information and the maximum return calculated by the target network at the next time to calculate the estimate of the value function in the current state as the update target. The update target formula is y=r+γmax a′ Q(τ,a′;θ - ). Using the double Q network, the optimal action at the next time is selected using the real Q network to alleviate the overestimation problem of the value function estimate. The time difference error is used as the objective function to update the agent cooperation communication strategy network and the hybrid network. The objective function is:

[0059]

[0060] Step 3: Use a shallow neural network based on recurrent neural network and multilayer perceptron to construct the individual value network of the agent, Figure 3 The framework of the individual value network of the agent is shown in the figure.

[0061] Step 31: Use a recurrent neural network to encode the agent observation information input from the individual value network into historical trajectory information. The recurrent neural network used here is implemented as GRU or LSTM, etc., and the recurrent neural network contains 64 hidden layer units.

[0062] Step 32: Use a multilayer perceptron to process the historical trajectory information output by the recurrent neural network to obtain the local value function value of the agent for each executable action. The multilayer perceptron uses a single hidden layer neural network with ReLU activation function.

[0063] Step 33: Combine the local value function output by the multilayer perceptron with the received communication information from other agents through vector addition to obtain the individual value function output of the agent.

[0064] Step 4: Use the distributed teammate modeling module to predict teammate information and obtain teammate modeling representation. The teammate modeling module uses a multilayer perceptron to obtain teammate information at the current time step from the trajectory information and represents it as a multidimensional Gaussian distribution. Figure 4 The calculation process of the teammate modeling module is demonstrated.

[0065] Step 41: Use a multilayer perceptron to process the trajectory information τ calculated by agent i in the individual value network. i As input, the network output is the multidimensional Gaussian distribution parameters μ to be modeled. ij , The multilayer perceptron used can be a single-hidden-layer neural network with the activation function ReLU.

[0066] Step 42: Use the calculated multidimensional Gaussian distribution parameters μ ij , Obtain the corresponding distribution, and sample from the distribution to obtain the modeling information z for the corresponding agent teammate j. ij .

[0067] Step 43: Calculate the optimization objective I(z) using a loss function based on mutual information. ij ;a j |τ i ,d j And maximize this objective during training. Since the mutual information is difficult to compute, its lower bound can be optimized during training, i.e., by introducing a variational distribution q. ξ (z ij |τ i ,a j ,d j In the case of minimizing the KL divergence term J m (θ)=D KL (p(z ij |τ i ,dj ||q ξ (z ij ||τ i ,a j ,d j ))。

[0068] Step 5: Using the information generation module based on attention mechanism, the trajectory information obtained from the individual value network and the teammate modeling information obtained from the distributed teammate modeling module, generate sparse information targeted at different agent teammates. Figure 5 The calculation process of the information generation module is shown.

[0069] Step 51: Using the trajectory information τ i , pass through the fully connected network to calculate the query information q i of the agent i, where the fully connected network is a single-layer linear model.

[0070] Step 52: Using the modeling information z ij of the teammate agent j, pass through the fully connected network to calculate the key value k jj of the agent j, where the fully connected network is a single-layer linear model.

[0071] Step 53: Using the trajectory information τ i and the modeling information z ij of the teammate agent j, pass through the multi-layer perception model to calculate the preliminary information value v ij of the agent i to the agent j, where the single-hidden-layer neural network with ReLU activation function can be used as the multi-layer perception.

[0072] Step 54: Calculate the inner product of the query information q i of the agent i and the key value k ij of the agent j, and pass the calculation results of the key value of all other agents of the agent i through the Softmax function to obtain the weight of the agent i to all other agents.

[0073] Step 55: Clip the weight α ij of the agent i to the agent j, if the weight is less than the set threshold, clip it to 0, and the threshold can be taken as where n is the number of agents.

[0074] Step 55: Calculate the final information value m ij of the agent i to the agent j by using the weight α ij of the agent i to the agent j and the preliminary information value v ij of the agent i to the agent j, and send the information to the agent j.

[0075] Step 56: all the weights generated by the agent i are sparsified by minimizing the objective function J c (θ) = -∑ i ∑ j α ij logα ij .

[0076] Step 6: after the policy training reaches convergence, the training process is completed.

[0077] Obviously, those skilled in the art should understand that the steps of the training method of the teammate-aware multi-agent cooperative communication policy of the embodiments of the present application or the modules of the training system of the teammate-aware multi-agent cooperative communication policy described above can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in different order, or they can be made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to realize. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

Claims

1. A training system for teammate-aware multi-agent coordination communication strategy, characterized in that, The system constructs an agent simulation operating environment to train a multi-agent cooperative communication strategy, and includes three modules. An individual value network module is used to determine action value from self observation; A distributed teammate modeling module is used to model teammates from self historical information; An information generation module is used to generate sparse and targeted teammate information; The three modules can be simultaneously trained in an end-to-end manner through a reinforcement learning method; The individual value network module uses a fully connected neural network to represent the module, and can obtain the optimal action and maximum return in a single-step agent control process according to the given observation information; in the training stage, the module uses a hybrid network combining all teammate individual value networks in the scene, uses the experience replay method of the deep Q network to learn, uses the historical data in the interaction process with the agent simulation operating environment during the training process, updates the network parameters according to the Bellman equation, and obtains a better network model; The distributed teammate modeling module models the corresponding teammates as a multi-dimensional Gaussian distribution according to the historical observation information and teammate coding information, and uses the vector obtained by sampling the Gaussian distribution as the teammate representation for information generation; in addition to the above end-to-end reinforcement learning objective function, the module additionally introduces an objective function based on mutual information to ensure the accuracy of teammate modeling; The information generation module can generate targeted information for different agents according to the teammate information obtained by the distributed teammate modeling module and the self observation; the information generation module uses a self-attention network to integrate self observation information and teammate modeling information, obtains different weights for different teammates, and selectively communicates with part of the teammates according to the weight size; in order to make the communication target more sparse, the information generation module additionally introduces an optimization objective based on information entropy, so that the agent information generation module can mainly consider a small number of important targets when communicating; the information generation module can generate information of the same dimension as the corresponding action, and the agent directly adds the available communication information to the output of the individual value network after obtaining the information at each time step, avoiding the need for additional processing of the information by the individual value network.

2. The system for training a teammate-aware based multi-agent collaboration communication policy according to claim 1, wherein, The intelligent agent simulation control environment It can be modeled as a distributed partially observable Markov decision process. Any intelligent agent in the scene Receive observation information at each decision step and from the action space Select an action that can be executed. Decision making, state space Defined as environmental information and information of all intelligent agents. This is the state transition function for the simulator. This is the reward function for the simulator. The set of observations that each agent may receive. Let this be the state observation function of the environment. This is the discount factor. 3.The system for training a teammate-aware based multi-agent collaboration communication policy according to claim 1, wherein, The individual value network module uses a full connection neural network representation, and can obtain an optimal action and a maximum return under a single-step agent control process according to given observation information; in a training stage, the module uses a mixed network combining all teammate individual value networks in a scene, uses an experience replay method of a deep Q network to learn, uses historical data in an interaction process between the agent and a simulation control environment in a training process , updates network parameters according to a Bellman equation, obtains a better network model, and a target function of the reinforcement learning process of the module is as follows: ； wherein is the history of observations received by all agents in the corresponding time step, is the corresponding action taken by all agents, is the parameter of the individual value network for the agent and the hybrid network used during the training process, is the maximum return prediction output by the hybrid network, is the expected target maximum return, is the discount factor, is the target network parameter used to improve the training efficiency, which is identical to the network to be trained and its parameters will be periodically updated to the parameters of the network to be trained.

4. The system for training a teammate-aware based multi-agent collaboration communication policy of claim 1, wherein, The distributed teammate modeling module is based on historical observation information. Teammate coding information , will the corresponding teammate The model is constructed as a multidimensional Gaussian distribution, and the vector obtained by sampling from this Gaussian distribution is used. As a teammate representation, it is used for information generation; in addition to the aforementioned end-to-end reinforcement learning objective function, this module introduces an additional objective based on mutual information. This is to ensure the accuracy of teammate modeling.

5. A method for training a teammate-aware based multi-agent cooperative communication strategy, characterized in that, The method comprises the following steps: Step 1: the interaction process of the agent with the agent simulation operating environment; Step 2: using a cooperative multi-agent reinforcement learning method, updating the agent cooperative communication strategy network according to the global reward in the interaction process with the simulation operating environment using the time difference method; the agent cooperative communication strategy network includes an individual value network, a distributed teammate modeling network and an information generation network; the agent cooperative communication strategy network finally gives the individual state action value function, and the global value function is obtained through a hybrid network, and the individual value network is updated according to the Bellman equation; Step 3: using a shallow neural network based on a recurrent neural network and a multilayer perceptron to construct the individual value network of the agent; Step 4: using the distributed teammate modeling module to predict teammate information to obtain teammate modeling representation; the teammate modeling module uses a multilayer perceptron to obtain the teammate information at the current time step from the trajectory information, and represents the information as a multi-dimensional Gaussian distribution; Step 5: using the attention mechanism-based information generation module, the trajectory information obtained from the individual value network and the teammate modeling information obtained from the distributed teammate modeling module, generate sparse information targeted at different agent teammates for agent cooperative communication; specifically including the following steps: Step 51: Use trajectory information Computational agents are connected via a fully connected network. Query information The fully connected network here is a single-layer linear model; Step 52: using the modeling information of the teammate agent , a key value of the agent is calculated through a fully connected network, where the fully connected network is a single-layer linear model ​​ Step 53: using trajectory information and modeling information of teammate agents , through a multi-layer perception model, the preliminary information value of the agent to the agent is calculated ;​ Step 54: Connect the intelligent agent Query information With intelligent agents key value Calculate the inner product, and then... The calculation results of all other agent key values ​​are passed through the Softmax function to obtain the agent's value. Weights for all other agents; Step 55: clipping the weights of the agent clipping the weights of the agent if the weight is less than a set threshold, then clip it to 0 ​ Step 55: Connect the intelligent agent For intelligent agents weight With intelligent agents For intelligent agents Preliminary information value To obtain the intelligent agent For intelligent agents The final information value, and for the intelligent agent Send this message; Step 56: The agent All the weights generated, using information entropy as the objective function, are sparsified by minimizing the objective function, which is ; Step 6: after the strategy training converges, the training process is completed.

6. The method of claim 5, wherein the training of the teammate-aware multi-agent collaboration communication policy is based on a reinforcement learning algorithm. Controlling an environment using agent simulations built based on real systems Interacting, the agent simulation controlling environment defines a distributed partially observable Markov decision process Enabling reinforcement learning of an agent in the agent simulation controlling environment; state space defined as environment information and all agent information, action space for a set of actions that an agent can take, for a state transition function of the simulator, for a reward function of the simulator, for a set of observations that each agent can receive, for a state observation function of the environment, for a discount factor; the simulator comprises the following steps: Step 11: initialize the simulator, initialize the initial state of the simulator, and add information of each agent; Step 12: calculate the observation information for each agent according to the state observation function; Step 13: the agent makes a decision according to its own observation and teammate information, selects an action and submits it to the simulator; Step 14: the simulator obtains the action selected by all agents and calculates the reward at this time step.

7. The method of claim 5, wherein the training of the teammate-aware multi-agent collaboration communication policy is based on a reinforcement learning algorithm. The step 2 includes the following steps: Step 21: initialize the agent strategy network and the training hybrid network, the agent strategy network shares network parameters according to the actual situation, and the network structure is referred to the following steps; Step 22: Collect historical trajectory information in the process of environmental interaction to constitute a reinforcement learning heterogeneous policy update data set, calculate individual value functions by using historical trajectory information of each agent , and input all individual value functions into a mixed network to calculate a global value function ​ Step 23: Compute the estimate of the value function at the current state using the current reward of each step in the history trajectory information and the maximum return at the next time instant computed by the target network as the update target, which is formulated as ; Using double Q network, the optimal action at the next time is selected by the real Q network to alleviate the overestimation problem of value function estimation; using the time difference error as the objective function to update the agent cooperative communication strategy network and the hybrid network, the objective function is: .

8. The method of claim 5, wherein the training of the teammate-aware multi-agent collaboration communication policy is based on a reinforcement learning algorithm. The implementation process of the distributed teammate modeling module is as follows: Step 41 : using a multi-layer perceptron to train the agent Trajectory information computed in the individual value network As input, the network outputs the parameters of the multidimensional Gaussian distribution to be modeled ; Step 42: Using the computed multi-dimensional Gaussian distribution parameters get the corresponding distribution, and sample from the distribution to get the modeling information of the corresponding teammate of the agent ;​ Step 43: Compute optimization objective using mutual information based loss function , and maximize this objective during training, where, The encoding information for teammates is difficult to compute due to the mutual information, so we optimize a lower bound of it during training, i.e., minimize the KL divergence term with respect to the variational distribution .

9. A computer device, characterized by: The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the above computer program to implement the training method of the multi-agent cooperative communication strategy based on teammate awareness as claimed in any one of claims 5-8.

Images