A communication reinforcement learning algorithm for promoting multi-agent cooperation

Abstract

The application belongs to the field of multi-agent cooperation and competition, and relates to a multi-agent communication reinforcement learning algorithm based on deep learning. The algorithm first calculates the interaction relationship between agents according to the current observed environment state and the messages sent by other agents, and then assigns attention weights to other agents based on the interaction relationship between agents to select a top strategy. Selecting different top strategies means that there is division of labor between agents. Then in the bottom strategy, the agent selects an appropriate action according to the selected high-level strategy and local observation. The application is based on a hierarchical deep reinforcement learning model, which can achieve good cooperation in a large-scale multi-agent system, which is very similar to the way human groups solve complex problems, that is, by communicating to select a top strategy instead of directly deciding an action. The model not only has a low communication frequency, but also has good scalability.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent cooperation and competition, and relates to a multi-agent communication reinforcement learning algorithm based on deep learning. Background Technology

[0002] Cooperative multi-agent reinforcement learning (MARL) is becoming increasingly popular and has been applied in many fields, such as autonomous vehicle teams, robot swarm control, and unmanned aerial vehicles. Human societies can achieve efficient collaboration because when a team tackles a complex problem, they communicate and assign different tasks to different individuals, thus achieving division of labor and collaboration—a foundation of general artificial intelligence. In fact, human society can be viewed as a large-scale multi-agent system where only partial information is observable, and many automated collaborative tasks improve the efficiency of handling these tasks. Therefore, the key to improving algorithm performance in large-scale multi-agent cooperative task systems is how to promote cooperation among agents by improving the communication efficiency between them.

[0003] In the real world, multi-agent systems typically contain a large number of agents of different types. These complex interactions pose significant challenges to policy learning, especially when agents have limited observations and face tasks requiring collaboration and synchronization. Communication components are a core element in learning coordinated behavior in multi-agent systems, attracting increasing attention. Communication channels between agents are considered to improve performance during the distributed execution phase of multi-agent tasks. Currently, all existing multi-agent communication algorithms directly treat the information exchanged between agents as part of the environment to predict actions at the current time step. This approach leads to extensive communication between agents, increasing their observation space and placing extremely high demands on the communication channels. Furthermore, agents struggle to discern important information from a large volume of messages, significantly increasing the learning difficulty.

[0004] Despite some achievements in multi-agent communication reinforcement learning algorithms, how agents can efficiently utilize the communication messages sent by other agents remains a significant challenge.

[0005] To address the aforementioned problems, this invention proposes a multi-agent hierarchical communication reinforcement learning algorithm. Unlike other multi-agent reinforcement learning communication methods, this invention decomposes the communication modules between agents into a hierarchical structure. This not only significantly reduces the communication frequency but also enables the model to learn more effective strategies for using messages passed between agents. Summary of the Invention

[0006] This invention proposes a multi-agent hierarchical communication algorithm based on deep learning. In this algorithm, agents first calculate the interaction relationships between themselves based on the currently observed environmental state and messages sent by other agents. Then, based on these interaction relationships, attention weights are assigned to other agents to select a top-level strategy. Choosing different top-level strategies implies a division of labor among agents. At the lower-level strategies, agents select appropriate actions based on the chosen high-level strategy data and local observations. By forming a hierarchical communication network structure, the learning process is simplified, and the difficulty of collaborative learning is reduced.

[0007] The algorithm employs end-to-end training via backpropagation of a neural network. The model of this invention closely resembles how humans solve complex problems, selecting a top-level policy through communication rather than directly choosing the action at the current time step. The model learns the interactions between agents through hierarchical communication in a partially observable distributed environment, thereby effectively utilizing messages between agents. The hierarchical structure of the model is more akin to how complex problems are solved in human societies. The top-level policy can be adjusted by communicating only once every specific number of time steps. This not only reduces the number of messages communicated between agents but also promotes collaboration among agents by not increasing the observation space used for predicting actions and improving the effectiveness of communication.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A deep learning-based multi-agent communication reinforcement learning algorithm includes the following steps:

[0010] Step (1) The model interacts with the environment to obtain the global state at each moment and the local observation information of each agent.

[0011] Step (2): Train the action semantic representation encoding model and use the action representation encoding to decompose the action space of the agent.

[0012] Step (3): The agent selects the top-level strategy. The agent selects roles by extracting the relationships between agents through communication, thereby achieving the purpose of division of labor and cooperation.

[0013] Step (4): The agent selects an appropriate action in the sub-action space according to the top-level policy.

[0014] Step (5): Integrate all the above components into a collaborative learning framework and optimize the loss function using an end-to-end training method.

[0015] Furthermore, step (2) includes the following steps:

[0016] (2.1) The training parameter is θ emotion encoder f e (·;θ e The encoder's input is the discrete action code a taken by agent i. i The output is a continuous d-dimensional action representation.

[0017] (2.2) Given the local observation o at the current time step of agent i i The action chosen by all other agents at the current time step. Simultaneously, it is passed as input to the observer predictor p. o and reward predictor p r This is to predict the rewards of local observations and the environment at the next time step, respectively.

[0018] (2.3) θ e Parameterized motion encoder f e , and by ξ e Parameterized observation predictor p o and reward predictor p r Simultaneously, updates are performed, and end-to-end training is conducted by minimizing the following loss function:

[0019]

[0020] Where D is the experience replay buffer. It is the prediction of agent i for the next local partial observation. It is the predicted reward received, o i ' is the agent's actual local observation at the next time step, and r is the actual environmental reward received. Scaling factor λ e It balances the error in predicting the next local observation with the error in predicting the reward received.

[0021] (2.4) Training f e It is used to encode all actions into their corresponding action representations. After training, the action encoder f e The network weight parameters are fixed and not updated during the subsequent Q-value training process.

[0022] (2.5) All executable actions of the agent are input into the action encoder to obtain a distinct action representation. The entire action space of the agent is divided into k groups by measuring the Euclidean distance of the action representations using k-means clustering. Different groups have different sub-action spaces, and k is a hyperparameter. After action grouping, training begins. The agent selects a high-level policy based on local observations and information from other agents, i.e., selecting one of the k sub-action spaces. During training, the action representations of each sub-action space remain fixed.

[0023] Furthermore, step (3) includes the following steps:

[0024] (3.1) Construct a top-level policy observation encoder that includes a linear layer and a memory unit network. Input the agent's local observation O and the hidden state h in the recurrent network memory unit into the recurrent network unit simultaneously and encode and output them as a vector hτ.

[0025] (3.2) Set a multi-round communicator parameterized by χ in the top-level policy network. The number of communication rounds is regarded as a hyperparameter. Use a key-value-based self-attention mechanism in the communication structure. Each communicator consists of two components: a sender and a receiver.

[0026] (3.3) Each agent generates a message through the sender, receives and processes the messages of other agents through the receiver. The communicator of the agent sends the observation encoding hτ and the message as inputs to other agents, abstracts the relationships between agents through the attention module, and finally integrates the messages of other agents into the vector h τ ′. Each message m i consists of two parts: and is defined as:

[0027]

[0028] (3.4) At the receiving end, each agent first predicts the query vector of from the input to the communicator, and the query vector is defined as:

[0029] Agent i receives the messages sent by all other agents, uses the query q i and receives the key-value of all messages to calculate the dot product to abstract the interaction relationship between agents. The relationship between agent i and other agents is defined as the weight vector The calculation method is:

[0030]

[0031] where is the scaling factor, is the attention weight calculated between agent i and agent j, and after passing through the softmax function, it outputs a value between 0 and 1. The sum of the attention weights calculated between agent i and all other agents is 1. Then aggregate the values of each message through the following method:

[0032] For multi-round communication, use the updated hidden state to perform repeated communication.

[0033] (3.5) The top-level strategy outputs the Q-value of the role, and the role selector calculates it based on the average representation of the executable actions. The mean of the average representation of the executable actions, where A j This refers to the limited operational space of character j. The character selector is a function with θ as its core. β Parameterized fully connected network f β (h τ ,h τ ";θ β ), h τ "and h τ Mapped to Where h τ " is the final hidden state of the communicator, h" τ It involves observing the encoder's output and then calculating the selected character ρ using the following formula. j Expected returns:

[0034] (3.6) The agent selects the role with the highest Q value. To determine the top-level policy, when the top-level policy is assigned, the agent remains unchanged for the next C time steps, during which each agent can only choose the appropriate action in the sub-action space according to the top-level policy.

[0035] Furthermore, step (4) includes the following steps:

[0036] (4.1) Construct the policy network for each different role, and the policy network for each role... They are all made of Parameterized fully connected networks, h τ It is the output of the observation encoder, which will locally observe O i Encoded as a vector hτ.

[0037] (4.2) Through Function networks will h τ Mapping to z τ , z τ yes Vectors in the vector.

[0038] (4.3) Using actions to represent and z τ To calculate the original action value a k The calculation formula is as follows:

[0039]

[0040] Furthermore, step (5) includes the following steps:

[0041] (5.1) Using each agent To estimate the global Q value

[0042] (5.2) The Q value Input by φ β A parameterized hybrid network is used to train the role selector using global rewards.

[0043] (5.3) Minimize the following TD losses to update the high-level strategy:

[0044]

[0045] in It is the target network, ρ=<ρ1,ρ2,···,ρ n > is the joint role of all agents, and the expected value Q is estimated using uniform samples from the replay buffer D. i The local Q value is input again to φ ρ Estimating global action value Q in parameterized hybrid networks tot (s,a). Minimize the following TD loss to learn the underlying policy:

[0046] in It is a target network, θ ρ These are the parameters for all character strategies.

[0047] (5.4) The total loss L = L is made up by backpropagation. ρ +L β minimize.

[0048] This invention, based on a hierarchical deep reinforcement learning model, can achieve excellent collaboration in large-scale multi-agent systems. It closely resembles how human groups solve complex problems, selecting a top-level strategy through communication rather than directly deciding actions. The model not only has a low communication frequency but also good scalability. Attached Figure Description

[0049] Figure 1 Overall model framework flowchart;

[0050] Figure 2 Feedforward model diagram of action encoding representation;

[0051] Figure 3 Local observation encoder structure diagram;

[0052] Figure 4 Overall structural model diagram of the communication module;

[0053] Figure 5 Information sending module structure diagram;

[0054] Figure 6 Information reception model structure diagram;

[0055] Figure 7 High-level strategy model structure diagram;

[0056] Figure 8 Underlying strategy model structure diagram. Detailed Implementation

[0057] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and technical solutions.

[0058] The present invention includes the following steps:

[0059] (1) At each time step, data is collected by interacting with the environment and stored in the experience replay buffer.

[0060] (2) Action representation encoding model training process: Construct encoder and decoder models. The encoder model includes the current agent's action input process, the actions selected by other agents and the current state input process, as well as the encoding output process. The decoder model needs to concatenate action features and state features, and includes the decoding process and the decoding output process. Initialize the parameters of the encoder and decoder models.

[0061] (3) Sub-action space division: After obtaining the action representations, the agent's complete action space is divided into 5 groups using k-means clustering by measuring the Euclidean distance between the action representations. Different groups have different sub-action spaces. Actions belonging to the same group have small Euclidean distances between their action representations, meaning they have similar effects on the environment and other agents. For example, movement actions and attack actions are usually divided into different groups. After action grouping, training begins. During training, the action representations and action spaces of each group remain fixed.

[0062] (4) Model training process: The model of this invention includes a top-level observation encoder and a bottom-level observation encoder, which have the same network structure and are respectively composed of δ β and δ ρ Parameterization. Using a 64-dimensional linear layer and a GRU layer, the observation O and the GRU hidden state h are encoded together into a vector hτ. Every 5 time steps, hτ is input to the communication module of each agent, and this communication is repeated twice to obtain the desired result. After selecting the maximum Q-value for a role, the agent executes actions belonging to that role over the next 5 time steps. The encoder will then convert the local observations O... i Encode it as a vector hτ, then h τ Mapping to z τ , z τ yes Vectors in the vector representation. Represented using actions. and z τ To calculate the original action value The exploration rate is set to 0.05. The model has a 95% probability of selecting the action with the largest Q value and a 5% probability of randomly selecting an executable action to interact with the environment. Based on the reward R returned by the environment, the loss of the top-level policy network model and the loss of the bottom-level policy network model are calculated respectively. After obtaining the total loss, backpropagation is performed to update the model.

[0063] (5) Model prediction process: After the loss functions of the two models converge to the minimum or the number of iterations reaches the preset number, the observation encoding model, the top-level policy network model, and the bottom-level policy network model are saved. Every 5 time steps, the agent's local observations are input into the model to perform top-level policy prediction. Based on the selected high-level policy, the bottom-level policy action prediction is executed, and finally, the action executed by each agent at this time step and its interaction with the environment are obtained.

Claims

1. A deep learning-based multi-agent communication reinforcement learning algorithm, characterized in that... Includes the following steps: Step (1) The model interacts with the environment to obtain the global state at each moment and the local observation information of each agent; Step (2): Train the action semantic representation encoding model and use the action representation encoding to decompose the agent's action space; Step (3): The agent selects the top-level strategy. The agent selects roles by extracting the relationships between agents through communication, thereby achieving the purpose of division of labor and cooperation. Step (4): The agent selects an appropriate action in the sub-action space according to the top-level policy; Step (5): Integrate all the above components into a collaborative learning framework and optimize the loss function using an end-to-end training method; Step (2) includes the following steps: (2.1) The training parameters are motion encoder The encoder's input is the intelligent agent. Discrete action coding adopted The output is a continuous d-dimensional action representation. ; (2.2) Given an agent Current time step local observation The action chosen by all other agents at the current time step. It is also passed as input to the observation predictor. and reward predictor To predict the rewards of local observations and the environment at the next time step, respectively; (2.3) will Parametric motion encoder , and by Parameterized observation predictor and reward predictor Simultaneously, updates are performed, and end-to-end training is conducted by minimizing the following loss function: ； in It is an experience replay buffer. It is an intelligent agent The prediction of the next local observation, It is the predicted reward received. It is the agent's actual local observation of the next time step. It is the actual environmental reward received; scaling factor It balances the error in predicting the next local observation with the error in predicting the reward received; (2.4) Training The action encoder is used to encode all actions into their corresponding action representations. After training, the action encoder... The network weight parameters are fixed and not updated during the subsequent Q-value training process; (2.5) Input all executable actions of the agent into the action encoder to obtain each different action representation. Use the k-means clustering method to divide the entire action space of the agent by measuring the Euclidean distance of the action representations. Groups, different groups have different sub-action spaces, numbers It is a hyperparameter; after the actions are grouped, training begins; the agent selects a high-level policy based on local observations and messages from other agents, that is, selects one of the k groups of sub-action spaces; during training, the action representation of each group of sub-action spaces remains fixed; Step (3) includes the following steps: (3.1) Construct a top-level policy observation encoder containing one linear layer and one memory unit network to process the agent's local observations. Hidden states in circular network memory units Simultaneously, the input into the recurrent network unit is encoded as a vector. ; (3.2) Set up the top-level policy network by A parameterized multi-round communicator; the number of communication rounds is treated as a hyperparameter; a key-based self-attention mechanism is used in the communication structure, and each communicator consists of two components: a transmitter and a receiver; (3.3) Each agent generates messages through a transmitter and receives and processes messages from other agents through a receiver; the agent's communicator encodes observations. The messages are sent as input to other agents, and the relationships between agents are abstracted through an attention module. Finally, the messages from other agents are integrated into a vector. In Chinese; each message It consists of two parts: and Defined as: ； ； (3.4) At the receiving end, each agent first receives input from the communicator. China's prediction The query vector, which is defined as: ; intelligent agent Receive messages sent by all other intelligent agents and use queries. It receives the key-value pairs of all messages to compute the dot product, thus abstracting the interaction relationships between agents; agents The relationship between other agents is defined as a weight vector. The calculation method is as follows: ； in It is a scaling factor. For intelligent agents With intelligent agents The calculated attention weights, after being processed by the softmax function, output values ​​between 0 and 1, which are then used by the agent. The attention weights calculated with all other agents sum to 1; then the values ​​of each message are aggregated as follows: For multi-round communication, the updated hidden state is used again. Repeated communication is performed; (3.5) Top-level strategy output role The value, calculated by the role selector based on the average representation of the executable actions. The mean of the average representation of the executable actions, where It is the limited operation space of character j; the character selector is a... Parameterized fully connected networks ,Will and Mapped to ,in This is the final hidden state of the communicator. It involves observing the encoder's output and then calculating the selected character using the following formula. Expected returns: ; (3.6) The agent selects the role with the greatest potential. value To determine the top-level policy, when assigning the top-level policy, the agent will proceed in the following... This remains unchanged throughout the time steps, during which each agent can only choose the appropriate action in the sub-action space according to the top-level policy; Step (4) includes the following steps: (4.1) Construct the policy network for each different role, and the policy network for each role. They are all made of A parameterized, fully connected network is the output of an observation encoder that takes local observations. Encoded as a vector ; (4.2) Through Function networks will Mapped to , yes Vectors in; (4.3) Using actions to represent and To calculate the original action value The calculation formula is as follows: ； Step (5) includes the following steps: (5.1) Using each agent To estimate the global value ; (5.2) will value Input by A parameterized hybrid network is used to train the character selector using global rewards; (5.3) Minimize the following TD losses to update the high-level strategy: ； in, It is the target network. It is a joint role of all agents and uses data from the replay buffer. Learning to estimate expected values ​​using a unified sample , again local Value input to by Estimating global action values ​​in parameterized hybrid networks To minimize the TD loss in the following learning strategy: ; in It is a target network. These are the parameters for all character strategies; (5.4) The total loss is reduced through backpropagation. minimize.

Images