Multi-agent reinforcement learning cooperation method based on behavior body behavior characteristic similarity

Abstract

The invention discloses a multi-agent reinforcement learning cooperation method based on behavior feature similarity of behavior bodies, and in the model, interaction among agents is established as a graph neural network. Specifically, a Pearson's correlation coefficient is provided to calculate the similarity of historical trajectories of a subject, and the similarity is used for modeling weights of edges in a graph neural network and serves as a means for determining common cognition of behaviors of the subject; furthermore, a structured state information complementation module is utilized that mainly relies on a self-attention-based transformer-encoder architecture, fusing the observations and actions of other agents to enhance the policy network representation capability of the agents. According to the model, the collaborative behavior between subjects is effectively enhanced, and the training efficiency of the collaborative model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning. In reinforcement learning, similar agents have certain common cognition of behavior, which makes it easier to understand the intentions of both parties and thus form a cooperative policy. The focus of the present invention is to enrich the representation ability of collaborative strategies with common cognition through modeling interaction graphs within intelligent agents. Background Art

[0002] Collaborative behaviors in human social activities can be modeled through multi-agent reinforcement learning and used to train collaborative strategies of agents to achieve effective cooperation. Generally speaking, agents with similar behaviors have certain common cognitions of behaviors and are more likely to understand the intentions of both parties, thus forming cooperative policies. Traditional methods focus on the collaborative allocation process between agents, either oversimplifying the connection relationship between all agents or ignoring factors such as the similarity of decision information or common cognition, resulting in high overall computational costs; ignoring the influence of similar behaviors and common cognitive features in collaborative interactions. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-agent reinforcement learning collaboration method based on the similarity of the behavioral characteristics of the agents. The performance of the present invention in calculating edge weights using trajectory information exceeds that of the commonly used attention mechanism. By utilizing the RNN network, the trajectory information of the agent encapsulates the observation and action characteristics from historical decisions, enabling the agent to formulate a more appropriate cooperation policy.

[0004] The objective of the present invention is achieved through the following technical solution: a multi-agent reinforcement learning collaboration method based on the similarity of behavior characteristics of the agents, comprising the following steps:

[0005] (1) Establish an interaction graph of agents in the collaborative task: abstract the agents in the environment as nodes in a graph network, and represent the observation process between agents as edges between nodes; the interaction relationship between agents is represented as an undirected graph G, G = (N, V); where N is a node, representing the entity of each agent; V is an edge, representing the bidirectional connection between agents;

[0006] (2) Establishing an intelligent agent model: including an intelligent agent decision model, a state information supplement module, and a Pearson correlation information fusion module;

[0007] The agent decision model includes the first MLP, GRU module, second MLP and ∈-greedy mechanism connected in sequence; the observation value o of the agent t and the action u taken in the previous step t-1 As the input of the first MLP, it is encoded and input into the GRU module, which combines the encoded data with the hidden state h of the previous step. t-1Get the hidden state h of the next step t And the state-action value function, and the state-action value function is input into the second MLP for processing to obtain the state-action value function Q of the intelligent agent i (τ i ,·); select the state-action value function Q through the ∈-greedy mechanism i (τ i ,·) The largest action maxQ i As the decision action of the intelligent agent, the state information supplement module is input to supplement the information;

[0008] The state information supplement module includes an MLP and a feature extraction superposition module Block connected in sequence, and the feature extraction superposition module includes a self-attention module, an MLP and an Encoder module; the global state information s obtained by each agent is used as input, and a hierarchical normalization operation is performed after encoding through an MLP;

[0009] Then the feature is input into the feature extraction and superposition module, and the encoding feature of the global state information s is assigned to the three elements of the self-attention network in the Block, namely Q, K, and V; the output of the self-attention network is concatenated with the input three-element features, and the concatenated feature vector is input into the second MLP for secondary encoding; the output and output of the second MLP are concatenated and input into the Encoder module, and the result is hierarchically normalized to obtain the weight relationship feature coefficient Score i ; Score i maxQ output by the agent decision model i Multiply them together to get the complementary utility value

[0010] The Pearson correlation information fusion module includes the following operations:

[0011] ① Perform Pearson calculation on the historical interaction trajectory of the agent, and then perform dot product operation with the adjacency matrix A of the agent's current decision time step to obtain the Pearson correlation coefficient between the interaction trajectory and the adjacency matrix; the specific calculation formula of the Pearson coefficient matrix E is:

[0012]

[0013] Where I is the unit matrix with dimension equal to the number of agents; τ i represents the historical interaction trajectory of the ith agent, Represents the mean of the historical interaction trajectories of all agents;

[0014] ② Combine the adjacency matrix A of the agent's current decision time step and the supplemented utility value The global state information s obtained by the agent is spliced ​​together as the input of the MLP, and the enhanced utility function is obtained through the linear mapping of the MLP

[0015]

[0016] And the enhanced utility function of each agent Integrate to form a total utility function set

[0017] ③ Input the global state s into a hypernetwork to generate The parameter vector W with the same dimension is then replaced by the absolute value of the i-th item in the parameter vector W |w i |As the utility value after complementation The weighted coefficient of the Pearson coefficient matrix E is the (i-1)th column vector as the supplemented utility value. The weight coefficient is weighted to obtain Q tot ; Its calculation expression is as follows:

[0018]

[0019] Among them, e i-1 represents the (i-1)th column vector of E; then, Q tot The update is performed in a manner similar to the DQN algorithm, and its objective function is expressed as:

[0020]

[0021] Where r is the reward function of environmental feedback, A′ represents the complemented utility value of the next time step, the global state information, and the adjacency matrix of the agent.

[0022] The calculation method of the self-attention network is:

[0023]

[0024] Among them, Q, K, and V are the three components of the self-attention mechanism; W Q and W K are the network parameters when neural network encoding is used for Q and K elements, Refers to the decision network output of n agents in the environment, that is, maxQ i ;d k Refers to (W Q ·s)(W K ·s) T The dimension of the vector formed.

[0025] The beneficial effects of the present invention are: (1) The performance of calculating edge weights using trajectory information exceeds the commonly used attention mechanism, indicating the importance of integrating trajectory information in interactive graphs. By utilizing the RNN network, the trajectory information of the agent encapsulates the observation and action features from historical decisions, enabling the agent to formulate more appropriate cooperation policies. (2) A state information supplementation module based on a converter-encoder enhances the policy network representation capability of the cooperative agent; (3) A q-value enhancement method for a multi-agent collaborative algorithm is adopted, which uses the Pearson correlation coefficient calculated from the historical trajectory to determine the information fusion strength between the interacting agents, thereby enhancing the representation capability of the agent's policy network. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the QEN model structure of the present invention;

[0027] Figure 2 Schematic diagram of the agent's observation range in the game scene;

[0028] Figure 3 A schematic diagram of the specific process of the status information supplement module;

[0029] Figure 4 The influence of the edge weights of the interaction graph in QEN. The shaded area indicates the 25%-75% quartile results.

[0030] Figure 5 The shaded area represents the 25%-75% quartile results for the status information module in QEN;

[0031] Figure 6 The shaded area represents the 25%-75% quartile results to reflect the effect of Pearson's coefficient. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0033] In order to better establish the cooperative relationship between agents, this paper proposes a multi-agent reinforcement learning cooperative algorithm QEN based on the similarity of the behavioral characteristics of the agents to update the utility function of the agents. The interaction between the agents is modeled as a graph, and the historical trajectories of the agents are used to calculate the Pearson correlation between the interacting agents in the model graph as the weight of their connecting edges. This method overcomes the bottleneck of the traditional GNN-based MARL method, focuses on the observation of the agent, and can improve the sample efficiency. In this model, the interaction between the agents is established as a graph neural network. Specifically, the Pearson correlation coefficient is proposed to calculate the similarity of the historical trajectory of the subject as a means to determine the common cognition of their behavior, which is used to establish the weight of the edge in the modeling graph neural network. In addition, a transformer-encoder structured state information complementary module is designed to enhance the decision representation of the agent.

[0034] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0035] The multi-agent reinforcement learning collaboration method based on the similarity of the behavior characteristics of the agents of the present invention comprises the following steps:

[0036] (1) Establish the interaction graph of the agents in the collaborative task: Use the observation information of the agents to construct the interaction relationship of the agents. When other agents appear within the observation range of the current agent, the current agent can observe and obtain these feature information and establish an implicit information interaction mechanism; this process is abstracted as the relationship between nodes and edges in the graph network, where the adjacency matrix represents the connection relationship between each point. The agents in the environment are abstracted as nodes in the graph network, and the observation process between agents is represented as the edges between nodes. Therefore, the interaction relationship of the agents is represented as an undirected graph G, G = (N, V); where N represents the node, representing the entity of each agent; V represents the edge, representing the bidirectional connection between agents.

[0037] The graph G describes the network topology of the mutual association or dependency between agents, where each node corresponds to an agent, the edges between nodes represent some form of association or dependency between these agents, and the connection relationship between the edges of the graph transmits the observation and action information between the agents.

[0038] After obtaining the agent's interaction graph, the model QEN will calculate the Pearson coefficient of each agent's observation-action history to determine the degree of common cognition of the behavior, and use it as the weight of the edge in the graph for information fusion. And use the transformer-encoder based state information supplement module to supplement the agent utility to better integrate environmental information and make appropriate actions.

[0039] (2) Establishing an agent model: In order to better establish a collaborative relationship between agents, the process of "humans with similar past experiences are more likely to understand and cooperate with each other" is abstracted and applied to the training of collaborative models. This paper proposes a practical Q-value enhanced collaborative model, called the QEN model, which can effectively use the information between different agents to complement each other to improve the representative ability of policies in the coordination process, so as to improve the training efficiency. The QEN model includes an agent decision model, a state information supplement module and a Pearson correlation information fusion module, such as Figure 1 shown.

[0040] The first part on the left side of the figure is the agent decision model, such as Figure 1 (a); the second part is the state information complementation module (State Information Complementation Module), as shown in Figure 1 (b); the rightmost is the Pearson correlation information fusion module, as shown in Figure 1 As shown in (c) and (d), (d) is the detailed structure diagram of the Pearson Correlation Module in (c); and finally outputs a comprehensive Qtot value. All modules are modeled using neural networks. Finally, the QEN model will use the feedback r obtained by interacting with the environment to train Qtot, and use back propagation to update the parameters of the neural network to train the entire model.

[0041] The agent decision model mainly includes the agent's strategy module, which emphasizes how the agent makes action decisions. The agent decision model mainly adopts the GRU module based on the recurrent neural network, including the first MLP, GRU module, second MLP and ∈-greedy mechanism connected in sequence; t and the action u taken in the previous step t-1 As the input of the first MLP, it is encoded and input into the GRU module, which combines the encoded data with the hidden state h of the previous step. t-1 Get the hidden state h of the next step t And the state-action value function, and the state-action value function is input into the second MLP for processing to obtain the state-action value function Q of the intelligent agent i (τ i ,·); select the state-action value function Q through the ∈-greedy mechanism i (τ i ,·) The largest action maxQ i As the decision action of the intelligent agent, the state information supplement module is input to supplement the information.

[0042] The agent in the present invention refers to a single decision-making individual modeled by the reinforcement learning method in the game scene, and the observation information of the agent is specifically set by the game environment. Taking a single attacking unit in the StarCraft II game as an example, the agent is set to have a certain observation range and attack range, such as Figure 2 As shown in the figure, with the agent as the center, the solid circle area is the observation range of the current agent, and the dotted line is the range of its attack action. Then the observation information of the attacking unit is all the feature information that can be obtained within its observation range, including the following attributes of friendly and enemy units within the field of view: distance, relative horizontal distance x, relative vertical distance y, health value, shield value. In addition, the agent can also view the last action made by the friendly units in the field of view in the previous time step, and can obtain the terrain feature information around them. The above feature information constitutes the global state information s.

[0043] The actions that the agent can take are contained in the output of the policy network. In this scenario, the discrete set of actions that the agent is allowed to perform includes moving [direction] (four directions: north, south, east or west), attacking [enemy_ID], stopping and no action. When the agent fails (dies or its actions are not considered), only the no action can be taken, and the real-time agent that is still interacting in the environment cannot perform this operation.

[0044] The second part is the State Information Complementation Module, which mainly relies on the encoder architecture in the Transformer based on the self-attention mechanism. This architecture integrates the observation and action information of other agents to enhance the representation ability of the decision model of a single agent and encourages the agent to pay more attention to the information in the global state to obtain better behavior. This module uses the encoder architecture to capture the state s information and the output Q of the agent decision model. i The potential connection of each agent is calculated by using the attention mechanism to calculate the complementary coefficient Score of the state-action value function Q of each agent. i , and then use the supplement coefficient Score i The state-action value function Q of each agent i Complement it and get the output after complementation: The state information supplement module includes a sequentially connected MLP and feature extraction superposition module (Block), which includes a self-attention module (Self-Attention), MLP and Encoder module. Feature extraction can also use multiple Block superposition and rollback to Blocks to provide necessary features to supplement the utility value of the agent. The specific number of blocks can be selected by the user.

[0045] The global state information s obtained by each agent is used as input, encoded by a feedforward neural network MLP, and the encoded features are layer-normalized (Layer-Norm). Then the encoded features are input into the feature extraction superposition module (Block). This structure consists of a self-attention network with a residual network structure, such as Figure 3 Specifically, the encoding features of the global state information s are assigned to the three elements of the self-attention network in the Block, namely Q, K, and V;

[0046] Then the output of the self-attention network is concatenated with the three input features, and the concatenated feature vector is input into the second MLP for secondary encoding. The output of the second MLP is concatenated with the output and input into the Encoder module, and the result is layer-normalized (Layer-Norm) to obtain the weight relationship feature coefficient Score i This Score i To make up for Q i value, thereby providing necessary information beyond the observation range of the agent. The specific method of supplementation is: Score and maxQ output by the agent decision model i Multiply them together to get the complementary utility value

[0047] The calculation method of the self-attention mechanism is:

[0048]

[0049] Among them, Q, K, and V are the three components of the self-attention mechanism; W Q and W K are the network parameters when neural network encoding is used for Q and K elements, Refers to the decision network output of n agents in the environment, that is, maxQ i ;d k Refers to (W Q ·s)(W K ·s) T The dimension of the vector formed.

[0050] The third part is the Pearson Correlation Module, which is used to calculate the correlation of historical interaction trajectories between different agents, representing the connection between the past observations and behaviors of the agents. Its specific structure is as follows: Figure 1In other words, agents that have more similar observations and actions in the past will have closer relationships, may have some kind of common cognition, and will be more influenced by each other in some ways.

[0051] This module includes the following operations:

[0052] ① Perform Pearson calculation on the historical interaction trajectories of the closely interacting agents connected in the interaction graph, and then perform dot product operation with the adjacency matrix A of the agent's current decision time step to obtain the Pearson correlation coefficient between the interaction trajectory and the adjacency matrix. This coefficient includes the importance of the interaction between agents and determines the intensity of the interactive information transmission. The specific calculation formula of the Pearson Coef matrix E is:

[0053]

[0054] Where I is a unit matrix with a dimension equal to the number of agents. Since the correlation of the historical interaction trajectories of all agents is calculated, τ i represents the historical interaction trajectory of the ith agent, represents the mean of the historical interaction trajectories of all intelligent agents; the final Pearson coefficient matrix is ​​a coefficient square matrix related to the number of intelligent agents n.

[0055] ② Combine the adjacency matrix A of the current decision time step of the agent and the supplemented utility value obtained by the state information supplement module The global state information s obtained by the agent itself is spliced ​​together as the input of the MLP neural network, and the final enhanced utility function is obtained through the linear mapping of the MLP.

[0056]

[0057] And the final enhanced utility function of each agent Integrate to form a total utility function set

[0058] ③ Input the global state s into a hypernet to generate The parameter vector W with the same dimension is then replaced by the absolute value of the i-th item in the parameter vector W |w i |As the utility value after complementation The weighted coefficient of the Pearson coefficient matrix E is the (i-1)th column vector as the supplemented utility value. The weight coefficient is weighted to obtain Q tot , its calculation expression is as follows:

[0059]

[0060] Among them, e i-1 represents the (i-1)th column vector of E. Then, Q tot The update is performed in a manner similar to the DQN algorithm, and its objective function is expressed as:

[0061]

[0062] Where r is the reward function of environmental feedback, A′ represents the complemented utility value of the next time step, the global state information, and the adjacency matrix of the agent.

[0063] So far, the QEN model is able to enhance the Integrate to get Q tot , and use the weight matrix E to measure the contribution made by the agent inside and outside the field of view. The above is the entire QEN data transmission process, as well as the model update and parameter update process.

[0064] Get the total After that, the interaction between the agent and the environment will feedback the global reward, which is updated by guiding

[0065]

[0066] is the target value; u is the joint action of all agents, θ and θ′ are the parameters of the behavior network and the target network respectively; γ is the discount factor in reinforcement learning, which represents the influence of the rewards brought by the actions that the agents may take in the future on the current decision; Refers to the Q of the agent's future action (the next time step to take action) tit Valuation, where s′ represents the global state information of the next time step, A′ represents the adjacency matrix between agents of the next time step, They represent the enhanced state-action estimates of each agent at the next time step.

[0067] It is a centralized value function generated by the fusion of the enhanced state action value functions of each agent;

[0068] is the objective function of the centralized value function, which is determined by the formula of the previous problem; refers to When updating the network function θ, the algorithm uses the experience pool The sampled data s,u,A,θ are input into the model to obtain the Q of each agent i operation.

[0069] The update follows the principle of the DQN algorithm, and the target value is updated through the environment's rewards. The process is as follows:

[0070] 1. Calculate the target value

[0071] 2. Update the loss function

[0072] 3. When the set update interval is reached, update the parameters of the target network;

[0073] 4. During training, if the size of the experience replay buffer D exceeds the batch size, a batch of episodes is randomly selected from D for training. For each episode in the batch, Then according to the target value renew

[0074] 5. Update Critics That is, through To update, then calculate And update the parameter θ = θ-α△θ. The whole updating process ensures It can reflect the team rewards of the intelligent agent during the interaction process and continuously adjust by comparing with the target network to improve the training efficiency and decision-making quality of the model.

[0075] QEN detailed algorithm steps:

[0076] Initialization network θ and target network θ - ←θ, learning rate a, buffer step=0;

[0077]

[0078]

[0079] Figure 4 The influence of the edge weights of the interaction graph in QEN. The shaded area indicates the 25%-75% quartile results. Figure 5 The shaded area represents the 25%-75% quartile results for the status information module in QEN; Figure 5 The shaded area represents the 25%-75% quartile results to reflect the effect of Pearson's coefficient. Figure 4In the paper, QEN outperforms the commonly used attention mechanism in calculating edge weights using trajectory information, indicating the importance of integrating trajectory information in interactive graphs. By utilizing the RNN network, the agent's trajectory information encapsulates the observation and action features from historical decisions, enabling the agent to formulate more appropriate cooperation policies. Figure 5 It shows that during the training process, the supplementation of state information not only makes the training of the agent's cooperative policy faster, but also makes the final effect of SMAC adversarial scenario better. This is attributed to the state information module that supplements the observation characteristics of the agent, indicating the importance of the QEN state information module. Figure 6 It can be seen that when the trajectory information of the agents is simply and directly integrated, the cooperation strategy cannot be improved, which proves the importance of the Pearson coefficient in the weight calculation process of the interaction graph edge. The calculation process of the Pearson coefficient reflects the correlation of the trajectory information of different agents, implicitly influencing the past decision-making characteristics and common characteristics, and the cognitive characteristics of the current collaborative interaction.

[0080] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

[0081] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multi-agent reinforcement learning collaboration method based on the similarity of behavioral characteristics of the agents, characterized in that: The following steps are involved: (1) Establish an interaction graph of agents in the collaborative task: abstract the agents in the environment as nodes in a graph network, and represent the observation process between agents as edges between nodes; the interaction relationship between agents is represented as an undirected graph G, G = (N, V); where N is a node, representing the entity of each agent; V is an edge, representing the bidirectional connection between agents; (2) Establishing an intelligent agent model: including an intelligent agent decision model, a state information supplement module, and a Pearson correlation information fusion module; The agent decision model includes the first MLP, GRU module, second MLP and ∈-greedy mechanism connected in sequence; the observation value o of the agent t and the action u taken in the previous step t-1 As the input of the first MLP, it is encoded and input into the GRU module, which combines the encoded data with the hidden state h of the previous step. t -1 gets the hidden state h for the next step t And the state-action value function, and the state-action value function is input into the second MLP for processing to obtain the state-action value function Q of the intelligent agent i (τ i , ·); select the state-action value function Q through the ∈-greedy mechanism i (τ i , ·) The largest action maxQ i As the decision action of the intelligent agent, the state information supplement module is input to supplement the information; The state information supplement module includes an MLP and a feature extraction superposition module Block connected in sequence, and the feature extraction superposition module includes a self-attention module, an MLP and an Encoder module; the global state information s obtained by each agent is used as input, and a hierarchical normalization operation is performed after encoding through an MLP; Then the feature is input into the feature extraction superposition module, and the encoding feature of the global state information s is assigned to the three elements of the self-attention network in the Block, namely Q, K, and V; the output of the self-attention network is spliced ​​with the input three-element features, and the spliced ​​feature vector is input into the second MLP for secondary encoding; the output and output of the second MLP are spliced ​​and input into the Encoder module, and the result is hierarchically normalized to obtain the weight relationship feature coefficient Score i ; Score i maxQ output by the agent decision model i Multiply them together to get the complementary utility value The Pearson correlation information fusion module includes the following operations: ① Perform Pearson calculation on the historical interaction trajectory of the agent, and then perform dot product operation with the adjacency matrix A of the agent's current decision time step to obtain the Pearson correlation coefficient between the interaction trajectory and the adjacency matrix; the specific calculation formula of the Pearson coefficient matrix E is: Where I is the unit matrix with dimension equal to the number of agents; τ i represents the historical interaction trajectory of the ith agent, Represents the mean of the historical interaction trajectories of all agents; ② Combine the adjacency matrix A of the agent's current decision time step and the supplemented utility value The global state information s obtained by the agent is spliced ​​together as the input of the MLP, and the enhanced utility function is obtained through the linear mapping of the MLP And the enhanced utility function of each agent Integrate to form a total utility function set ③ Input the global state s into a hypernetwork to generate The parameter vector W with the same dimension is then replaced by the absolute value of the i-th item in the parameter vector w |w i |As the utility value after complementation The weighted coefficient of the Pearson coefficient matrix E is the (i-1)th column vector as the supplemented utility value. The weight coefficient is weighted to obtain Q tot ; Its calculation expression is as follows: Among them, e i-1 represents the (i-1)th column vector of E; then, Q tot The update is performed in a manner similar to the DQN algorithm, and its objective function is expressed as: Where r is the reward function of environmental feedback, s′; A′ represent the supplemented utility value of the next time step, the global state information and the adjacency matrix of the agent respectively.

2. The multi-agent reinforcement learning collaboration method based on similarity of behavioral characteristics of actors according to claim 1 is characterized in that: The calculation method of the self-attention network is: Among them, Q, K, and V are the three components of the self-attention mechanism; W Q and W K are the network parameters when Q and K elements are encoded using neural network, Refers to the decision network output of n agents in the environment, that is, maxQ i ;d k Refers to (W Q ·s)(w K ·s) T The dimension of the vector formed.

Images