Multi-agent cooperative learning method based on deep reinforcement learning

Abstract

The invention relates to multi-agent control, and provides a multi-agent cooperative learning method based on deep reinforcement learning, and the method comprises the steps: taking a Q value containing teammate information as a first Q value, taking a Q value not containing the teammate information as a second Q value, and quantifying the actual contribution of the teammate information to a current decision through the difference between the two Q values; q value decoupling is introduced, and a Q value is decomposed into a state value and an action advantage; the difference between the two Q values is divided into a state correction value used for evaluating the global cognition improvement amplitude of cooperation and an action correction value used for measuring the specific decision correction intensity of cooperation; on the basis, a mechanism for dynamically adjusting a Q value through a state correction value and an action correction value is designed, so that the method can actively explore actions beneficial to a team through teammate information instead of passively waiting for global reward updating, better adapts to environmental changes and is suitable for multi-agent cooperation tasks such as games and unmanned aerial vehicle formation.

Description

Technical Field

[0001] The present invention relates to the control of multiple intelligent agents, and in particular to a multi-agent cooperative learning method based on deep reinforcement learning. Background Art

[0002] Multi-agent cooperative learning is an important research direction in multi-agent systems. It aims to accomplish complex tasks through collaboration between agents. Its core approach is to enable multiple agents to learn how to collaborate and achieve a common goal through interaction with the environment through a reinforcement learning framework. Compared to single-agent learning, multi-agent cooperative learning faces the following challenges:

[0003] 1. Partial observability: Each agent can usually only obtain local environmental information, making it difficult to make globally optimal decisions; 2. Non-steady-state environment: When multiple agents learn simultaneously, the strategy changes of each agent will change the environmental dynamics of other agents, causing the Markov assumption of traditional reinforcement learning to fail, which makes strategy convergence complicated; 3. Dimensionality explosion: As the number of agents increases, the joint action space grows exponentially; 4. Credit allocation: The global reward signal cannot directly reflect the contribution of a single agent, and traditional algorithms find it difficult to trace the strategy value of a specific individual; 5. Communication efficiency and redundancy: In partially observable environments, agents need to exchange local observation information through communication to support collaboration.

[0004] To solve the above problems, a lot of research has been done and many solutions have been proposed.

[0005] Among them, the value decomposition-based methods include: VDN: By decomposing the global value function into the sum of the local value functions of each agent, multi-agent collaboration is achieved; QMIX: Further expands the idea of VDN, by introducing a hybrid network to more flexibly combine the value functions of each agent, so as to better handle complex collaborative tasks; QPLEX: Based on QMIX, it further optimizes the decomposition and combination of value functions, thereby improving collaboration efficiency. Although the above-mentioned value decomposition-based methods can effectively deal with the problem of dimensionality explosion, they still have certain difficulties in allocating credit to global rewards, especially in non-stationary environments, which are prone to local optimal problems. Secondly, traditional value decomposition methods rely on linear addition or monotonicity assumptions to decompose global Q values, which may not be able to capture complex interactive relationships and make it difficult to characterize complex collaborative relationships between agents. This is because, in some tasks, the relationship between individual action value functions may be non-monotonic, resulting in limited policy adaptability in dynamic environments. Third, the traditional value decomposition method does not distinguish the local contributions of individual behaviors and only updates the strategy through global reward backpropagation, which is prone to policy optimization deviation and slow algorithm convergence or even oscillation. At the same time, when using a fixed value decomposition function to handle multi-agent interactions, as the number of agents increases, the scale of network parameters grows exponentially, making it difficult to support large-scale training and exacerbating the difficulty of exploration.

[0006] Communication-based methods include: DIAL: By designing a communication mechanism, it enables agents to exchange information, thereby solving some observability problems; TarMAC: By optimizing the content and timing of communication, the efficiency of collaboration between agents is further improved. However, for communication-based methods, communication rules need to be designed in advance. If the environment changes, the communication rules may no longer apply. Secondly, if the types of agents are different, such as some responsible for reconnaissance and some responsible for attack, the fixed communication method may fail because different tasks require different information. Thirdly, the efficiency of information transmission may also be a problem. For example, TarMAC uses a fixed attention mechanism, which cannot dynamically select important information according to the situation. It contains a lot of useless information, resulting in long communication time and affecting the effectiveness of team collaboration. Summary of the Invention

[0007] The technical problem to be solved by this invention is to propose a multi-agent cooperative learning method based on deep reinforcement learning, which designs a new mechanism for dynamically adjusting Q values. It actively explores actions that are beneficial to the team through teammate information, rather than passively waiting for global reward updates, and better adapts to environmental changes.

[0008] The technical solution adopted by the present invention to solve the above technical problems is:

[0009] A multi-agent cooperative learning method based on deep reinforcement learning. The multi-agents are divided into two opposing parties. One of the parties is selected as the self-agent. For each of the self-agent agents, an independent deep neural network is configured. The deep neural network includes an encoding network and a policy network. The deep reinforcement learning algorithm is used for training.

[0010] During the interactive phase of training, at each time step, each agent uses its policy network to determine its decision action based on its state and the information it receives from its teammates. Based on the change in global state, global environmental rewards, and the agent's action set, samples are constructed and stored in the experience replay pool.

[0011] During the parameter update phase of training, samples (s n ,a n ,r n ,s n+1 ), calculate the loss according to the following formula:

[0012]

[0013] in, Indicates the experience replay pool The sample extracted from takes the expectation, Q tot (s n ,a n ; θ) represents the global Q value of the policy network θ, s n represents the global state before each agent performs a decision action at the nth time step, a n Indicates that the nth time step uses the policy network θ in the global state s n The set of decision actions of each agent determined by tot (s n+1 ,a′ n+1 ; θ′) represents the global Q value of the policy network θ corresponding to the target network θ′, s n+1 Indicates that each agent completes a in the nth time step n The global state after the decision action is a′ n+1 Indicates using the target network θ′ in the global state s n+1 The set of decision actions of each agent determined under r n Indicates that each agent completes a in the nth time step n where is the global environmental reward after the decision action; γ is the discount factor;

[0014] The decision action is: based on the current network of the i-th agent, the Q value of each executable action is calculated, and the action is performed at the current time step determined by the decision strategy; the current network calculates Qtot (s n ,a n ; θ) is the policy network, when calculating Q tot (s n+1 ,a′ n+1 ; θ′) is the target network;

[0015] Define the first Q value as: the Q value calculated based on the state of the i-th agent and the information it receives from its teammates, using its current network; define the second Q value as: the Q value calculated based only on the state of the i-th agent, using its current network;

[0016] Using the current network of the i-th agent, calculate the first Q value of each executable action of the i-th agent, and use the largest first Q value or the average of the first Q values of each executable action as the first state value V of the i-th agent. i ; Calculate the first Q value of the i-th agent's decision action relative to the first state value V i The additional benefit is the first action advantage A of the i-th agent i ;

[0017] Using the current network of the i-th agent, calculate the second Q value of each executable action of the i-th agent, and use the maximum second Q value or the average of the second Q values of each executable action as the second state value V′ of the i-th agent. i ; Calculate the second Q value of the i-th agent's decision action relative to the second state value V' i The additional benefit is the second action advantage A′ of the i-th agent i ;

[0018] The global Q value Q tot , calculated as follows:

[0019]

[0020] Among them, M i is the correction value of the ith agent, is the state correction value of the i-th agent, is the action correction value of the i-th agent, φ i and ψ i To utilize the weights generated by the hypernetwork, N is the number of agents; concat means concatenation.

[0021] Furthermore, the encoding network is a recurrent neural network, which calculates the state of each agent at the current time step based on the observation of the agent at the current time step and its state at the previous time step.

[0022] Furthermore, the encoding network is a GRU gated network, which is based on the observation of the i-th agent at the current time step t according to the following formula: and its state at the previous time step t-1 Calculate its state at the current time step t

[0023]

[0024] Among them, z i Update the output of the gate for the GRU gating network, r i Reset the output of the gate for the GRU gated network, is the hidden state of the i-th agent at time step t; W h 、W z and W r are weights, b h 、b z and b r are all bias terms; tanh is the tanh activation function, σ is the sigmoid function; ⊙ represents matrix element multiplication.

[0025] Furthermore, the policy network adopts a multi-layer perceptron.

[0026] Furthermore, using the current network, the process of calculating the first Q value includes:

[0027] First, the information received by the i-th agent at time step t from each friend Perform self-attention calculation to obtain its recommended information at time step t The suggested information Dimensions and friendly information The dimensions are the same;

[0028] Then, the state of the i-th agent at time step t is And its recommended information Splicing constitutes its strategic feature f containing teammate information i t ;

[0029] Afterwards, the strategy feature f of the i-th agent containing the communication information of teammates i t , input the multi-layer perceptron to obtain the first Q value of each executable action performed by the i-th agent;

[0030] Using the current network, the process of calculating the second Q value includes:

[0031] First, according to the input requirements of the multi-layer perceptron, the state of the i-th agent at time step t is Fill in the positions and obtain the strategic features that do not contain teammate information

[0032] Then, the strategy features of the i-th agent that do not contain the communication information of teammates are Input the multi-layer perceptron to obtain the second Q value of each executable action performed by the i-th agent.

[0033] Furthermore, according to the following formula, the information received by the i-th agent at time step t from each friend is: Perform self-attention calculation to obtain its recommended information at time step t

[0034]

[0035] Among them, W Q 、W K and W V are the transformation matrices of query vector, key vector and value vector respectively, is the state of the ith agent at time step t, is the information received by the i-th agent from the j-th friendly agent, d k for The dimension of ; T represents the matrix transpose.

[0036] Furthermore, the strategy feature f of the i-th agent containing the communication information of teammates i t In the corresponding suggestion information The characteristic value of is set to 0, and the strategic feature that does not contain teammate communication information is obtained.

[0037] Or, randomly generate a Gaussian noise of the same dimension, the generated Gaussian noise is combined with the state of the i-th agent at time step t The strategic features of the splicing structure do not include teammates' communication information

[0038] Furthermore, the first state value V is calculated according to the following formula: i and first action advantage A i :

[0039]

[0040] A i (u i )=Q i (u i )-V i

[0041] Calculate the second state value V′ according to the following formula: i and the second action advantage A′ i :

[0042]

[0043] A′ i (u i ) = Q′ i (u i )-V′ i

[0044] Among them, u m Indicates the mth executable action, u i represents the decision action of the i-th agent.

[0045] Furthermore, the global Q value Q tot , calculated as follows:

[0046]

[0047] in, is the dynamic adjustment coefficient of the i-th agent at the s-th time step; the time step s is the time step of the current calculation. tot (s n ,a n ; θ), the time step s is n, and when calculating Q tot (s n+1 ,a′ n+1 ;θ′), the time step s is n+1;

[0048] The dynamic adjustment coefficient Calculated as follows:

[0049]

[0050] in, is the dynamic adjustment coefficient of the i-th agent at the previous time step s-1 at the s-th time step, β′ is the correction term of the dynamic adjustment coefficient; clip is the clip function, η′ is the learning rate, and γ′ is the discount factor;

[0051] It represents the maximum first Q value that the i-th agent can obtain by executing each executable action according to its current network. represents the minimum first Q value that the i-th agent can obtain by executing each executable action according to its current network, u m Indicates the mth executable action.

[0052] Furthermore, the set of decision actions of each agent and the global state are used as input, and the absolute value of its output is used as the weight through an MLP super network. and In calculating Q tot (s n ,a n ;θ), the input of the hypernetwork is s n and a n , in calculating Q tot (s n+1 ,a′ n+1 ;θ′), the input of the hypernetwork is s n+1 and a′ n+1 .

[0053] The beneficial effects of the present invention are:

[0054] The method of the present invention employs a communication-based cooperative learning approach as its basic framework. During the interactive phase of training, agents first receive information from teammates before making decisions, allowing them to proactively utilize this information to improve their own decisions. The main improvements include: a novel mechanism for dynamically adjusting Q-values. This mechanism uses a Q-value containing teammate information as the first Q-value, corresponding to team decisions. This first Q-value allows agents to exchange information through a communication mechanism, leading to more coordinated decision-making. Simultaneously, a second Q-value, excluding teammate information, is introduced, corresponding to individual decisions. This second Q-value does not rely on communication with other agents; each agent makes decisions based solely on its own local observations. The difference between the first and second Q-values is then used to calculate the incremental benefit from teammate information, dynamically modifying the Q-value. Thus, the method of the present invention combines the advantages of both individual and team decision-making, clearly identifying the specific benefits of teammate communication. This allows agents to self-adjust their actions based on real-time feedback, proactively exploring team-benefiting actions using teammate information rather than passively waiting for global reward updates, thus better adapting to environmental changes. Furthermore, the present invention incorporates self-attention to optimize teammate communication, filtering information through the attention mechanism to reduce duplication and improve information quality.

[0055] The present invention is applicable to the optimization and execution of multi-agent collaborative tasks, such as games, drone formations, intelligent transportation, robot collaboration, distributed sensor networks and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of the dynamic Q value adjustment mechanism of the present invention;

[0057] Figure 2 Schematic diagram of the Q value calculation of the present invention. DETAILED DESCRIPTION

[0058] The present invention aims to provide a multi-agent cooperative learning method based on deep reinforcement learning. By designing a new mechanism for dynamically adjusting Q values, it uses teammate information to actively explore actions that are beneficial to the team, rather than passively waiting for global reward updates, and better adapts to environmental changes. It also solves the problem of existing communication-based cooperative learning methods that when the environment changes, the communication rules may no longer apply, resulting in the inability to adapt well to environmental changes.

[0059] The core improvement of the method of the present invention lies in the design of a new mechanism for dynamically adjusting Q values. This mechanism dynamically corrects Q values during the parameter update phase of training, allowing the inverse update to combine the advantages of individual and team decision-making. Therefore, the basic framework of the method of the present invention can still adopt existing communication-based collaborative learning methods. Except for the update step, other steps can be configured and adapted from existing technologies based on the specific task.

[0060] Specifically, the multi-agent system is divided into two opposing sides, one of which is chosen as the "self-agent." For each of the self-agents, an independent deep neural network is configured. This deep neural network includes an encoding network and a policy network. The policy network can be any existing network, for example, a multilayer perceptron.

[0061] Because the basic framework still uses existing communication-based cooperative learning methods, a deep reinforcement learning algorithm is used for training. During the interactive phase of training, at each time step, each agent uses its policy network to determine its decision-making action based on its state and the information it receives from its teammates. Based on the decision-making strategy, it constructs samples based on changes in the global state, global environmental rewards, and the agent's action set, and stores them in the experience replay pool.

[0062] During the parameter update phase of training, samples (s n ,a n ,r n ,s n+1 ), calculate the loss according to the following formula:

[0063]

[0064] in, Indicates the experience replay pool The sample extracted from takes the expectation, Q tot (s n ,a n ; θ) represents the global Q value of the policy network θ, s n represents the global state before each agent performs a decision action at the nth time step, a n Indicates that the nth time step uses the policy network θ in the global state sn The set of decision actions of each agent determined by tot (s n+1 ,a′ n+1 ; θ′) represents the global Q value of the policy network θ corresponding to the target network θ′, s n+1 Indicates that each agent completes a in the nth time step n The global state after the decision action is a′ n+1 Indicates using the target network θ′ in the global state s n+1 The set of decision actions of each agent determined under r n Indicates that each agent completes a in the nth time step n where corresponds to the global environmental reward after the decision action; γ is the discount factor.

[0065] The above loss function is a mean square error loss function used to measure the difference between the predicted Q value and the target Q value. However, it is different from the existing one in that its Q value calculation introduces a Q value calculated by dynamically adjusting the Q value. tot The decision action is: according to the current network of the i-th agent, the Q value of each executable action is calculated, and the action is performed at the current time step determined by the decision strategy; the current network calculates Q tot (s n ,a n ; θ) is the policy network, when calculating Q tot (s n+1 ,a′ n+1 ; θ′) is the target network.

[0066] In a multi-agent environment, relying on teammates' information or making decisions entirely on one's own may not be effective. Therefore, in order to dynamically adjust the Q value, the first Q value is defined as: the Q value calculated based on the state of the i-th agent and the information it receives from its teammates, using its current network; the second Q value is defined as: the Q value calculated based only on the state of the i-th agent, using its current network. By comparing the first Q value with the second Q value Q' i , the following contribution indicators can be defined:

[0067] ΔQ i =Q i -Q′ i

[0068] If ΔQ i >0, indicating that teammates’ information improves the quality of decision-making; if ΔQ i <0, indicating that teammates' information may mislead decision-making.

[0069] Therefore, through ΔQ iCalculating the increment brought by teammate information can quantify the actual contribution of teammate information to the current decision, clarify the specific advantages brought by teammate communication, and enable the intelligent agent to better understand the decision-making process of the intelligent agent, thereby improving the efficiency and accuracy of decision-making. Based on the contribution index ΔQ i , building a mechanism for dynamically adjusting the Q value allows the intelligent agent to self-adjust according to real-time feedback when taking action, and actively explore actions that are beneficial to the team through teammate information, rather than passively waiting for global rewards to be updated, so as to better adapt to environmental changes.

[0070] Furthermore, the first Q value and the second Q value are decoupled:

[0071] Using the current network of the i-th agent, calculate the first Q value of each executable action of the i-th agent, and use the largest first Q value or the average of the first Q values of each executable action as the first state value V of the i-th agent. i ; Calculate the first Q value of the i-th agent's decision action relative to the first state value V i The additional benefit is the first action advantage A of the i-th agent i .

[0072] Using the current network of the i-th agent, calculate the second Q value of each executable action of the i-th agent, and use the maximum second Q value or the average of the second Q values of each executable action as the second state value V′ of the i-th agent. i ; Calculate the second Q value of the i-th agent's decision action relative to the second state value V' i The additional benefit is the second action advantage A′ of the i-th agent i .

[0073] The above state values indicate the agent's optimal or average reward in the current environment, reflecting the global state value. Furthermore, action advantage indicates the additional benefit of choosing a decision action relative to the state value in the current environment. During training, due to the influence of decision strategies, such as greedy strategies, the obtained decision actions may not align with the optimal action corresponding to the optimal reward. Generally, training aims to align the two. Therefore, the optimal reward is chosen for decoupling.

[0074] Correspondingly, the contribution index ΔQ i Can also be divided into state correction value and action modifiers This decomposition method lays the foundation for quantitative credit allocation: Comparing V i and V′ i It can evaluate the extent to which collaboration improves global cognition and analyze A i and A′i The difference in performance can measure the strength of collaboration in modifying specific decisions. This decoupling measures the contribution of teammate information from two dimensions: global state assessment and local action decision-making.

[0075] Based on the above preparation, the inventors designed a new mechanism for dynamically adjusting the Q value by introducing the state correction value and action modifiers Dynamically modify the Q value, such as Figure 1 As shown, the global Q value Q is calculated according to the following formula: tot :

[0076]

[0077] Among them, M i is the correction value of the ith agent, is the state correction value of the i-th agent, is the action correction value of the i-th agent, φ i and ψ i To utilize the weights generated by the hypernetwork, N is the number of agents; concat means concatenation.

[0078] In the above formula, the Q value of the agent is decoupled into the state value V i and action advantage A i ; and introduce the correction value M i , quantify the gains of group collaboration on individual decision-making and eliminate the impact of non-independence. is the global state value, is the union of the optimal actions of the individuals, (φ i A i +ψ i M i ) is used to indicate the individual value of each agent’s “advantage”, through the weight φ i and ψ i The non-negative weights that constitute the individual value part ensure that the individual value is consistent with the global state value, and the union of the individual optimal actions is the global optimal.

[0079] In order to ensure the adaptability and flexibility of the model in multi-agent multi-task or dynamic environments, φ i and ψ i To use the weights generated by the hypernetwork. For example, take the set of decision actions of each agent and the global state as input, pass it through an MLP hypernetwork, and use the absolute value of its output as the weight and In calculating Q tot (s n ,a n ;θ), the input of the hypernetwork is s n and an , in calculating Q tot (s n+1 ,a′ n+1 ;θ′), the input of the hypernetwork is s n+1 and a′ n+1 During the training process, the weights φ output by the hypernetwork i and ψ i Used for Q tot The calculation and Q tot The feedback of is used to back-propagate and update the parameters of the hypernetwork.

[0080] In order to further achieve flexible strategy updates, a dynamic adjustment coefficient is introduced to control the influence of teammates' information. At this time, the global Q value Q tot , calculated as follows:

[0081]

[0082] in, is the dynamic adjustment coefficient of the i-th agent at the s-th time step; the time step s is the time step of the current calculation. tot (s n ,a n ; θ), the time step s is n, and when calculating Q tot (s n+1 ,a′ n+1 ;θ′), the time step s is n+1;

[0083] The dynamic adjustment coefficient Calculated as follows:

[0084]

[0085] in, is the dynamic adjustment coefficient of the i-th agent at the previous time step s-1 at the s-th time step, β′ is the correction term of the dynamic adjustment coefficient; clip is the clip function, η′ is the learning rate, and γ′ is the discount factor; It represents the maximum first Q value that the i-th agent can obtain by executing each executable action according to its current network. represents the minimum first Q value that the i-th agent can obtain by executing each executable action according to its current network, u m Indicates the mth executable action.

[0086] The following is further described with reference to the embodiments.

[0087] Example:

[0088] This embodiment provides a multi-agent cooperative learning method based on deep reinforcement learning. The multi-agents are divided into two opposing parties, one of which is selected as the self-agent. An independent deep neural network is configured for each of the self-agent agents. The deep neural network includes an encoding network and a policy network. Training is performed using a deep reinforcement learning algorithm, including the following steps:

[0089] S1. Model initialization

[0090] In this step, the experience replay pool and the strategy network and target network of each agent are initialized.

[0091] S2. Environment initialization

[0092] In this step, the environment and each agent are initialized according to the training project, and the initial global state is obtained by interacting with the environment.

[0093] The global state is used to describe the overall information of the environment. Taking the game as an example, it includes the state of one's own agent and the state of the opponent's agent. The state of the agent includes health, energy, skills, position, shield, and unit type.

[0094] S3, Environment Coding

[0095] In this step, each agent interacts with the environment to obtain local observations. Using the encoding network, each agent's state is derived based on these local observations. Local observations are each agent's observations of its local environment. In the case of a game, this includes its own state and the states of friendly and enemy units within its field of view.

[0096] The encoding network can adopt any existing encoding network, for example: the encoding network is a recurrent neural network, which calculates the state of each agent at the current time step based on the observation of the current time step and the state of each agent at the previous time step.

[0097] In this embodiment, the encoding network is a GRU gated network, which is based on the observation of the i-th agent at the current time step t according to the following formula: and its state at the previous time step t-1 Calculate its state at the current time step t

[0098]

[0099] Among them, z i Update the output of the gate for the GRU gating network, r i Reset the output of the gate for the GRU gated network, is the hidden state of the i-th agent at time step t; W h 、W zand W r are weights, b h 、b z and b r are all bias terms; tanh is the tanh activation function, σ is the sigmoid function; ⊙ represents matrix element multiplication.

[0100] S4. Action Decision

[0101] In this step, each agent's policy network calculates its Q-value, which includes its teammate communication information, based on its state and the communication information it receives from teammates. Then, based on the decision strategy, each agent's Q-value, which includes its teammate communication information, determines its decision action. The communication information sent by teammates and its processing are handled using existing technologies. This information, determined based on the training project, may include the agent's state, environment state, coordination information, cooperation information, and action information.

[0102] In this invention, the Q-value of an action decision is the first Q-value, which is defined as the Q-value calculated based on the state of the i-th agent and the information it receives from its teammates, using its current network. Furthermore, this invention defines a second Q-value for correction, which is the Q-value calculated based only on the state of the i-th agent, using its current network.

[0103] Simply put, the first Q value can be understood as the Q value including teammate information, and the second Q value can be understood as the Q value excluding teammate information. To ensure the consistency of the two dimensions and facilitate calculation, in this embodiment, Figure 2 As shown, using the current network, the process of calculating the first Q value includes:

[0104] First, the information received by the i-th agent at time step t from each friend Perform self-attention calculation to obtain its recommended information at time step t The suggested information Dimensions and friendly information The dimensions are the same as ; then, the state of the i-th agent at time step t is And its recommended information Splicing constitutes its strategic feature f containing teammate information i t ; Then, the strategy feature f of the i-th agent containing the communication information of teammates i t , input the multi-layer perceptron to obtain the first Q value of each executable action performed by the i-th agent.

[0105] Specifically, in this embodiment, according to the following formula, the information received by the i-th agent at time step t from each friendly party is: Perform self-attention calculation to obtain its recommended information at time step t

[0106]

[0107] Among them, W Q 、W K and W V are the transformation matrices of query vector, key vector and value vector respectively, is the state of the ith agent at time step t, is the information received by the i-th agent from the j-th friendly agent, d k for The dimension of ; T represents the matrix transpose.

[0108] Through the above attention calculation, it is possible to Dynamically adjust teammate messages The weight of , effectively extracts the most useful information.

[0109] In this embodiment, if Figure 2 As shown, using the current network, the process of calculating the second Q value includes:

[0110] First, according to the input requirements of the multi-layer perceptron, the state of the i-th agent at time step t is Fill in the positions and obtain the strategic features that do not contain teammate information Then, the strategy features of the i-th agent that do not contain the communication information of teammates are Input the multi-layer perceptron to obtain the second Q value of each executable action performed by the i-th agent.

[0111] Specifically, in this embodiment, the strategy feature f of the i-th agent containing the teammate communication information i t In the corresponding suggestion information The characteristic value of is set to 0, and the strategic feature that does not contain teammate communication information is obtained.

[0112] In addition, it can also randomly generate a Gaussian noise of the same dimension, the generated Gaussian noise is combined with the state of the i-th agent at time step t The strategic features of the splicing structure do not include teammates' communication information In this way, Gaussian noise can be understood as different from the actual suggestion information. A type of counterfactual information representing a fact, relative to the corresponding suggestion information The eigenvalue of is set to 0, which can introduce greater differences.

[0113] After calculating the first Q-value using the policy network, a greedy strategy can be used for decision making. Executable actions are all possible actions. For example, in games, these include move [direction], attack [id], stop, and no action. The decision action is the Q-value of each executable action calculated based on the current network of the i-th agent, and the action to be performed at the current time step, as determined by the decision policy. For ease of calculation, it is best to calculate the second Q-value of the policy network simultaneously in this step. However, since the Q-value of the target network will also need to be calculated later, the second Q-value of the policy network can also be calculated in a subsequent step.

[0114] S5. Interaction

[0115] In this step, each agent performs its decision-making action separately; then, by interacting with the environment, a new global state s is obtained. t+1 , and the global environmental reward r given based on the state change of the environment t .

[0116] Taking games as an example, rewards include rewards for killing enemy units, rewards for life and shield damage caused to the enemy, negative rewards for life and shield damage caused to one's own units, and negative rewards when one's own units are killed.

[0117] Global environmental reward r t , which is the collective reward for the team, usually derived from a reward function. The specific composition of the reward function is defined by the environment. In the SMAC environment, the concentrated reward is the total health and shield value inflicted on all enemy units, plus the number of killed units returned by the environment, after some normalization.

[0118] S6. Build samples

[0119] In this step, based on the changes in the global state, the global environmental rewards, and the action set of the agent, samples are constructed and stored in the experience replay pool. Specifically, the sample (s t ,a t ,r t ,s t+1 ) and stored in the experience replay pool, where s t Indicates the current global state, a t represents the set of decision actions of each agent, r t Represents the global environmental reward after executing the decision action, s t+1 Represents the new global state after the decision action is executed.

[0120] S7. Update of policy network θ

[0121] In this step, it is determined whether the update condition of the policy network θ is met. If so, a sample (s n ,a n ,r n ,s n+1 ), calculate the loss and update the policy network θ according to the following formula:

[0122]

[0123] in, Indicates the experience replay pool The sample extracted from takes the expectation, Q tot (s n ,a n ; θ) represents the global Q value of the policy network θ, s n represents the global state before each agent performs a decision action at the nth time step, a n Indicates that the b-th time step uses the policy network θ in the global state s n The set of decision actions of each agent determined by tot (s n+1 ,a′ n+1 ; θ′) represents the global Q value of the policy network θ corresponding to the target network θ′, s n+1 Indicates that each agent completes a in the nth time step n The global state after the decision action is corresponding to a′ n+1 Indicates using the target network θ′ in the global state s n+1 The set of decision actions of each agent determined under r n Indicates that each agent completes a in the nth time step n The global environment reward after the corresponding decision action; γ is the discount factor; the decision action is: according to the current network of the i-th agent, the Q value of each executable action is calculated, and the action performed at the current time step determined by the decision strategy; the current network, when calculating Q tot (s n ,a n ; θ) is the policy network, when calculating Q tot (s n+1 ,a′ n+1 ; θ′) is the target network.

[0124] Using the current network of the i-th agent, calculate the first Q value of each executable action of the i-th agent, and use the largest first Q value or the average of the first Q values of each executable action as the first state value V of the i-th agent. i ; Calculate the first Q value of the i-th agent's decision action relative to the first state value V iThe additional benefit is the first action advantage A of the i-th agent i .

[0125] Specifically, in this embodiment, the first state value V is calculated according to the following formula: i and first action advantage A i :

[0126]

[0127] A i (u i )=Q i (u i )-V i

[0128] Among them, u m Indicates the mth executable action, u i represents the decision action of the i-th agent.

[0129] Using the current network of the i-th agent, calculate the second Q value of each executable action of the i-th agent, and use the maximum second Q value or the average of the second Q values of each executable action as the second state value V′ of the i-th agent. i ; Calculate the second Q value of the i-th agent's decision action relative to the second state value V' i The additional benefit is the second action advantage A′ of the i-th agent i .

[0130] Specifically, in this embodiment, the second state value V′ is calculated according to the following formula: i and the second action advantage A′ i :

[0131]

[0132] A′ i (u i ) = Q′ i (u i )-V′ i

[0133] Among them, u m Indicates the mth executable action, u i represents the decision action of the i-th agent.

[0134] The global Q value Q tot , calculated as follows:

[0135]

[0136]

[0137] Among them, M i is the correction value of the ith agent, is the state correction value of the i-th agent, is the action correction value of the i-th agent, φ i and ψ i To utilize the weights generated by the hypernetwork, is the dynamic adjustment coefficient of the i-th agent at the n-th time step; N is the number of agents; concat means concatenation.

[0138] The dynamic adjustment coefficient Calculated as follows:

[0139]

[0140] in, is the dynamic adjustment coefficient of the i-th agent at the previous time step s-1 at the s-th time step, β′ is the correction term of the dynamic adjustment coefficient; clip is the clip function, η′ is the learning rate, γ′ is the discount factor; time step s is the time step of the current calculation, and when calculating Q tot (s n ,a n ; θ), the time step s is n, and when calculating Q tot (s n+1 ,a′ n+1 ;θ′), the time step s is n+1; It represents the maximum first Q value that the i-th agent can obtain by executing each executable action according to its current network. represents the minimum first Q value that the i-th agent can obtain by executing each executable action according to its current network, u m Indicates the mth executable action.

[0141] The set of decision actions of each agent and the global state are used as input, and the absolute value of its output is used as the weight through an MLP super network. and In calculating Q tot (s n ,a n ;θ), the input of the hypernetwork is s n and a n , in calculating Q tot (s n+1 ,a′ n+1 ;θ′), the input of the hypernetwork is s n+1 and a′ n+1 .

[0142] S8. Update of target network θ′

[0143] In this step, it is determined whether the update conditions of the target network θ′ are met. If so, the parameters of the strategy network are copied to update the target network.

[0144] S9. Complete training

[0145] In this step, it is determined whether the task goal of the training project has been achieved. If not, the process returns to step S3. If it has been achieved, it is determined whether the training has been completed. If it has been completed, the training is terminated. Otherwise, the process returns to step S2.

[0146] Furthermore, a matrix game simulation was conducted using two agents, A and B, each with three possible actions. The team payoff matrix is composed of the combinations of the two agents and their actions as rows and columns. The values in the team payoff matrix represent the team's payoff after the two agents perform the corresponding actions. The actual team payoff matrix is shown in Table 1, where the optimal joint action is (A1, B1).

[0147] Table 1. Real team benefit matrix

[0148] A1 A2 A3 B1 12 -12 -12 B2 -12 0 0 B3 -12 0 0

[0149] The experiment analyzes three methods, namely VDN, QMix and the embodiment, and focuses on comparing the fitting ability of different methods on the team payoff matrix and whether they can find the optimal joint action.

[0150] The team payoff matrix of the VDN experiment is shown in Table 2. In the VDN experiment, the Q value of agent A's action is [-12.0, 0.0, 0.0], and the Q value of agent B's action is [-12.0, 0.0, 0.0]. They fail to fit the true team payoff matrix, and the strategies selected by the agents cannot approach the optimal joint action (A1, B1).

[0151] Table 2. VDN team benefit matrix

[0152] A1 A2 A3 B1 -24 -12 -12 B2 -12 0 0 B3 -12 0 0

[0153] The team payoff matrix of the QMix experiment is shown in Table 3. In the QMix experiment, the Q value of agent A's action is [-75.98, -0.00, -0.00], and the Q value of agent B's action is [-5.75, 0.28, 0.28]. The team payoff matrix shows obvious deviations. The agents always fail to select the optimal joint action (A1, B1) and can only achieve suboptimal local solutions.

[0154] Table 3. QMix team profit matrix

[0155] A1 A2 A3 B1 -7.19 -7.19 -7.19 B2 -7.19 3.01 3.01 B3 -7.19 3.01 3.01

[0156] The team payoff matrix for the embodiment method is shown in Table 4. In the embodiment method experiment, the Q-values of agent A's actions were [0.6, -9.9, -9.7], and the Q-values of agent B's actions were [0.6, -9.9, -9.7]. The team payoff matrix demonstrated excellent fitting ability, not only reproducing the actual team payoff matrix well but also selecting the optimal joint action (A1, B1). The distribution of the agents' Q-values indicates that they were able to absorb suggestions from teammates, balance their own interests with those of the team, and ultimately achieve the global optimal strategy. Therefore, the embodiment method has strong strategic coordination capabilities, enabling the establishment of effective collaborative relationships between agents, achieving optimal team payoffs while sacrificing local interests.

[0157] Table 4. Team benefit matrix of the embodiment method

[0158] A1 A2 A3 B1 11.9 -6.1 -6.1 B2 -6.1 -6.2 -6.2 B3 -6.1 -6.2 -6.2

[0159] The experimental results above demonstrate that traditional methods for fitting team payoffs can lead to Nash equilibrium oscillations or difficulty converging to the team's optimal point when some agents need to make their own suboptimal decisions. However, the method presented in this paper, after incorporating the opinions of teammates, can better fit the team payoff matrix.

[0160] Furthermore, multiple competitive scenarios in StarCraft II were selected for validation testing. The tests demonstrated that the proposed method, by decoupling the Q-value into state V, action advantage A, and suggestion correction M, allows the agent to dynamically adjust based on the suggestions of teammates, achieving better compromise and coordination within the collaborative action space to reach the global optimal solution. However, this requires the agent to gradually learn how to distinguish between its own contributions and the influence of teammates' suggestions, resulting in slower strategy optimization and a slow initial increase in win rate. However, in later learning stages, the proposed method outperformed other algorithms, demonstrating a higher win rate and more stable collaborative performance, with a win rate consistently above 0.75 in all competitive scenarios.

[0161] Finally, it should be noted that the above embodiments are merely preferred implementations and are not intended to limit the present invention. It should be noted that those skilled in the art will be able to make modifications, equivalent substitutions, and improvements without departing from the spirit and scope of the present invention and the claims, all of which should be included within the scope of protection of the present invention.

Claims

1. A multi-agent cooperative learning method based on deep reinforcement learning. The multi-agents are divided into two opposing parties. One of the parties is selected as the self-agent. For each of the self-agent agents, an independent deep neural network is configured. The deep neural network includes an encoding network and a policy network. The training is performed using a deep reinforcement learning algorithm. During the interactive phase of training, at each time step, each agent uses its policy network to determine its decision action based on its state and the information it receives from its teammates. Based on the decision strategy, it constructs samples based on the changes in the global state, the global environmental rewards, and the agent's action set and stores them in the experience replay pool. This is characterized by: During the parameter update phase of training, samples (s n ,a n ,r n ,s n+1 ), calculate the loss according to the following formula: in, Indicates the experience replay pool The sample extracted from takes the expectation, Q tot (s n ,a n ; θ) represents the global Q value of the policy network θ, s n represents the global state before each agent performs a decision action at the nth time step, a n Indicates that the nth time step uses the policy network θ in the global state s n The set of decision actions of each agent determined by tot (s n+1 ,a′ n+1 ; θ′) represents the global Q value of the policy network θ corresponding to the target network θ′, s n+1 Indicates that each agent completes a in the nth time step n The global state after the decision action is a′ n+1 Indicates using the target network θ′ in the global state s n+1 The set of decision actions of each agent determined under r n Indicates that each agent completes a in the nth time step n where is the global environmental reward after the decision action; γ is the discount factor; The decision action is: based on the current network of the i-th agent, the Q value of each executable action is calculated, and the action is performed at the current time step determined by the decision strategy; the current network calculates Qtot(s n ,a n ; θ) is the policy network, when calculating Q tot (s n+1 ,a′ n+1 ; θ′) is the target network; Define the first Q value as: the Q value calculated based on the state of the i-th agent and the information it receives from its teammates, using its current network; define the second Q value as: the Q value calculated based only on the state of the i-th agent, using its current network; Using the current network of the i-th agent, calculate the first Q value of each executable action of the i-th agent, and use the largest first Q value or the average of the first Q values of each executable action as the first state value V of the i-th agent. i ; Calculate the first Q value of the i-th agent's decision action relative to the first state value V i The additional benefit is the first action advantage A of the i-th agent i ; Using the current network of the ith agent, calculate the second Q value of each executable action of the ith agent, and use the maximum second Q value or the average of the second Q values of each executable action as the second state value V of the ith agent. i '; Calculate the second Q value of the i-th agent's decision action relative to the second state value V i ′’s additional benefit, as the second action advantage A′ of the i-th agent i ; The global Q value Q tot , calculated as follows: Among them, M i is the correction value of the ith agent, is the state correction value of the i-th agent, is the action correction value of the i-th agent, φ i and ψ i To utilize the weights generated by the hypernetwork, N is the number of agents; concat means concatenation.

2. The multi-agent cooperative learning method based on deep reinforcement learning according to claim 1, characterized in that: The encoding network is a recurrent neural network that calculates the state of each agent at the current time step based on the observation of the agent at the current time step and its state at the previous time step.

3. The multi-agent cooperative learning method based on deep reinforcement learning according to claim 1, characterized in that: The encoding network is a GRU gated network, which is based on the observation of the i-th agent at the current time step t according to the following formula: and its state at the previous time step t-1 Calculate its state at the current time step t Among them, z i Update the output of the gate for the GRU gating network, r i Reset the output of the gate for the GRU gated network, is the hidden state of the i-th agent at time step t; W h 、W z and W r are weights, b h 、b z and b r are all bias terms; tanh is the tanh activation function, σ is the sigmoid function; ⊙ represents matrix element multiplication.

4. The multi-agent cooperative learning method based on deep reinforcement learning according to claim 1, characterized in that: The policy network adopts a multi-layer perceptron.

5. The multi-agent cooperative learning method based on deep reinforcement learning according to claim 1, characterized in that: Using the current network, the process of calculating the first Q value includes: First, the information received by the i-th agent at time step t from each friend Perform self-attention calculation to obtain its recommended information at time step t The suggested information Dimensions and friendly information The dimensions are the same; Then, the state of the i-th agent at time step t is And its recommended information Splicing constitutes its strategic feature f containing teammate information i t ; Afterwards, the strategy feature f of the i-th agent containing the communication information of teammates i t , input the multi-layer perceptron to obtain the first Q value of each executable action performed by the i-th agent; Using the current network, the process of calculating the second Q value includes: First, according to the input requirements of the multi-layer perceptron, the state of the i-th agent at time step t is Fill in the positions and obtain the strategic features that do not contain teammate information Then, the strategy features of the i-th agent that do not contain the communication information of teammates are Input the multi-layer perceptron to obtain the second Q value of each executable action performed by the i-th agent.

6. The multi-agent cooperative learning method based on deep reinforcement learning according to claim 5, characterized in that: According to the following formula, the information received by the i-th agent at time step t from each friend is Perform self-attention calculation to obtain its recommended information at time step t Among them, W Q 、W K and W V are the transformation matrices of query vector, key vector and value vector respectively, is the state of the ith agent at time step t, is the information received by the i-th agent from the j-th friendly agent, d k for The dimension of ; T represents the matrix transpose.

7. The multi-agent cooperative learning method based on deep reinforcement learning according to claim 5, characterized in that: The strategy feature f of the i-th agent containing the communication information of teammates i t In the corresponding suggestion information The characteristic value of is set to 0, and the strategic feature that does not contain teammate communication information is obtained. Or, randomly generate a Gaussian noise of the same dimension, the generated Gaussian noise is combined with the state of the i-th agent at time step t The strategic features of the splicing structure do not include teammates' communication information 8. The multi-agent cooperative learning method based on deep reinforcement learning according to any one of claims 1 to 7, characterized in that: Calculate the first state value V according to the following formula: i and first action advantage A i : A i (u i )=Q i (u i )-V i Calculate the second state value V according to the following formula: i ' and the second action advantage A' i : A′ i (u i )=Q′ i (u i )-V i ′ Among them, u m Indicates the mth executable action, u i represents the decision action of the i-th agent.

9. The multi-agent cooperative learning method based on deep reinforcement learning according to any one of claims 1 to 7, characterized in that: The global Q value Q tot , calculated as follows: in, is the dynamic adjustment coefficient of the i-th agent at the s-th time step; the time step s is the time step of the current calculation. n ,a n ; θ), the time step s is n, and when calculating Q tot (s n+1 ,a′ n+1 ;θ′), the time step s is n+1; The dynamic adjustment coefficient Calculated as follows: in, is the dynamic adjustment coefficient of the i-th agent at the previous time step s-1 at the s-th time step, β′ is the correction term of the dynamic adjustment coefficient; clip is the clip function, η′ is the learning rate, and γ′ is the discount factor; It represents the maximum first Q value that the i-th agent can obtain by executing each executable action according to its current network. represents the minimum first Q value that the i-th agent can obtain by executing each executable action according to its current network, u m Indicates the mth executable action.

10. A multi-agent cooperative learning method based on deep reinforcement learning according to any one of claims 1 to 7, characterized in that: The set of decision actions of each agent and the global state are used as input, and the absolute value of its output is used as the weight through an MLP super network. and In calculating Qtot(s n ,a n ;θ), the input of the hypernetwork is s n and a n , in calculating Q tot (s n+1 ,a′ n+1 ;θ′), the input of the hypernetwork is s n+1 and a′ n+1 .

Images