Multi-agent reinforcement learning model training method, learning method and device

Abstract

The embodiment of the invention provides a multi-agent reinforcement learning model training method, learning method and device, and the method comprises the steps: carrying out the multi-round iteration training of a multi-agent reinforcement learning model, and obtaining a multi-agent reinforcement learning model in each iteration training process, executing the following action selection steps for multiple times: inputting the current global state matrix and the current observation matrix into an agent network for attention output analysis and agent Q value calculation to obtain Q value data of each agent, the Q value of each feasible action being used for representing the action value of each feasible action; inputting the Q value data of each agent into a hybrid network for weighted hybrid processing to obtain joint Q value data, the joint Q value data being used for representing action values of the plurality of agents when the plurality of agents adopt joint actions under the current global state matrix; and based on the joint Q value data, performing current network parameter updating by adopting a loss function. According to the invention, the learning accuracy and efficiency of the multi-agent reinforcement learning algorithm are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a multi-agent reinforcement learning model training method, learning method, and device. Background Art

[0002] Real-world multi-agent scenarios often involve a combination of agents of varying types and numbers. To address this complexity, many researchers have introduced attention mechanisms into multi-agent reinforcement learning. Existing attention mechanisms in multi-agent reinforcement learning typically place the agent in a query matrix, and simultaneously place teammates and opponents in the query matrix and content matrix within the attention mechanism module. This allows the agent's decision to be calculated based on the attention weights.

[0003] However, this existing attention mechanism architecture has significant shortcomings. It indiscriminately directs attention to both teammates and opponents, failing to specifically address the relationships between different types of agents. This indiscriminate approach makes it difficult for agents to develop optimal strategies in complex and ever-changing multi-agent environments, limiting their adaptability to real-world tasks and decision-making efficiency. Furthermore, this approach fails to fully exploit the potential for collaboration between agents in complex collaborative tasks, resulting in low accuracy and efficiency in multi-agent reinforcement learning algorithms. Summary of the Invention

[0004] The embodiments of the present disclosure provide a multi-agent reinforcement learning model training method, learning method and device to solve the problems of low accuracy and efficiency of existing multi-agent reinforcement learning algorithms.

[0005] Based on the above problems, in a first aspect, an embodiment of the present disclosure provides a multi-agent reinforcement learning model training method, comprising: pre-building a multi-agent reinforcement learning model, wherein the multi-agent reinforcement learning model includes: an agent network and a hybrid network;

[0006] The multi-agent reinforcement learning model is trained for multiple rounds of iterations. During each round of iterative training, the following action selection steps are performed multiple times:

[0007] Inputting the current global state matrix and the current observation matrix into the agent network for attention output analysis and agent Q value calculation to obtain Q value data for each agent, wherein the Q value data for each agent includes the Q value of each action corresponding to each agent, and the Q value of each action is used to represent the action value of each action;

[0008] Inputting the Q value data of each agent into the hybrid network for weighted mixing processing to obtain joint Q value data, which is used to represent the action value of multiple agents when taking joint actions under the current global state matrix;

[0009] Based on the joint Q value data, a preset loss function is used to update the network parameters.

[0010] Secondly, a multi-agent reinforcement learning method is provided, comprising:

[0011] The obtained global state matrix and observation matrix are input into the multi-agent reinforcement learning model to obtain the joint Q value data;

[0012] Constructing an agent training strategy based on the joint Q-value data;

[0013] Among them, the multi-agent reinforcement learning model is trained based on the multi-agent reinforcement learning model training method described in the first aspect.

[0014] In a third aspect, a multi-agent reinforcement learning device is provided, comprising:

[0015] The input module is used to input the acquired global state matrix and observation matrix into the multi-agent reinforcement learning model to obtain joint Q value data;

[0016] A construction module is used to construct an intelligent agent training strategy based on the joint Q value data.

[0017] Advantages of the disclosed embodiments include: The classified attention mechanism disclosed herein can distinguish between teammates and opponents and perform targeted attention calculations for different types of agents. This allows agents to more accurately consider their respective characteristics and relationships when formulating strategies, thereby improving the accuracy and effectiveness of decisions. By analyzing the attention output of the global state matrix and observation matrix through the agent network, the Q-value data for each agent can be more accurately calculated. The Q-value of each action corresponding to each agent clearly reflects the value of that action in the current context, helping the agent make more optimal action choices. The Q-value data of each agent is input into a hybrid network for weighted mixing, resulting in a joint Q-value that comprehensively considers the decision-making of multiple agents. This joint Q-value calculation method ensures the integrity and coordination of decision-making in complex multi-agent environments, thereby improving the performance of the algorithm in practical applications. The classified attention mechanism not only improves decision accuracy but also increases algorithm speed by optimizing the calculation steps. During multiple rounds of iterative training, the action selection step is repeated and network parameters are updated based on the joint Q-value data. This can accelerate training convergence and improve training efficiency while ensuring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a multi-agent reinforcement learning model training method provided in an embodiment of the present disclosure;

[0019] Figure 2 A model structure diagram of a multi-agent reinforcement learning model provided in an embodiment of the present disclosure;

[0020] Figure 3 A network architecture diagram of the intelligent agent network provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The present disclosure provides a multi-agent reinforcement learning model training method. The preferred embodiments of the present disclosure are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only intended to illustrate and explain the present disclosure and are not intended to limit the present disclosure. Furthermore, the embodiments and features of the embodiments in this application may be combined with each other unless there is a conflict.

[0022] The present disclosure provides a multi-agent reinforcement learning model training method, such as Figure 1 Shown, including:

[0023] S101. Pre-build a multi-agent reinforcement learning model, which includes an agent network and a hybrid network.

[0024] S102. Perform multiple rounds of iterative training on the multi-agent reinforcement learning model. During each round of iterative training, perform the following action selection steps multiple times:

[0025] S1021. Input the current global state matrix and the current observation matrix into the agent network for attention output analysis and agent Q-value calculation to obtain Q-value data for each agent. The Q-value data for each agent includes the Q-value of each action corresponding to each agent. The Q-value of each action is used to represent the action value of each action.

[0026] S1022. Input the Q value data of each agent into the hybrid network for weighted mixing processing to obtain joint Q value data. The joint Q value data is used to represent the action value of multiple agents when taking a joint action under the current global state matrix;

[0027] S1023. Based on the joint Q value data, a preset loss function is used to update the network parameters.

[0028] First, let's briefly explain the theoretical ins and outs used in this paper. This paper considers a decentralized partially observable Markov decision process (Dec-POMDP). Dec-POMDP is often described as an array (S, A, O, P, r, N, μ), where all agents i have a state (state) s i , the global state can be a set: s = {s i |i∈N}. Where N is the set of all agents (including teammate agents and opponent agents), and the state of each agent consists of two parts: s i =[φ i ,ψ i ], where φ i represents the current state of agent i (such as position, speed, etc.), ψ i Represents the type of agent i (e.g., goalkeeper, defender, midfielder, forward, etc. in football), which is usually represented by a discrete set. The type of agent is fixed during its lifetime and affects the state change range and reward function of the agent. At the same time, not all agents are observed by other agents, so we can define an observability mask matrix consisting of 0 and 1: μ(s i ,s j )∈{1,0}, where any agent can always observe itself: Therefore, an agent's observation is defined by: i ={s i |μ(s i ,s j )=1,j∈N}, each agent i can take action a at each moment i , the joint action of the entire multi-agent group is defined as: a={a i |i∈N}. P(s′|s,a) is the agent’s transition probability, which represents the probability of transitioning to the next state s′ by taking the joint action a in the joint state s. r(s,a) is the reward function, which maps the global state and the joint action to a single scalar.

[0029] The goal of the whole problem is to learn a strategy that maximizes the expected value of discounted rewards. Q-learning is to learn an action value function Q tot , and use this function to select the action with the highest expectation. In one possible implementation, the entire Q function can be defined as follows:

[0030]

[0031] Due to partial observability, s is unavailable information, but the state can be approximated by the trajectory information composed of historical actions and observations:

[0032] Q tot (τ t ,a t )≈Q tot (s t ,a t )

[0033] in Represents the entire trajectory information, represents the trajectory information of agent i, γ represents the discount factor, and t represents the time step.

[0034] It can be considered that the Q function value of the entire group is a "synthesis" of the Q function values ​​of each agent in the group, so the Q function can be decomposed:

[0035]

[0036] Among them, Q i (τ i ,a i ) is the Q value of agent i, |N ally | represents the number of agents in the entire group, g θ (·) represents a neural network with parameter θ.

[0037] Based on the above theoretical basis, in the embodiment of the present disclosure, a multi-agent reinforcement learning model is pre-built. The model structure diagram of the multi-agent reinforcement learning model is as follows: Figure 2 As shown in Figure 2, the model includes an agent network and a hybrid network. During the construction of the model, the agent network is used to receive the input global state matrix and observation matrix, analyze them through the attention mechanism, and calculate the Q value data of each agent. Specifically, the network architecture of the agent network is shown in Figure 2. Figure 3As shown, the agent network includes a matrix analysis layer, multiple classification attention mechanism layers, a gated recurrent unit, and a fully connected layer. In the matrix analysis layer, the global state matrix and observation matrix are first input, and the state analysis is performed on each agent to obtain the state information and observation information of each agent. Subsequently, this information is sequentially input into the multi-head classification attention mechanism layer, and the attention output analysis is performed on the state information of teammates and opponents respectively to obtain the teammate attention output data and the opponent attention output data. Afterwards, these data are spliced ​​and input into the gated recurrent unit for data processing, and finally linearly transformed through the fully connected layer to obtain the Q-value data of each agent. In other words, the entire network architecture can be divided into two parts. The first part extracts the Q-value of each agent, and the second part mixes the Q-values ​​of all agents to calculate the overall Q-value of the entire multi-agent: Q tot Among them, the global state matrix can be used to represent the state information of all agents in this iteration, and the observation matrix can be used to represent the current observation information of each agent.

[0038] The global state information matrix can be expressed as: s∈R |N|×d ; The observable matrix can be expressed as: μ∈R |N|×|N| ; Where |N| is the number of agents, d is the dimension of each agent representation, and each row of the matrix s is s i,* =s i Represents the status information of each entity.

[0039] Next, the constructed multi-agent reinforcement learning model is subjected to multiple rounds of iterative training. In each round of iterative training, the action selection step is first executed. In this step, the current global state matrix and observation matrix are input into the agent network, attention output analysis is performed, and the agent Q value is calculated. The Q value data of each agent contains the Q value of each possible action corresponding to the agent, and these Q values ​​represent the action value of each possible action. Afterwards, these Q value data are input into the hybrid network for weighted mixing processing to obtain joint Q value data. The joint Q value data is used to represent the overall action value when multiple agents take joint actions under the current global state matrix. Based on these joint Q value data, the preset loss function is used to update the network parameters. During the training process, the decision-making ability and accuracy of the model are improved by continuously adjusting the network parameters.

[0040] To further illustrate the practical application of this method, let's take the example of robot soccer strategy training. Suppose two teams of robot players compete on a standard soccer field. Each robot, acting as an intelligent agent, makes decisions based on the situation on the field, executing actions such as passing, shooting, and stealing the ball. In this scenario, the global state matrix represents the position, velocity, and other state information of all robot players on the field, while the observation matrix contains each robot player's perception of its surroundings. During the game, each robot player must not only coordinate with teammates but also implement defensive strategies against opponents.

[0041] During actual training, the global state matrix and observation matrix are first input into the agent network. The attention mechanism then calculates the teammate and opponent attention outputs for each robot player. Then, through gated recurrent units and fully connected layers, Q-value data for each robot player's different actions is obtained. These Q-value data are then input into a hybrid network for weighted processing to generate a joint Q-value, representing the value of the entire team's joint action in the current state. Using a pre-set loss function, the network parameters are adjusted and optimized. Repeated rounds of iterative training continuously improve the robot players' decision-making abilities and game strategies.

[0042] Through this method, robot players can quickly make reasonable decisions in complex and ever-changing game environments, effectively cooperating with teammates in offense while also accurately defending against opponents' actions. For example, when a robot player has the ball, it will calculate the Q values ​​of multiple possible actions through the intelligent agent network based on the current global state matrix and observation matrix, and select the optimal action to execute. If the action is a pass, it will comprehensively consider the position of teammates and the opponent's defensive strategy to select the optimal passing path; if the action is a shot, it will evaluate the success rate of the shot and shoot at the appropriate time. Similarly, when the robot player is in a defensive state, it will decide whether to move forward to steal the ball or return to defend and fill in the position based on the opponent's state information, thereby improving the defensive effectiveness.

[0043] In the disclosed embodiments, the classified attention mechanism can distinguish between teammates and opponents and perform targeted attention calculations for different types of agents. This allows agents to more accurately consider their respective characteristics and relationships when formulating strategies, thereby improving the accuracy and effectiveness of decisions. By analyzing the attention output of the global state matrix and observation matrix through the agent network, the Q-value data for each agent can be more accurately calculated. The Q-value of each action corresponding to each agent clearly reflects the value of the action in the current context, helping the agent make more optimal action choices. The Q-value data of each agent is input into the hybrid network for weighted mixing, resulting in a joint Q-value data that comprehensively considers the decision-making of multiple agents. This joint Q-value calculation method ensures the integrity and coordination of decision-making in complex multi-agent environments, thereby improving the performance of the algorithm in practical applications. The classified attention mechanism not only improves decision accuracy but also increases algorithm speed by optimizing the calculation steps. During multiple rounds of iterative training, the action selection step is repeated and network parameters are updated based on the joint Q-value data. This can accelerate training convergence and improve training efficiency while ensuring accuracy.

[0044] In another embodiment of the present disclosure, the specific process of executing step S1021 includes the following steps:

[0045] (1) Input the global state matrix S and the observation matrix μ into the matrix analysis layer to analyze the state of each agent and obtain the state information S of each agent. i ;

[0046] (2) The state information S of each agent i Input the multi-head classification attention mechanism layer in turn to perform attention output analysis on the global state matrix S and the observation matrix μ, and obtain the teammate attention output data Z X And the opponent's attention output data Z Y ;

[0047] (3) Output data Z of teammates’ attention X And the opponent's attention output data Z Y Splice and pass through a layer of network to obtain the output data of a single classification attention mechanism, and splice all the classification attention mechanism outputs to obtain the output data of the multi-head classification attention mechanism;

[0048] (4) Input the output data of the multi-head classification attention mechanism into the gated recurrent unit for data processing to obtain the recurrent unit output data;

[0049] (5) The output data of the cyclic unit is input into the fully connected layer for linear transformation processing to obtain the Q value data of each agent, wherein the Q value data of each agent includes the Q value of each action corresponding to each agent.

[0050] Specifically, the agent network sequentially inputs the extracted state information and observation information into a multi-head classification attention mechanism layer. The classification attention mechanism layer performs attention output analysis on the state information of teammates and opponents separately, generating teammate attention output data and opponent attention output data. Specifically, the classification attention mechanism layer calculates the query vector for each agent's state information using a preset query weight matrix and processes the state information using key vectors and value vectors. The agent network can separately focus on the state information of teammates and opponents, thereby considering the relationships between different types of agents when making decisions. The agent network then concatenates the teammate attention output data and opponent attention output data and inputs the concatenated data into a gated recurrent unit for further processing. The gated recurrent unit combines the current input with the previous memory state to generate the recurrent unit output data. Finally, the recurrent unit output data is input into a fully connected layer for linear transformation, generating the Q-value data for each agent. The Q-value data for each agent includes the Q-value for each of its corresponding available actions, indicating the value of each action.

[0051] During multiple rounds of iterative training, the network parameters of the multi-agent reinforcement learning model are first initialized, and multiple rounds of training are performed. During each round of training, the model is optimized by repeatedly executing the action selection step. The specific steps are as follows: the current global state matrix and observation matrix are input into the agent network, and the Q-value data of each agent is calculated using a classification attention mechanism. Next, an action is selected for each agent based on the Q-value data, using the action selection probability. It should be noted that the action selection probability here generally adopts an ε-greedy strategy, that is, an action is randomly selected with a probability of ε, and the action with the highest Q-value is selected with a probability of 1-ε. After the action is selected, the action is executed and the feedback from the environment is observed to obtain the new global state matrix and total reward.

[0052] The Q-value data for each agent is then fed into the hybrid network for weighted mixing. The hybrid network then performs a weighted combination of the Q-value data from each agent to generate a joint Q-value. This joint Q-value represents the overall value of the actions performed jointly by all agents under the current global state. This approach effectively evaluates the effectiveness of the collaborative work of multiple agents. Finally, based on this joint Q-value data, the network parameters are updated using a preset loss function. Specifically, the error between the target Q-value and the predicted Q-value is calculated, and the network parameters are adjusted using a backpropagation algorithm. Through multiple rounds of iterative training, the network parameters are continuously optimized to improve model performance and decision-making effectiveness.

[0053] In practical applications, consider robot soccer strategy training as an example. Imagine two robot soccer teams competing on a field. Each robot, acting as an intelligent agent, makes decisions based on the situation on the field. The global state matrix S represents the position, velocity, and other state information of all robot players, while the observation matrix μ represents each robot's observability of the others. During the game, each robot must not only coordinate with its teammates but also implement defensive strategies against its opponents.

[0054] In each training step, the global state matrix and observation matrix are input into the intelligent agent network. The state information of teammates and opponents is calculated separately through the classification attention mechanism to obtain the Q value data corresponding to each robot. For example, when a robot has the ball, it will calculate the Q values ​​of multiple possible actions based on the current global state and observation matrix through the intelligent agent network, and select the optimal action to execute. If the action is a pass, it will comprehensively consider the position of teammates and the opponent's defensive strategy to select the optimal passing path; if the action is a shot, the success rate of the shot will be evaluated and the shot will be taken at the appropriate time. Similarly, when the robot is in a defensive state, it will decide whether to move forward to steal the ball or return to defend and fill in the position based on the opponent's state information, thereby improving the defensive effectiveness.

[0055] In another embodiment of the present disclosure, the specific process of performing the attention output analysis step on the current global state matrix S and the current observation matrix μ includes the following steps:

[0056] (1) The state information S of each agent i Input the multi-head classification attention mechanism layer in sequence, through the preset query weight matrix W Q For each agent's state information S i Perform query vector calculation to obtain the query vector Q of each agent, where the expression of the query vector Q of each agent is Q=s i W Q ;

[0057] (2) Through the state information S of each agent i , classify the state information of each agent in the agent set including multiple teammate agents and multiple opponent agents, and obtain the teammate state information S X And the opponent's state information S Y , teammate status information S X The expression is: S X =eliminate(S,X,μ i ), eliminate(S,X,μ i ) means deleting the elements in the global state matrix S that do not belong to the teammate set X, and then deleting μ in the global state matrix S.i The elements in the column that are 0, μ i is the i-th row of the observable matrix μ; the opponent state information S Y The expression is: S Y =eliminate(S,Y,μ i ), eliminate(S,Y,μ i ) means deleting the elements in the global state matrix S that do not belong to the opponent set Y, and then deleting μ in the global state matrix S. i The elements listed as 0 in ; that is, delete the information of the objects that the i-th agent cannot observe.

[0058] (3) Teammate status information S X Calculate the key vector and value vector to obtain the first key vector K1 and the first value vector V1, and calculate the opponent state information S Y Calculate the key vector and value vector to obtain the second key vector K2 and the second value vector V2. The expression of the first key vector K1 is: K1 = S X W K(1) , W K(1) is the first key vector weight matrix, and the expression of the first value vector V1 is: V1=S X W V(1) , W V(1) is the first value vector weight matrix, and the expression of the second key vector K2 is: K2 = S Y W K(2) , W K(2) is the second key vector weight matrix, and the expression of the second value vector V2 is: V2 = S Y W V(2) , W V(2) is the second value vector weight matrix;

[0059] (4) The teammate status information S is obtained through the first key vector K1 and the first value vector V1 X Perform attention output analysis to obtain teammate attention output data Z X , through the second key vector K2 and the second value vector V2, the opponent state information S Y Perform attention output analysis to obtain the opponent's attention output data Z Y .

[0060] In another embodiment of the present disclosure, the teammate status information S is processed by the first key vector K1 and the first value vector V1. X Perform attention output analysis to obtain teammate attention output data Z X , through the second key vector K2 and the second value vector V2, the opponent state information S Y Perform attention output analysis to obtain the opponent's attention output data ZY The specific process of the steps includes the following steps:

[0061] (1) The teammate status information S is obtained through the first key vector K1 X Perform attention weight calculation to obtain teammate attention weight W1. The expression of teammate attention weight W1 is: The teammate status information S is processed by the teammate attention weight W1 and the first value vector V1 X Perform attention output analysis to obtain teammate attention output data Z X , teammate attention output data Z X The expression is:

[0062] (2) The second key vector K2 is used to match the hand state information S Y Perform attention weight calculation to obtain the opponent's attention weight W2. The expression of the opponent's attention weight W2 is: The opponent's state information S is obtained by the opponent's attention weight W2 and the second value vector V2 Y Perform attention output analysis to obtain the team's attention output data Z Y , team attention output data Z Y The expression is:

[0063] In another embodiment of the present disclosure, the specific process of executing the step of splicing teammate attention output data and opponent attention output data to obtain target spliced ​​output data includes the following steps:

[0064] (1) Output data Z of teammates’ attention through the preset weight matrix W′ X And the opponent's attention output data Z Y Splicing is performed to obtain the single classification attention mechanism output data M, where the expression of the single classification attention mechanism output data M is:

[0065] M=CA({i},s,μ i ;w Q ,W K(1) ,W V(1) ,W K(2) ,W V(2) ,w′)=[Z X ,Z Y ] T ·W′;

[0066] (2) Perform multi-layer splicing on the output data M of the single classification attention mechanism to obtain the output data P of the multi-head classification attention mechanism. The expression of the output data P of the multi-head classification attention mechanism is as follows:

[0067] in,

[0068] n h is the number of heads in the attention mechanism.

[0069] In the disclosed embodiment, a "classification attention mechanism" is introduced into the existing learning framework. By modifying the existing attention mechanism, opponents and teammates are respectively sent to different check matrices (K-key) and content matrices (V-value), realizing two thinking modes, and quickly realizing different responses to teammates and opponents, thereby improving learning efficiency.

[0070] In another embodiment of the present disclosure, a specific process of inputting the spliced ​​output data into a gated recurrent unit for data processing includes the following steps:

[0071] (1) The output data P of the multi-head classification attention mechanism is recorded as Get the target hidden state h of the gated recurrent unit t-1 , the target hidden state

[0072] (2) Output data z of the multi-head classification attention mechanism t And the target hidden state h t-1 Input the gated recurrent unit for data processing to obtain the recurrent unit output data Z GRU .

[0073] Among them, the output of the gated recurrent unit can be recorded as:

[0074]

[0075] The output of GRU (i.e. Z GRU ) Input the fully connected layer.

[0076] It's important to note that in multi-agent reinforcement learning models, the target hidden state of the gated recurrent unit is used to maintain and update memory information across time steps. At each time step, the gated recurrent unit receives the current input data and the previous hidden state and computes a new hidden state. These hidden states incorporate historical information about the input data, enabling the model to better memorize and predict data across consecutive time steps.

[0077] Specifically, at each time step, the current concatenated output data P is obtained. This concatenated output data is obtained by processing the global state matrix and observation matrix through a multi-head classification attention mechanism layer. The concatenated output data P contains the attention output information of teammates and opponents and is the input data for the current time step. The concatenated output data P is then input into the gated recurrent unit along with the previous hidden state. The gated recurrent unit calculates the new hidden state through three main steps: the update gate, the reset gate, and the calculation of the candidate hidden state. The update gate determines how much information from the previous time step to retain, the reset gate determines how much information from the previous time step to forget, and the candidate hidden state is calculated by jointly calculating the current input and the hidden state from the previous time step. Ultimately, the new hidden state is obtained through the weighted sum of the update gate and the candidate hidden state.

[0078] In another embodiment of the present disclosure, a specific process of inputting the output data of the recurrent unit into the fully connected layer for linear transformation processing includes the following steps:

[0079] (1) Extract the number of executable actions for each agent to obtain the number of executable actions A for each agent;

[0080] (2) Based on the number of executable actions A of each agent, the recurrent unit output data Z GRU Input the fully connected layer for linear transformation processing to obtain the Q value data of each agent. The Q value data of each agent includes the Q value of each action corresponding to each agent. The expression of the Q value data of each agent is: Z FC =W q ×Z GRUT +b, where Z FC is the Q value data of each agent, W q is the weight matrix of the fully connected layer. The dimension of the weight matrix of the fully connected layer is A×(d×n h ), b is the bias vector of the fully connected layer, and the dimension of the bias vector of the fully connected layer is A×1.

[0081] The fully connected layer controls the output dimension so that the output dimension of the final single Q-value network is aligned with the number of actions |A| that the agent can perform.

[0082] The intermediate process of the fully connected layer:

[0083] FC(Z MHCSA ; W, b) = W·Z GRUT +b

[0084]

[0085] Fully connected layer output:

[0086] ZFC ∈R |A|×1

[0087] Specifically, each agent's state and observation information is input into a multi-head classification attention mechanism layer. Attention output analysis is performed on the global state matrix and observation matrix to obtain teammate and opponent attention output data. Each agent's state and observation information is used to calculate a query vector using a preset query weight matrix. The query vector is then used to match the corresponding key and value vectors. Key and value vectors are calculated for teammates and opponents separately, ensuring that different types of agents are treated differently and thus receive different attention weights. This classification attention mechanism improves the accuracy and effectiveness of the attention mechanism by distinguishing between teammates and opponents, enabling agents to better understand and process relationships between different types of agents.

[0088] The teammate status information is analyzed for attention output using the first key vector and the first value vector to obtain teammate attention output data. Similarly, the opponent status information is analyzed for attention output using the second key vector and the second value vector to obtain opponent attention output data. In the specific implementation, the teammate status information is calculated using the first key vector and the value vector to obtain the teammate attention weights, which are then weighted and summed using these weights to obtain the final teammate attention output data. For opponent status information, the same method is used to calculate the opponent attention output data using the second key vector and the value vector. This multi-layered computation of the classified attention mechanism enables the intelligent agent to accurately pay attention to and process teammates and opponents separately, thereby comprehensively considering the relationship between collaboration and competition when making decisions, improving the rationality and effectiveness of decisions.

[0089] The teammate and opponent attention output data are concatenated to obtain the target concatenated output data. Specifically, the teammate and opponent attention output data are concatenated using a preset weight matrix to obtain the initial concatenated output data. These initial concatenated output data are then concatenated in multiple layers to obtain the target concatenated output data. This concatenation operation integrates the attention information of teammates and opponents, providing a unified input for subsequent gated recurrent unit processing. This approach not only improves data processing efficiency but also better integrates multiple types of information, providing a richer and more comprehensive input for the agent's Q-value calculation.

[0090] The concatenated output data is input into a gated recurrent unit for data processing, yielding the recurrent unit output data. The concatenated output data serves as the input to the gated recurrent unit, and the target hidden state of the gated recurrent unit is simultaneously obtained. Both the concatenated output data and the target hidden state are then input into the gated recurrent unit for processing. Through its gating mechanism and memory unit, the gated recurrent unit combines current input with historical information to generate the recurrent unit output data. The introduction of the gated recurrent unit enables the model to possess better memory and prediction capabilities when processing data at consecutive time steps, enabling more accurate calculation of the agent's Q-value data.

[0091] The recurrent unit output data is fed into a fully connected layer for linear transformation, yielding the Q-value data for each agent. Specifically, the recurrent unit output data is fed into a fully connected layer, which uses a linear transformation to map the recurrent unit output data to the Q-value for each action available to each agent. The design of the fully connected layer ensures that the output Q-value data has a clear representation of the action value, providing a direct basis for each agent's decision-making. This approach, through meticulous layer-by-layer calculation and processing, ultimately yields accurate Q-value data, ensuring that the agent can make optimal decisions when executing actions.

[0092] In practical applications, consider training robot soccer strategies. Imagine two robot soccer teams competing on a field. Each robot acts as an agent, making decisions based on the situation on the field. During the game, the global state matrix S represents the position, velocity, and other state information of all robot players, while the observation matrix μ represents each robot's observability of the other robots. To train the robot soccer team's strategy, the global state matrix and observation matrix are first input into the agent network. A multi-head classification attention mechanism layer calculates the state information of teammates and opponents, respectively, generating teammate attention output data and opponent attention output data. This data is then combined through a concatenation operation and fed into a gated recurrent unit for processing. Finally, a fully connected layer calculates the Q-value data corresponding to each robot.

[0093] For example, when a robot holds the ball, the system uses the agent network to calculate the Q-values ​​of multiple possible actions based on the current global state and observation matrix, and selects the optimal action to execute. If the action is a pass, the system considers the positions of teammates and the opponent's defensive strategy to select the optimal passing path. If the action is a shot, the system evaluates the success rate of the shot and executes it at the appropriate time. Similarly, when the robot is on defense, the system uses the opponent's state information to decide whether to advance to steal the ball or retreat to cover the position, thereby improving defensive effectiveness.

[0094] In another embodiment of the present disclosure, the specific process of inputting the Q value data of each agent into the mixing network for weighted mixing processing includes the following steps:

[0095] (1) Based on the preset action selection probability, the Q value data of each agent is input into the hybrid network to select the agent action, and the current execution action set is obtained. The current execution action set includes the current execution action of each agent;

[0096] (2) Based on the action selection probability, the agent action of multiple agents at the next moment is selected to obtain the action set to be executed at the next moment. The action set to be executed at the next moment includes the action to be executed at the next moment of each agent;

[0097] (3) Based on the current set of executed actions, each agent executes the next action of each agent to obtain the updated global state matrix and the total reward of each agent;

[0098] (4) The currently executed action set, the next moment executed action set, the updated global state, the current observation matrix, and the current global state matrix are stored in a preset experience pool;

[0099] (5) Sampling trajectory samples from the experience pool to obtain multiple trajectory samples, and performing a first-layer weighted mixing of the Q value data of each agent using the multiple trajectory sample data to obtain the initial mixed Q value of each agent;

[0100] (6) Perform a second-layer weighted mixing on the initial mixed Q value of each agent to obtain the joint Q value data, wherein the global state matrix is ​​updated to be used as the new global state matrix when the action selection step is executed next time.

[0101] Specifically, the Q-value data for each agent is first calculated. Through the aforementioned steps, the Q-value data for each available action corresponding to each agent has been obtained. These Q-value data represent the value of different actions in the current state. To select the current action for each agent, these Q-value data are input into the hybrid network. The hybrid network processes the Q-value of each agent based on the preset action selection probability and selects the action with the highest value. This action selection mechanism typically adopts an ε-greedy strategy, which randomly selects an action with a probability of ε and selects the action with the highest Q-value with a probability of 1-ε. This ensures that the new action space is explored while also leveraging existing experience to make the optimal choice.

[0102] The action selection probability is used to select the next action for multiple agents. Each agent selects the next action based on its current Q-value and action selection probability, resulting in a set of actions to be executed at the next moment. This set of actions includes the possible actions that each agent can perform at the next moment. To simulate the collaborative and competitive behavior of multiple agents in a real-world environment, the current action set and the next action set are combined for processing.

[0103] After executing the actions in the current action set, each agent obtains an updated global state matrix and total reward based on its actions and environmental feedback. The global state matrix contains the state information of all agents after executing the actions, while the total reward reflects the feedback each agent receives from executing the action in the current state. Specifically, the current action set, the next action set, the updated global state, the current observation matrix, and the current global state matrix are all stored in the experience pool. The data in the experience pool is used to train the model. By sampling trajectory samples from the experience pool, multiple trajectory samples are obtained. The trajectory samples contain the state, action, and reward information of the agent at different time steps. By processing these trajectory sample data, the Q value data of each agent can be weighted mixed.

[0104] First, the Q-value data of each agent is initially processed through the first layer of weighted mixing to obtain the initial mixed Q-value of each agent. Then, the initial mixed Q-value is further processed through the second layer of weighted mixing to obtain the joint Q-value data. The joint Q-value data represents the overall value of the actions jointly performed by multiple agents in the current state.

[0105] The updated global state matrix is ​​used as the new global state input for the next action selection step. Through this iterative training process, the model can continuously optimize its decisions and strategies, improving its adaptability and performance in complex and changing environments.

[0106] In practical applications, consider training strategies for a robot soccer match. Imagine two robot soccer teams competing on a field. Each robot, acting as an intelligent agent, makes decisions based on the situation on the field. During the game, each robot must not only coordinate with its teammates but also employ defensive strategies against its opponents. To train the robot soccer team's strategy, the Q-value data for each robot's current state is first calculated using the aforementioned steps. Then, based on preset action selection probabilities, this Q-value data is input into a hybrid network to select the set of actions to be performed. For example, when a robot has the ball, it decides whether to pass, shoot, or dribble past its opponent based on its current Q-value data and action selection probabilities.

[0107] After selecting an action, the robot executes it and obtains an updated global state matrix and total reward based on environmental feedback. For example, when the robot selects a pass action, it records the new position and state after the pass and assigns a total reward based on whether the pass was successful or not. This data is then stored in the experience pool and used as samples for subsequent training.

[0108] Through repeated rounds of training, trajectory samples are randomly sampled from the experience pool and weighted blending is performed on the Q-value data of each agent. First, an initial mixed Q-value is calculated, followed by a second layer of weighted blending to obtain a joint Q-value. For example, by analyzing multiple training samples, optimal strategies for passing, shooting, and defending in different game situations can be learned, thereby improving overall performance and win rate.

[0109] This approach enables robot soccer teams to make swift and effective decisions in complex and ever-changing competitive environments, effectively coordinating with teammates on offense while precisely defending against opponents' moves, thereby improving their overall strategic capabilities and winning rate. By continuously optimizing models and training methods, the intelligence level and collaborative efficiency of multi-agent systems are further enhanced. This approach is not only applicable to robot soccer competitions but can also be widely applied to other multi-agent collaborative scenarios, such as autonomous vehicle fleets and multi-robot collaboration in intelligent manufacturing.

[0110] It should be noted that when performing a first-layer weighted mixing of the Q-value data of each agent using multiple trajectory sample data to obtain the initial mixed Q-value of each agent; and performing a second-layer weighted mixing of the initial mixed Q-value of each agent to obtain the joint Q-value data, the specific implementation steps are as follows:

[0111] Get the Q value data of each agent, the expression is:

[0112]

[0113] Where τ is the trajectory data of the ith agent, a is the action of the ith agent, N is the number of agents, and A is the number of executable actions for each agent;

[0114] The Q value data of each agent is mixed with the first layer of weighted data through multiple trajectory sample data, assuming that the state action value Q of each agent is i , joint Q value data Q tot satisfy:

[0115]

[0116] Then the global optimal joint action can be obtained by each agent greedily selecting the action with the highest value Q:

[0117]

[0118] Therefore, when calculating the global value, the state action value of each agent is multi-layered weighted mixing, and the global state matrix S is used to generate the weight matrix W l (s) and bias matrix b l (s); where l represents the number of layers in the multi-layer weighted hybrid network.

[0119] For the two-layer mixing case (l=1 or 2), the first layer output y1 is

[0120]

[0121] Where ELU is the exponential linear unit.

[0122] The final mixed joint Q value data Q tot for:

[0123] Q tot =W2(s)y1+b2(s).

[0124] In another embodiment of the present disclosure, the specific process of executing step S1023 includes the following steps:

[0125] (1) Calculate the TD target value for the joint Q value data to obtain the TD target value;

[0126] (2) Calculate the loss value of the TD target value through the loss function to obtain the loss value;

[0127] (3) When the loss value meets the preset threshold, the target network parameters are obtained and the network parameters are updated according to the target network parameters.

[0128] In another embodiment of the present disclosure, when the number of times the action selection step is executed reaches a preset network update frequency, the target network parameters are updated to a preset target multi-agent reinforcement learning model, wherein the target multi-agent reinforcement learning model has the same network structure as the multi-agent reinforcement learning model.

[0129] Specifically, for the joint Q value data Q tot Calculate the TD target value to obtain the TD target value, where the TD target value is the timing difference value. The calculation formula of the TD target value is as follows:

[0130]

[0131] τ′ is the trajectory data of the ith agent at the next moment, a′ is the action of the ith agent at the next moment, s′ is the updated global state matrix of the ith agent at the next moment, θ -are the target network parameters of the target multi-agent reinforcement learning model, is the TD target value, r is the total reward of each agent, and γ is the discount factor, which ranges from 0 to 1 and is used to measure the importance of future rewards; It is used to represent the maximum value operation for all possible next actions a′;

[0132] Furthermore, the loss value of the TD target value is calculated by the loss function to obtain the loss value, wherein the expression of the loss function is as follows:

[0133]

[0134] Where L(θ) is the loss function, θ is the network parameter of the multi-agent reinforcement learning model, b is the number of trajectory samples, τ is the current trajectory data of the i-th agent, a is the current execution action of the i-th agent, and s is the global state matrix of the i-th agent.

[0135] The following is an example to illustrate the entire algorithm process:

[0136] 1. Initialize the parameters of the agent network and the hybrid network, and copy these two parameters to the asynchronously updated agent network and hybrid network. Initialize the experience pool D with a capacity of M, a total number of iterations T, a maximum number of simulation steps MAX, and an update frequency p for the two asynchronously updated networks (the multi-agent reinforcement learning model and the target multi-agent reinforcement learning model).

[0137] 2. The first layer of loop, in the loop from time t = 1 to T:

[0138] 1) Initialize the environment;

[0139] 2) Obtain the global state S, the observation matrix μ, the possible actions of each agent, and the reward R;

[0140] 3. The second loop, in the loop with step number step = 1 to MAX:

[0141] 1) Each agent calculates the Q-value of each action through the agent network;

[0142] 2) Selecting the action of each agent in an ∈-greedy manner (randomly selecting an action with probability ∈, and selecting the action with the largest Q-value with probability (1-∈));

[0143] 3) S, S next , the observation matrix μ, the current and next possible actions of each agent, the reward R, and the action set a selected by all agents are stored in the experience pool D;

[0144] 4) Randomly sample multiple batches of trajectory samples from the experience pool;

[0145] 5) Update parameters in the same way as DQN:

[0146]

[0147] 6) Let S = S next , the action available at the current moment = the action available at the next moment;

[0148] 7) If t can divide p at this time, copy the network parameters to the asynchronously updated network parameters.

[0149] In the disclosed embodiment, the stability and reliability of the strategy during the training process can be ensured by regularly updating the target network parameters. The model will not fluctuate dramatically due to the parameter updates at each step, making the training process smoother and the ultimately learned strategy more stable and reliable. The target multi-agent reinforcement learning model can effectively reduce high variance and divergence during training. Since the Q value remains unchanged for a certain period of time, the training target of the model is more stable, avoiding the uncertainty and oscillation caused by frequent updates. The introduction of the target multi-agent reinforcement learning model can improve the convergence speed of the model. Through stable target Q value evaluation, the model can find the optimal strategy more quickly, reduce training time, and improve efficiency.

[0150] Based on the same technical concept, the embodiment of the present application also provides a multi-agent reinforcement learning method, including:

[0151] (1) Input the obtained global state matrix and observation matrix into the multi-agent reinforcement learning model to obtain joint Q value data;

[0152] (2) Constructing an agent training strategy based on the joint Q-value data.

[0153] Based on the same technical concept, the embodiment of the present application further provides a multi-agent reinforcement learning device, which is characterized by comprising:

[0154] The input module is used to input the acquired global state matrix and observation matrix into the multi-agent reinforcement learning model to obtain joint Q value data;

[0155] A construction module is used to construct an intelligent agent training strategy based on the joint Q value data.

[0156] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented through hardware or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present disclosure.

[0157] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present disclosure.

[0158] Those skilled in the art will appreciate that the modules in the devices of the embodiments may be distributed in the devices of the embodiments as described in the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module or further split into multiple submodules.

[0159] The serial numbers of the above-mentioned embodiments of the present disclosure are for description only and do not represent the advantages or disadvantages of the embodiments.

[0160] Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A multi-agent reinforcement learning model training method, characterized in that: include: Pre-building a multi-agent reinforcement learning model, wherein the multi-agent reinforcement learning model includes: an agent network and a hybrid network; The multi-agent reinforcement learning model is trained for multiple rounds of iterations. During each round of iterative training, the following action selection steps are performed multiple times: Inputting the current global state matrix and the current observation matrix into the agent network for attention output analysis and agent Q value calculation to obtain Q value data for each agent, wherein the Q value data for each agent includes the Q value of each action corresponding to each agent, and the Q value of each action is used to represent the action value of each action; Inputting the Q value data of each agent into the hybrid network for weighted mixing processing to obtain joint Q value data, which is used to represent the action value of multiple agents when taking joint actions under the current global state matrix; Based on the joint Q value data, a preset loss function is used to update the network parameters.

2. The method according to claim 1, wherein The agent network includes: a matrix analysis layer, a multi-head classification attention mechanism layer, a gated recurrent unit, and a fully connected layer. The global state matrix and the observation matrix are input into the agent network for attention output analysis and agent Q value calculation to obtain the Q value data of each agent, including: The global state matrix S and the observation matrix μ are input into the matrix analysis layer to perform state analysis on each agent, and the state information S of each agent is obtained. i ; The state information S of each agent i Input the multi-head classification attention mechanism layer in turn to perform attention output analysis on the global state matrix S and the observation matrix μ to obtain the teammate attention output data Z X And the opponent's attention output data Z Y ; Output data Z for the teammate's attention X And the opponent attention output data Z Y Splice and pass through a layer of network to obtain the output data of a single classification attention mechanism, and splice all the classification attention mechanism outputs to obtain the output data of the multi-head classification attention mechanism; Inputting the output data of the multi-head classification attention mechanism into the gated recurrent unit for data processing to obtain recurrent unit output data; The output data of the cyclic unit is input into the fully connected layer for linear transformation processing to obtain the Q value data of each intelligent agent, wherein the Q value data of each intelligent agent includes the Q value of each movable action corresponding to each intelligent agent.

3. The method according to claim 2, wherein The state information S of each intelligent agent i Input the multi-head classification attention mechanism layer in turn to perform attention output analysis on the global state matrix S and the observation matrix μ to obtain the teammate attention output data Z X And the opponent's attention output data Z Y ,include: The state information S of each agent i Input the multi-head classification attention mechanism layer in sequence, through the preset query weight matrix W Q For each agent's state information S i Perform query vector calculation to obtain the query vector Q of each agent, where the expression of the query vector Q of each agent is Q=s i W Q ; Through the state information S of each agent i , classify the state information of each agent in the agent set including multiple teammate agents and multiple opponent agents, and obtain the teammate state information S X And the opponent's state information S Y , the teammate status information S X The expression is: S X =eliminate(S,X,μ i ), eliminate(S,X,μ i ) means deleting the elements in the global state matrix S that do not belong to the teammate set X, and then deleting the elements in the i-th row of the observable matrix μ of each agent in the global state matrix S, μ i is the i-th row of the observable matrix μ; the opponent state information S Y The expression is: S Y =eliminate(S,Y,μ i ), eliminate(S,Y,μ i ) means deleting the elements in the global state matrix S that do not belong to the opponent set Y, and then deleting the observation information μ of each agent in the global state matrix S i The elements listed as 0 in The teammate status information S X Calculate the key vector and value vector to obtain the first key vector K1 and the first value vector V1, and calculate the opponent state information S Y Calculate the key vector and value vector to obtain the second key vector K2 and the second value vector V2. The expression of the first key vector K1 is: K1 = S X W K(1) , W K(1) is the first key vector weight matrix, and the expression of the first value vector V1 is: V1=S X W V(1) , W V(1) is the first value vector weight matrix, and the expression of the second key vector K2 is: K2 = S Y W K(2) , W K(2) is the second key vector weight matrix, and the expression of the second value vector V2 is: V2 = S Y W V(2) , W V(2) is the second value vector weight matrix; The teammate status information S is obtained by using the first key vector K1 and the first value vector V1. X Perform attention output analysis to obtain teammate attention output data Z X , the opponent state information S is processed by the second key vector K2 and the second value vector V2 Y Perform attention output analysis to obtain the opponent's attention output data Z Y .

4. The method according to claim 3, wherein The teammate status information S is processed by the first key vector K1 and the first value vector V1. X Perform attention output analysis to obtain teammate attention output data Z X , the opponent state information S is processed by the second key vector K2 and the second value vector V2 Y Perform attention output analysis to obtain the opponent's attention output data Z Y ,include: The teammate status information S is mapped to the teammate status information S by the first key vector K1. X Perform attention weight calculation to obtain teammate attention weight W1, the expression of teammate attention weight W1 is: The teammate status information S is analyzed by the teammate attention weight W1 and the first value vector V1. X Perform attention output analysis to obtain teammate attention output data Z X , teammate attention output data Z X The expression is: The team state information S is processed by the second key vector K2 Y Perform attention weight calculation to obtain the opponent's attention weight W2, and the expression of the opponent's attention weight W2 is: The opponent state information S is analyzed by the opponent attention weight W2 and the second value vector V2. Y Perform attention output analysis to obtain the team's attention output data Z Y , team attention output data Z Y The expression is:

5. The method according to claim 2, wherein The teammate's attention output data Z X And the opponent attention output data Z Y Splice and pass through a layer of network to obtain the output data of a single classification attention mechanism, and splice all classification attention mechanism outputs to obtain the output data of the multi-head classification attention mechanism, including: The teammate attention output data Z is processed by the preset weight matrix W′ X And the opponent attention output data Z Y Splicing is performed to obtain the single classification attention mechanism output data M, wherein the expression of the single classification attention mechanism output data M is: M=CA({i},s,μ i ;IN Q ,IN K(1) ,IN V(1) ,IN K(2) ,IN V(2) ,W′)=[Z X ,WITH Y ] T ·IN'; The single classification attention mechanism output data M is multi-layered spliced ​​to obtain the multi-head classification attention mechanism output data P, wherein the expression of the multi-head classification attention mechanism output data P is as follows: Among them, n h is the number of heads in the attention mechanism.

6. The method according to claim 5, wherein The step of inputting the output data of the multi-head classification attention mechanism into the gated recurrent unit for data processing to obtain recurrent unit output data includes: The output data P of the multi-head classification attention mechanism is recorded as Get the target hidden state h of the gated recurrent unit t-1 , the target hidden state The multi-head classification attention mechanism outputs data z t And the target hidden state h t-1 Input the gated cycle unit for data processing to obtain the cycle unit output data Z GRU .

7. The method according to claim 6, wherein The step of inputting the output data of the cyclic unit into the fully connected layer for linear transformation processing to obtain the Q value data of each agent includes: Extract the number of executable actions for each agent to obtain the number of executable actions A for each agent; Based on the number of executable actions A of each agent, the recurrent unit output data Z GRU The input is linearly transformed into the fully connected layer to obtain the Q value data of each agent. The Q value data of each agent includes the Q value of each action corresponding to each agent. The expression of the Q value data of each agent is: Among them, Z FC is the Q value data of each agent, W q is the weight matrix of the fully connected layer, and the dimension of the weight matrix of the fully connected layer is A×(d×n h ), b is the bias vector of the fully connected layer, and the dimension of the bias vector of the fully connected layer is A×1.

8. The method according to claim 7, wherein The step of inputting the Q value data of each agent into the hybrid network for weighted hybrid processing to obtain joint Q value data includes: Based on a preset action selection probability, the Q value data of each agent is input into the hybrid network to select an agent action, thereby obtaining a currently executed action set, wherein the currently executed action set includes the currently executed action of each agent; Selecting the next-moment action of multiple agents based on the action selection probability to obtain a set of actions to be executed at the next moment, wherein the set of actions to be executed at the next moment includes the next-moment action to be executed by each agent; Based on the current set of actions, each agent executes the next action to be executed by each agent, and obtains an updated global state matrix and a total reward for each agent; Storing the currently executed action set, the next moment executed action set, the updated global state, the current observation matrix, and the current global state matrix in a preset experience pool; Sampling trajectory samples from the experience pool to obtain multiple trajectory samples, and performing a first-layer weighted mixing of the Q value data of each agent using the multiple trajectory sample data to obtain an initial mixed Q value for each agent; The initial mixed Q value of each agent is subjected to a second layer of weighted mixing to obtain the joint Q value data, wherein the updated global state matrix is ​​used as the new global state matrix when the action selection step is performed next time.

9. The method according to claim 8, wherein The network parameter update based on the joint Q value data is performed using a preset loss function, including: Calculating a TD target value on the combined Q value data to obtain a TD target value; Calculating the loss value of the TD target value using the loss function to obtain a loss value; When the loss value meets a preset threshold, the target network parameters are obtained and the network parameters are updated according to the target network parameters.

10. The method according to claim 9, wherein When the number of times the action selection step is executed reaches a preset network update frequency, the target network parameters are updated to a preset target multi-agent reinforcement learning model, wherein the target multi-agent reinforcement learning model has the same network structure as the multi-agent reinforcement learning model.

11. A multi-agent reinforcement learning method, comprising: The obtained global state matrix and observation matrix are input into the multi-agent reinforcement learning model to obtain the joint Q value data; Constructing an agent training strategy based on the joint Q-value data; Wherein, the multi-agent reinforcement learning model is trained based on the multi-agent reinforcement learning model training method described in any one of claims 1 to 10.

12. A multi-agent reinforcement learning device, characterized in that: include: The input module is used to input the acquired global state matrix and observation matrix into the multi-agent reinforcement learning model to obtain joint Q value data; A construction module is used to construct an intelligent agent training strategy based on the joint Q value data.

Images