A value decomposition multi-agent reinforcement learning training method using attention networks

Abstract

The present invention provides a value decomposition multi-agent reinforcement learning training method using an attention network, including experience collection using an exploration strategy and calculation of a joint action value function. Two methods are used to improve the effect when calculating the joint action value function: using an attention network and a separated value decomposition structure. The attention network uses the global state as input, so that the network can pay attention to the important parts of the global state and calculate the weights more accurately. The separated value decomposition structure uses a two-layer weighted network, and each layer of the network uses a different global state encoding to reduce the difficulty of super network learning. The independent action value function of each agent is fused using a super network, and finally the joint action value function is output. The attention network and the separated value decomposition structure used in the present invention effectively improve the learning efficiency, accelerate the convergence of the model, and improve the effect of the multi-agent collaboration strategy.

Description

Technical Field

[0001] The present invention relates to a reinforcement learning training method, in particular to a value decomposition multi-agent reinforcement learning training method using an attention network. Background Art

[0002] Reinforcement Learning (RL) is a subfield of machine learning, mainly used to solve sequential decision-making problems. Agents interact with the environment, explore, and optimize strategies to maximize rewards or achieve a given goal. In multi-agent reinforcement learning, multiple agents need to work together to achieve the same goal. For this reason, multi-agent reinforcement learning is closer to real-world problems and has more practical significance.

[0003] In a multi-agent environment, there are some problems that do not exist in a single-agent environment, such as environmental instability, dimensionality explosion, and credit allocation problems. Environmental instability means that when the strategies of other agents change, the environment faced by a single agent will also change, affecting the convergence of the algorithm. Dimensional explosion means that the joint action space increases exponentially with the increase in the number of agents, increasing the difficulty of learning. The credit allocation problem means that the environment only gives one reward, which is the result of all agents performing joint actions, and each agent needs to know how much its action contributes to the whole, so as to optimize its own strategy.

[0004] Q-learning is a classic single-agent reinforcement learning method that optimizes strategies by estimating the value function (the cumulative rewards that can be obtained in the current state). Value decomposition is a method in multi-agent reinforcement learning that decomposes the value function of the joint action into a function of each agent's value function, allowing the network to automatically learn credit allocation and improve learning efficiency and effectiveness.

[0005] Centralized learning and distributed execution is a common solution paradigm. During training, the network can use global information as additional input. During execution, it cannot obtain global information and can only use its own state to make decisions.

[0006] QMIX is a typical multi-agent reinforcement learning algorithm based on value decomposition, which uses the weighted sum of each independent value function to generate a joint action value function, and the weight is generated using the global state. Summary of the invention

[0007] Purpose of the invention: The technical problem to be solved by the present invention is to provide a value decomposition multi-agent reinforcement learning training method using an attention network in view of the shortcomings of the prior art.

[0008] In order to solve the above technical problems, the present invention discloses a value decomposition multi-agent reinforcement learning training method using an attention network, comprising the following steps:

[0009] Step 1, initialize the environment and all agent states; including: initializing the agent network and environment, initializing the value function network of each agent and the corresponding target network, initializing the network used to calculate the joint action value function, initializing the environment, and obtaining the initial state;

[0010] Step 2: The agent uses the exploration strategy to interact with the environment and saves the experience into the experience cache pool;

[0011] Step 3: Execute the training of the agent network: that is, sample from the experience buffer pool and calculate the loss function;

[0012] Step 4: Loop through steps 2 and 3 until the end. Finally, the agent greedy strategy is obtained according to the agent value function and used as the output strategy to complete the multi-agent reinforcement learning training.

[0013] In the present invention, step 2 comprises:

[0014] Step 2-1: The agent uses the current observation and the action at the previous moment as the input of the value function network, and outputs the value function of all actions at the current moment, that is, the cumulative reward obtained by each action;

[0015] Step 2-2: The agent decides to take a random action or the action with the maximum value function based on the current exploration rate;

[0016] In step 2-3, the agent obtains rewards and new observations from the environment, and saves the current observation, current action, reward, and new observation as one experience into the experience cache pool.

[0017] In the present invention, step 2-1 comprises:

[0018] Each agent i is based on the current observation and the action at the previous moment Get the value function of all current actions; The value function network consists of 3 layers, a linear layer, a recurrent neural network layer and the last linear layer; The recurrent neural network (reference: Chung J, Gulcehre C, Cho KH, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling [J]. arXiv preprint arXiv: 1412.3555, 2014.) uses the hidden state at the current moment as input and outputs the encoding at the current moment and the hidden state at the next moment; The process is expressed as follows:

[0019]

[0020]

[0021]

[0022] Where MLP stands for multi-layer perceptron, RNN stands for recurrent neural network, and is a temporary variable, represents the hidden state at the current moment, represents the hidden state at the next moment, Q i Represents the action-value function of the agent.

[0023] In the present invention, the step 2-2 comprises:

[0024] The agent decides the action to take based on the current value function. In the experience accumulation stage, the agent adopts an exploration strategy rather than simply executing the optimal strategy. The agent maintains an exploration rate ε. The exploration strategy means that the agent will execute random actions with a probability of ε and execute the optimal action with a probability of 1-ε, that is, the action with the largest value function. The exploration rate decreases with time but is not zero, that is, the probability of executing random actions is getting smaller and smaller, but it is not zero.

[0025] The steps 2-3 include:

[0026] All agents maintain an experience buffer pool D. When agent i makes an action, the agent receives feedback from the environment, including the reward r shared by all agents. t and new observations The agent saves the current observation, current action, reward, and new observation as experience into the experience buffer pool to prepare for subsequent learning; if the experience buffer pool is full, the earliest experience is discarded.

[0027] In the present invention, step 3 comprises:

[0028] Step 3-1, sampling from the experience buffer pool, and using the agent value function network to obtain the action value function of the agent at the current moment;

[0029] Step 3-2, take the global state as an additional input and use the attention network (reference: Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [J]. Advances in neural information processing systems, 2017, 30.) to calculate the global state encoding;

[0030] Step 3-3, use the hypernetwork to calculate the value decomposition weights and biases, and the input of the hypernetwork is the global state encoding;

[0031] Step 3-4, use the separated value decomposition to calculate the value function of the joint actions of all agents. The value decomposition weights and biases are the outputs of step 3-3;

[0032] Step 3-5, use the target network to calculate the loss of the joint action value function and optimize the agent value function network.

[0033] In the present invention, step 3-1 comprises:

[0034] During each training, N experience replays are sampled from the experience cache pool, that is, a sequence including the current observation, action, reward, and next observation. The agent value function network is used to calculate the cumulative reward that can be obtained by executing the action under the current observation, that is, the value function.

[0035] In the present invention, step 3-2 comprises:

[0036] Use the attention network to encode the global state; the input of the attention network is the global state s t , the attention network is calculated as:

[0037]

[0038]

[0039]

[0040] Among them, LF represents the linear function, Attention represents the attention network, and Q t ,K t ,V t is the input matrix of the attention network, SF t represents the global state encoding, Represents the final global state encoding; the global state s t After the linear function, we get Divided into multiple parts, each part Using linear function alone to calculate the attention mechanism and Matrix, the calculation method of the attention mechanism is:

[0041]

[0042] Among them, exp represents the exponential function, Q, K, V are the attention network input matrices, and f is the dot multiplication function used to calculate K i The similarity with all Q, the denominator is all K j The sum of similarities with all Q is used for normalization; the attention network outputs the final global state encoding

[0043] In the present invention, step 3-3 comprises:

[0044] Use the hypernetwork to calculate the value decomposition weights and biases, and the input is the global state encoding obtained in step 3-2; the hypernetwork consists of two linear layers, and the global state encoding Divided into two parts and The values ​​in each part are concatenated together as the input to the hypernetwork; this process is:

[0045]

[0046]

[0047] Where LF represents the hypernetwork, which is a linear function, and w 1 and b 1 denote the value decomposition weights and biases generated by the first layer of the hypernetwork, w 2 and b 2 Represents the value decomposition weights and biases generated by the second layer hypernetwork.

[0048] In the present invention, steps 3-4 include:

[0049] The joint action value function is calculated using the separated value decomposition structure. The input is the weights and biases obtained in step 3-3. The final joint action value function is the independent value function Q of each agent. i The output of the fusion of weights w and bias b obtained using the hypernetwork; this process is:

[0050] h=Q i *w 1 +b 1

[0051] Q tot =h*w 2 +b 2

[0052] Where Q tot Represents the value function of the joint actions of all agents, and h is a temporary variable.

[0053] In the present invention, steps 3-5 include:

[0054] Use TD(λ) to calculate (reference: Sutton RS, Barto A G. Reinforcement learning: An introduction [M]. MIT press, 2018.) the expected value y of the joint action value function tot Then the loss function for:

[0055]

[0056] Where θ represents the network parameters of the agent value function, E represents the mathematical expectation, D represents the experience cache, Q tot represents the joint action-value function.

[0057] Beneficial effects:

[0058] The final model obtained by using the multi-agent reinforcement learning training method can better control multiple agents. Each agent can cooperate with other agents based on its own limited information to achieve the same goal. Compared with existing methods, it can more efficiently utilize global information, more accurately measure the contribution of each agent to the entire system, improve the effectiveness of cooperation strategies between agents, and enhance the performance of multiple agents in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0060] Figure 1 It is a flow chart of the present invention.

[0061] Figure 2 This is a structural diagram of the model of the invention.

[0062] Figure 3 This is the structure diagram of the attention network. DETAILED DESCRIPTION

[0063] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above-mentioned beneficial effects or other advantages of the present invention will become more clear.

[0064] The present invention discloses a value decomposition multi-agent reinforcement learning training method using an attention network. The method is applied to solve sequential decision problems in complex scenarios and improve the efficiency of collaboration among multiple agents. The method comprises the following steps:

[0065] Step 1: Initialize the environment and all agent states;

[0066] Step 2: The agent uses the exploration strategy to interact with the environment and saves the experience into the experience cache pool:

[0067] Step 2-1: The agent uses the current observation and the action at the previous moment as the input of the value function network, and outputs the value function of all actions at the current moment, that is, the cumulative reward that can be obtained for each action;

[0068] Step 2-2: The agent decides to take a random action or the action with the maximum value function based on the current exploration rate;

[0069] Step 2-3, the agent gets the reward and new observation from the environment, and saves the current observation, current action, reward, and new observation as one experience into the experience cache pool;

[0070] Step 3: Execute the training of the agent network:

[0071] Step 3-1, sample from the experience buffer pool and use the agent value function network to obtain the action value function of the agent at the current moment;

[0072] Step 3-2, take the global state as an additional input and use the attention network to calculate the global state encoding;

[0073] Step 3-3, use the hypernetwork to calculate the value decomposition weights and biases, and the input of the hypernetwork is the global state encoding;

[0074] Step 3-4, use the separated value decomposition to calculate the value function of the joint actions of all agents. The value decomposition weights and biases are the outputs of step 3-3;

[0075] Step 3-5, use the target network to calculate the loss of the joint action value function and optimize the agent value function network.

[0076] Step 4: Loop through steps 2 and 3 until the end. Finally, the agent greedy strategy is obtained according to the agent value function as the output strategy.

[0077] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0078] The present invention discloses a value decomposition multi-agent reinforcement learning training method using an attention network, such as Figure 1As shown in Figure 1, it includes: first initializing the model and environment, including initializing the agent value function network, initializing the environment, and initializing the experience buffer pool. After that, the agent interacts with the environment. This process can be described as the agent first obtaining the observations given by the environment, and then based on the current observations and historical information encoding Get the value function of all current actions, and then choose to perform random actions or greedy actions according to the exploration rate ∈. The environment receives the action of the agent and executes it, and the next observation and return r t It is returned to the agent, and the agent stores the current observation, action, reward, and next observation in the experience pool. After the experience collection is completed, the training phase begins: first randomly sample from the experience buffer pool, calculate the action value function of each agent, then use the attention network to calculate the value decomposition weight, use the weight to calculate the weighted sum of the agent value function as the joint action value function, calculate the expected reward through the target network and minimize the loss to update the network parameters.

[0079] The structure of the model is as follows Figure 2 As shown, it consists of two parts: the agent value function network and the hybrid network. Figure 2 The right side of the figure shows the detailed structure of the value function network, and the left side shows the detailed structure of the hybrid network. The input of the value function network is the current observation of the agent. and the action at the previous moment The value function network structure is divided into three layers: two linear layers (MLP) and one recurrent neural network (RNN), where the recurrent neural network is used to encode the hidden state at the current moment Get the hidden state at the next moment The agent gets the value function Q at the current moment i Then execute the exploration strategy, select the action to be executed according to the current exploration rate ∈, and get the value function Q of the corresponding action i .

[0080] The input of the hybrid network is the action value function of all agents and the global state s t . Global state s t The global state encoding is obtained by encoding using the attention network. The global state encoding is divided into two parts, which are used as the input of the two-layer hypernetwork to generate value decomposition weights and biases. Finally, all agent action value functions are fused to obtain the final joint action value function Q tot . Figure 3 The detailed structure of the attention network is shown, and the input is the global state encoding obtained using a linear function The code is divided into several parts: Until Each part calculates the attention network input matrix separately, calculates the attention weight, and finally uses the dot multiplication (Dot) method to calculate the output global state encoding It includes Until by As an example, first use the linear function to calculate the attention network input matrix and Then the attention weight α1 is obtained according to the attention calculation formula, and finally the output is calculated using dot multiplication (Dot)

[0081] In this example, the value decomposition multi-agent reinforcement learning training method using the attention network specifically includes:

[0082] Step 1: Initialize the agent network and environment; initialize the value function network of each agent and the corresponding target network; initialize the network used to calculate the joint action value function; initialize the environment and get the initial state

[0083] Step 2: The agent executes the exploration strategy, collects experience, and saves the experience into the experience buffer pool. This step includes:

[0084] Step 2-1: Each agent observes the current and the action at the previous moment Get the value function of all current actions. The value function network consists of 3 layers, a linear layer, a recurrent neural network layer and the last linear layer. The recurrent neural network uses the hidden state of the previous moment as input and outputs the encoding of the current moment and the hidden state of the next moment. This process can be expressed as:

[0085]

[0086]

[0087]

[0088] Where MLP stands for multi-layer perceptron, RNN stands for recurrent neural network, and is a temporary variable, represents the hidden state at the current moment, represents the hidden state at the next moment, Q i Represents the action-value function of the agent.

[0089] Step 2-2: The agent decides the action to take based on the current value function. In the experience accumulation stage, the agent adopts an exploration strategy rather than simply executing the optimal strategy. The agent maintains an exploration rate ε. The exploration strategy means that the agent will perform random actions with a probability of ε and perform the optimal action with a probability of 1-ε, that is, the strategy with the largest value function. The exploration rate will continue to decrease with time but will not reach 0, that is, the probability of performing random actions will become smaller and smaller, but will not be zero.

[0090] Step 2-3: All agents maintain an experience buffer pool D. Every time an agent makes an action, it receives feedback from the environment, including a reward r. t and new observations The agent will save the current observation, current action, reward, and new observation as experience into the experience buffer pool to prepare for subsequent learning. The experience buffer pool has a certain size. If it is full, the earliest experience will be discarded.

[0091] Step 3: Perform agent training, i.e. sample from the experience buffer pool and calculate the loss function. This includes:

[0092] Step 3-1: Each time you train, you need to sample N experience replays from the experience cache pool, which is a sequence of current observation, action, reward, and next observation. Use the agent value function network to calculate the cumulative reward that can be obtained by executing the action under the current observation, that is, the value function.

[0093] Step 3-2: Use the attention network to calculate the value decomposition weights. The input of the attention network is the global state s t , the calculation method of the attention network can be expressed as:

[0094]

[0095]

[0096]

[0097] Among them, LF represents the linear function, Attention represents the attention network, and Q t ,K t ,V t is the input matrix of the attention network, SF t represents the global state encoding, Represents the final global state encoding; the global state s t After the linear function, we get Divided into multiple parts, each part Using linear function alone to calculate the attention mechanism and Matrix, the calculation method of the attention mechanism is:

[0098]

[0099] Where exp represents the exponential function and f is the dot multiplication function used to calculate K i The similarity with all Q, the denominator is all K j The sum of similarities with all Q is used for normalization; the attention network outputs the final global state encoding

[0100] Step 3-3: Use the hypernetwork to calculate the value decomposition weights and biases, and input the global state encoding obtained in step 3-2. The hypernetwork consists of a linear layer, and the global state encoding is divided into two parts. The values ​​in each part are concatenated together as the input of the hypernetwork. This process can be expressed as:

[0101]

[0102]

[0103] Where LF represents the hypernetwork, which is a linear function, and w 1 and b 1 denote the value decomposition weights and biases generated by the first layer of the hypernetwork, w 2 and b 2 Represents the value decomposition weights and biases generated by the second layer hypernetwork.

[0104] Step 3-4: Use the separated value decomposition structure to calculate the joint action value function, with the input being the weights and biases obtained in step 3-3. The final joint action value function is the output of the fusion of the weights and biases obtained by each independent agent value function using the hypernetwork. This process can be expressed as:

[0105] h=Q i *w 1 +b 1

[0106] Q tot =h*w 2 +b 2

[0107] Where Q tot Represents the value function of the joint actions of all agents, and h is a temporary variable.

[0108] Step 3-5: Use TD(λ) to calculate the expected value y of the joint action-value function tot The loss function can be expressed as:

[0109]

[0110] Where θ represents the agent value function network parameters, D represents the experience cache, Q tot represents the joint action-value function.

[0111] Step 4: Loop through steps 2 and 3 until the termination condition is reached and output the final strategy evaluation result.

[0112] Example 2

[0113] The value decomposition multi-agent reinforcement learning training method using the attention network of the present invention can be applied to multiple fields such as robot control, traffic coordination, and manufacturing control.

[0114] The value decomposition multi-agent reinforcement learning training method using the attention network described in the present invention is used for the control of the robot to coordinate the work of each module, wherein the agent represents each module and part of the robot, including the power module, the arm module, the leg module, and the head module. The strategy of the agent is the current action, wherein the power module controls the size of the output, and other modules that can act control the direction and amplitude of the action. All these actions are controlled by the agent value function network, and the reward represents the distance of the entire robot action. Each module can only see its own working status, and cannot see the working status of the entire robot. The global state of the environment contains the working status of each module and the state of the entire robot.

[0115] The training process includes:

[0116] Step 1, initialize the state of the entire robot and each module;

[0117] Step 2: Each module uses an exploration strategy to interact with the environment and saves the experience into the experience cache pool:

[0118] Step 2-1: Each module uses its current state and the action at the previous moment as the input of the value function network, and outputs the value function of all actions at the current moment, that is, the cumulative reward that can be obtained for each action;

[0119] Step 2-2: Each module decides to take a random action or the action with the maximum value function based on the current exploration rate;

[0120] Step 2-3, each module obtains the reward and new observation of the environment, and saves the current observation, current action, reward, and new observation as one experience into the experience cache pool;

[0121] Step 3: Execute the training of the agent network:

[0122] Step 3-1, sample from the experience buffer pool and use the agent value function network to obtain the action value function of the agent at the current moment;

[0123] Step 3-2, take the global state as an additional input and use the attention network to calculate the global state encoding;

[0124] Step 3-3, use the hypernetwork to calculate the value decomposition weights and biases, and the input of the hypernetwork is the global state encoding;

[0125] Step 3-4, use the separated value decomposition to calculate the value function of the joint actions of all agents. The value decomposition weights and biases are the outputs of step 3-3;

[0126] In steps 3-5, the target network is used to calculate the loss of the joint action value function of all modules and optimize the agent value function network of each module.

[0127] Step 4: loop through steps 2 and 3 until the end. Finally, the greedy strategy of each module is obtained according to the agent value function of each module as the output strategy.

[0128] The output strategy obtained by using the present invention can improve the efficiency of robot control, increase the convergence speed of cooperative strategies, and complete tasks faster. The characteristics of separate execution enable each module of the robot to effectively cooperate with other modules based on its own state to achieve overall balance and stability. Compared with the rule-based method, it has stronger adaptability and can be applied to different environments, and can also play a role in complex environments.

[0129] It can be seen that the value decomposition multi-agent reinforcement learning training method using the attention network can more efficiently achieve the collaboration between multiple agents. The attention network used can automatically pay attention to the favorable parts of the environment, while ignoring the unimportant parts, and more efficiently encode complex global state information. The separated value decomposition structure can reduce the learning difficulty of the hypernetwork, more accurately calculate the value decomposition weights, accelerate the convergence of the model, improve the efficiency of learning, and finally learn the strategy to obtain the maximum reward.

[0130] The present invention provides a method and idea for a multi-agent reinforcement learning training method using value decomposition of an attention network. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A value decomposition multi-agent reinforcement learning training method using an attention network, characterized in that: The following steps are involved: Step 1, initialize the environment and all agent states; including: initializing the agent network and environment, initializing the value function network of each agent and the corresponding target network, initializing the network used to calculate the joint action value function, initializing the environment, and obtaining the initial state; Step 2: The agent uses the exploration strategy to interact with the environment and saves the experience into the experience cache pool; Step 3: Execute the training of the agent network: that is, sample from the experience buffer pool and calculate the loss function; Step 4: loop through steps 2 and 3 until the end; finally, the agent greedy strategy is obtained according to the agent value function and used as the output strategy to complete the multi-agent reinforcement learning training; Wherein, step 3 comprises: Step 3-1, sampling from the experience buffer pool, and using the agent value function network to obtain the action value function of the agent at the current moment; Step 3-2, take the global state as an additional input and use the attention network to calculate the global state encoding; Step 3-3, use the hypernetwork to calculate the value decomposition weights and biases, and the input of the hypernetwork is the global state encoding; Step 3-4, use the separated value decomposition to calculate the value function of the joint actions of all agents. The value decomposition weights and biases are the outputs of step 3-3; Step 3-5, use the target network to calculate the loss of the joint action value function and optimize the agent value function network; Wherein, step 3-1 comprises: During each training, N experience replays are sampled from the experience cache pool, that is, a sequence including the current observation, action, reward, and next observation. The agent value function network is used to calculate the cumulative reward that can be obtained by executing the action under the current observation, that is, the value function; Step 3-2 includes: Use the attention network to encode the global state; the input of the attention network is the global state s t , the attention network is calculated as: Among them, LF represents the linear function, Attention represents the attention network, and Q t , K t , V t is the input matrix of the attention network, SF t represents the global state encoding, Represents the final global state encoding; the global state s t After the linear function, we get Divide into multiple parts, each part Using linear function alone to calculate the attention mechanism and Matrix, the calculation method of the attention mechanism is: Among them, exp represents the exponential function, Q, K, V are the attention network input matrices, and f is the dot multiplication function used to calculate K i The similarity with all Q, the denominator is all K j The sum of similarities with all Q is used for normalization; the attention network outputs the final global state encoding Step 3-3 includes: Use the hypernetwork to calculate the value decomposition weights and biases, and the input is the global state encoding obtained in step 3-2; the hypernetwork consists of two linear layers, and the global state encoding Divided into two parts and The values ​​in each part are concatenated together as the input to the hypernetwork; this process is: Where LF represents the hypernetwork, which is a linear function, and w 1 and b 1 denote the value decomposition weights and biases generated by the first layer of the hypernetwork, w 2 and b 2 Represents the value decomposition weights and biases generated by the second layer hypernetwork; Steps 3-4 include: The joint action value function is calculated using the separated value decomposition structure. The input is the weights and biases obtained in step 3-3. The final joint action value function is the independent value function Q of each agent. i The output of the fusion of weights w and bias b obtained using the hypernetwork; this process is: h=Q i *w 1 +b 1 Q tot =h*w 2 +b 2 Where Q tot Represents the value function of the joint actions of all agents, h is a temporary variable; Steps 3-5 include: Use TD(λ) to calculate the expected value y of the joint action value function tot Then the loss function for: Where θ represents the network parameters of the agent value function, E represents the mathematical expectation, D represents the experience cache, Q tot represents the joint action-value function.

2. According to claim 1, a value decomposition multi-agent reinforcement learning training method using an attention network is characterized in that: The step 2 comprises: Step 2-1: The agent uses the current observation and the action at the previous moment as the input of the value function network, and outputs the value function of all actions at the current moment, that is, the cumulative reward obtained by each action; Step 2-2: The agent decides to take a random action or the action with the maximum value function based on the current exploration rate; In step 2-3, the agent obtains rewards and new observations from the environment, and saves the current observation, current action, reward, and new observation as one experience into the experience cache pool.

3. According to claim 2, a value decomposition multi-agent reinforcement learning training method using an attention network is characterized in that: The step 2-1 comprises: Each agent i is based on the current observation and the action at the previous moment Get the value function of all current actions; the value function network consists of 3 layers, a linear layer, a recurrent neural network layer and the last linear layer; the recurrent neural network uses the hidden state at the current moment as input and outputs the encoding at the current moment and the hidden state at the next moment; the process is expressed as follows: Where MLP stands for multi-layer perceptron, RNN stands for recurrent neural network, and is a temporary variable, represents the hidden state at the current moment, represents the hidden state at the next moment, Q i Represents the action-value function of the agent.

4. According to claim 3, a value decomposition multi-agent reinforcement learning training method using an attention network is characterized in that: The step 2-2 comprises: The agent decides the action to take based on the current value function. In the experience accumulation stage, the agent adopts an exploration strategy rather than simply executing the optimal strategy. The agent maintains an exploration rate ε. The exploration strategy means that the agent will execute random actions with a probability of ε and execute the optimal action with a probability of 1-ε, that is, the action with the largest value function. The exploration rate decreases with time but is not zero, that is, the probability of executing random actions is getting smaller and smaller, but it is not zero. The steps 2-3 include: All agents maintain an experience buffer pool D. When agent i makes an action, the agent receives feedback from the environment, including the reward r shared by all agents. t and new observations The agent saves the current observation, current action, reward, and new observation as experience into the experience buffer pool to prepare for subsequent learning; if the experience buffer pool is full, the earliest experience is discarded.

Images