A method and device for enhancing the stability of multi-agent strategy learning

Abstract

The present application relates to a method and apparatus for enhancing the stability of multi-agent strategy learning. The method extends the concept of value averaging in a single-agent algorithm to a multi-agent collaborative environment. Based on the multi-agent value decomposition algorithm, the method switches between the Max and Mellowmax operators during the iterative optimization process, effectively leveraging the strong exploratory nature of the Max operator and the good stability of the Mellowmax operator. When switching to the Max operator, the variance of the target approximation error is reduced by averaging multiple target network estimates, thereby alleviating learning instability caused by overestimation. In the mobile robot's autonomous obstacle avoidance task, the use of this method significantly reduced the overestimation error of the original value decomposition method, reduced the variance of the target approximation error, enhanced strategy stability, and significantly reduced the instability of strategy learning, which can cause the robot to make continuous erroneous movements or hardware damage caused by the robot during the collaborative obstacle avoidance process.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a method and apparatus for enhancing the stability of multi-agent strategy learning. Background Art

[0002] A key hallmark of mobile robot intelligence is the ability to autonomously accomplish specific tasks through collaboration. However, this autonomous task requires robots to meet a fundamental requirement: obstacle avoidance. Obstacle avoidance involves using sensors to detect static and dynamic obstacles that obstruct a mobile robot's path, using collected obstacle status information. The robot then uses a specific method to effectively avoid these obstacles and ultimately reach its destination.

[0003] In autonomous obstacle avoidance tasks involving mobile robots, multiple robots form a cooperative relationship and need to obtain global obstacle information through their own observations. Multi-agent value decomposition reinforcement learning plays an important role in these tasks, effectively solving the problem of cooperative strategy learning under incomplete observation conditions. However, obstacle avoidance strategy learning based on value decomposition multi-agent algorithms suffers from inherent errors and instabilities, leading to persistent erroneous actions or hardware damage caused by the robots during cooperative obstacle avoidance. The main issues are:

[0004] 1) The maximization operator leads to an overestimation of the target TD value during the iterative optimization process;

[0005] 2) The algorithm network is affected by the randomness and uncertainty of network parameters, and there is a large variance in the approximation of the true value during the iterative optimization process.

[0006] The above two points cause the strategy learning curve obtained by using the multi-agent value decomposition algorithm to have severe oscillations and performance degradation in multiple mobile robot obstacle avoidance tasks.

[0007] Furthermore, currently there are few efforts focused on improving stability in multi-agent scenarios. The MATD3 algorithm focuses on policy gradient methods, extending the TD3 algorithm to the MADDPG algorithm, using a centralized dual critic to reduce bias caused by overestimation. Some have replaced the Max maximization operator used to calculate the TD objective in the value decomposition method with the Soft Mellowmax operator, effectively improving the algorithm's stability. The Res algorithm recommends adding a regularization term to penalize Q-values ​​that deviate from the baseline and using the Softmax operator to calculate the target Q-value. These efforts are preliminary attempts to reduce overestimation in multi-agent scenarios. While they can mitigate the impact of overestimation, replacing the Max operator often leads the algorithm to a suboptimal solution, ultimately affecting its learning performance. Summary of the Invention

[0008] Based on this, it is necessary to provide a method and device for enhancing the stability of multi-agent strategy learning without losing strategy learning efficiency and effectively reducing overestimation in response to the above technical problems.

[0009] A method for enhancing the stability of multi-agent strategy learning, the method comprising:

[0010] The multi-agent collaborative task to be solved is modeled as a multi-agent decentralized partially observable Markov decision process, where the information of each agent includes: local observation information, action and system state.

[0011] Choose any value decomposition multi-agent reinforcement learning algorithm that uses the Max operator in the optimization process, and randomly initialize the parameters of the hybrid network and the policy network of each agent.

[0012] The initial values ​​of the K target hybrid network parameters are set as the hybrid network parameters; the initial values ​​of the L target policy network parameters are set as the policy network parameters;

[0013] Initialize the experience replay pool, exploration rate, and discount factor.

[0014] According to the experience replay pool, the exploration rate and the discount factor, the value decomposition multi-agent reinforcement learning algorithm is trained with the current local observation information and the action of each agent at the previous moment as input. During the iterative optimization process, the TD target is calculated by switching between the Max operator and the Mellowmax operator. When switching to the Max operator, the TD target is optimized using the average value method until the preset conditions are met, the iterative optimization is stopped, and the optimal strategy for each agent is obtained.

[0015] Each agent adopts the corresponding optimal strategy to complete the multi-agent collaborative task.

[0016] A device for enhancing the stability of multi-agent strategy learning, comprising:

[0017] The task modeling module is used to model the multi-robot cooperative obstacle avoidance task as a multi-agent decentralized partially observable Markov decision process. The information of each robot includes: local observation information, action and system state;

[0018] The multi-agent reinforcement learning model initialization module selects any value decomposition multi-agent reinforcement learning algorithm that uses the Max operator in the optimization process and randomly initializes the parameters of the hybrid network and the policy network of each robot; sets the initial values ​​of the K target hybrid network parameters to the hybrid network parameters; sets the initial values ​​of the L target policy network parameters to the policy network parameters; and initializes the experience replay pool, exploration rate, and discount factor.

[0019] The optimal strategy determination module is used to train the value decomposition multi-agent reinforcement learning algorithm based on the experience replay pool, exploration rate, and discount factor, with the current local observation information and the action of each robot at the previous moment as input. During the iterative optimization process, the TD target is calculated by switching between the Max operator and the Mellowmax operator. When switching to the Max operator, the TD target is optimized using the average value method until the preset conditions are met, at which time the iterative optimization is stopped to obtain the optimal strategy for each robot.

[0020] The multi-robot collaboration module is used for each robot to adopt the corresponding optimal strategy to complete the multi-robot cooperative obstacle avoidance task.

[0021] The aforementioned method and apparatus for enhancing the stability of multi-agent policy learning extends the concept of averaging in single-agent algorithms to a multi-agent collaborative environment. Based on the multi-agent value decomposition algorithm, the method switches between the Max and Mellowmax operators during the iterative optimization process, effectively leveraging the strong exploratory power of the Max operator and the good stability of the Mellowmax operator. When switching to the Max operator, the variance of the target approximation error is reduced by averaging multiple target network estimates, thereby alleviating learning instability caused by overestimation. In the mobile robot's autonomous obstacle avoidance task, this method significantly reduced the overestimation error of the original value decomposition method, reduced the variance of the target approximation error, enhanced policy stability, and significantly reduced the instability of policy learning, which can lead to persistent erroneous actions or robot-induced hardware damage during the collaborative obstacle avoidance process. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 1 is a flow chart of a method for enhancing the stability of multi-agent strategy learning in one embodiment;

[0023] Figure 2 The overall structure of the value averaging method in one embodiment;

[0024] Figure 3 Figure 2 is a performance parameter comparison of the VAOS-QMIX algorithm and the QMIX algorithm in the Predator-prey environment in one embodiment, where (a) is the reward, (b) is the variance, (c) is the action value estimate on a logarithmic scale, and (d) is the comparison between the true value and the estimated value;

[0025] Figure 4 This figure shows a performance parameter comparison between the VAOS-QMIX algorithm and the QMIX algorithm in a physical deception environment in another embodiment, where (a) is the reward and (b) is the variance comparison.

[0026] Figure 5 This is a performance parameter comparison between the VAOS-QMIX algorithm and the QMIX algorithm in the world common environment in another embodiment, where (a) is the reward and (b) is the variance comparison;

[0027] Figure 6 Figure 2 is a comparison of the complete learning curves of four algorithms combined with the VAOS method in the Predator-prey environment in another embodiment, where (a) is VAOS-VDN, (b) is VAOS-WQMIX, (c) is VAOS-QTRAN, and (d) is VAOS-QPLEX.

[0028] Figure 7 This figure shows the performance comparison of the Predator-prey, Physical deception, and World common environment algorithms in another embodiment, where (a) to (c) are the average reward results of the three algorithms in each MPE task, and (d) to (f) are the corresponding reward variances.

[0029] Figure 8 The influence of parameters K and L in another embodiment;

[0030] Figure 9 is the influence of parameters ω and λ in another embodiment, wherein (a) is the influence of parameter ω, and (b) is the influence of parameter λ;

[0031] Figure 10 This is an analysis of ablation experiment results in another embodiment, where (a) the ablation analysis results of the value averaging and operator switching components are performed, and (b) the impact of the two components on the variance of the average return;

[0032] Figure 11 A block diagram of the structure of a device for enhancing the stability of multi-agent strategy learning in one embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0034] In one embodiment, Figure 1 As shown, a method for enhancing the stability of multi-agent strategy learning is provided, which includes the following steps:

[0035] Step 100: Model the multi-robot cooperative obstacle avoidance task as a multi-agent decentralized partially observable Markov decision process, where the information of each robot includes: local observation information, action, and system state.

[0036] Specifically, the decentralized partially observable Markov decision process (Dec-POMDP) ​​model is defined as the tuple in Represents the global state, is the set of actions of all robots. At any time t, robot i∈{1,2,...,N} selects an action according to its strategy Thus forming a joint action In u and state transition probability Under the action of t Transfer to s t+1 , and the robot gets rewarded Assume that all robots share the same reward and the reward function is bounded, that is, r(s t ,u)≤R max At the same time, the robot has a private local observation of the environment By observing the function Determine the target approximation error. Represents the robot's historical observation-action data. Random strategy represents the probability that robot i chooses action a under τ. The learning goal of multi-robots is to find a maximum The joint optimal strategy π=<π1,π2,...,π N >, where γ∈[0,1) represents the discount factor.

[0037] Step 102: Select any value decomposition multi-agent reinforcement learning algorithm that uses the Max operator in the optimization process, and randomly initialize the parameters of the hybrid network and the policy network of each robot; set the initial values ​​of the K target hybrid network parameters as the hybrid network parameters; set the initial values ​​of the L target policy network parameters as the policy network parameters.

[0038] Specifically, a key issue in multi-agent reinforcement learning is how to learn the joint action-value function, because the parameters of this function grow exponentially with the number of agents. If the input space of the action-value function is too large, it is difficult to fit a suitable function to represent the true joint action-value function. Another problem is how to extract an excellent distributed policy from the joint action-value function after learning it. This is actually the core issue of extending single-agent reinforcement learning to MARL. Value decomposition methods use value functions to approximate the solution policy, essentially approaching the optimal policy through iterative optimization. However, because value decomposition methods use techniques such as the Max operator during the iteration process, the estimated Q-values ​​are often higher than the true Q-values, causing the algorithm to easily fall into the error of reinforcing low-value actions.

[0039] This invention is based on the framework of the value decomposition algorithm and incorporates the two techniques of value decomposition and operator switching. Therefore, the iterative optimization steps in the training process are the same as those of the value decomposition algorithm. Both adopt the TD update method to minimize the mean square error between the TD target value and the current global Q value. The local action value Q output by the average target policy network and the global action value Q obtained by the target hybrid network are tot Reduce overestimation errors.

[0040] Wherein, K and L are both integers greater than 0. The target network includes a target hybrid network and a target policy network.

[0041] Step 104: Initialize the experience replay pool, exploration rate, and discount factor.

[0042] Step 106: Based on the experience replay pool, exploration rate, and discount factor, the value decomposition multi-agent reinforcement learning algorithm is trained with the current local observation information and the action of each robot at the previous moment as input. During the iterative optimization process, the TD target is calculated by switching between the Max operator and the Mellowmax operator. When switching to the Max operator, the TD target is optimized using the average value method until the preset conditions are met. The iterative optimization is stopped to obtain the optimal strategy for each robot.

[0043] Specifically, the Max operator can cause overestimation, but completely replacing it can easily lead the algorithm into suboptimal solutions. (Using the Max operator to select joint actions and iteratively optimize Q-values ​​can quickly solve for the optimal joint policy, but it can easily lead to overestimation of the true Q-value.) Therefore, switching between the Max operator and the Mellowmax operator effectively leverages the Max operator's strong exploration and Mellowmax's stability. While operator switching preserves the respective advantages of both operators, overestimation still occurs when switching to the Max operator. Therefore, value averaging is used to further stabilize the learning process. Value averaging is an extension of the Average DQN algorithm to the multi-agent value decomposition method. The overall idea is to leverage the variance reduction advantage of averaging the estimates of multiple Q networks to improve the stability of policy learning in multi-agent environments. Value averaging inherits the common CTDE training architecture of centralized training and decentralized execution in value decomposition methods and averages the Q-values ​​output by the global target hybrid network and each robot's target policy network.

[0044] The overall structure of the value averaging method is as follows Figure 2 As shown, it includes n robots and 1 hybrid network module, each robot includes 1 policy network and L target policy networks for storing previously learned targets, and the hybrid network module includes 1 hybrid network and K previously learned target hybrid networks.

[0045] Assume that the target approximation error (TAE) is the error in a single-agent setting. is a random variable with zero mean and uniform distribution, and it is deduced that when the maximization operator Max is used Often leads to overestimation errors has a positive expected value, i.e. When it is extended to the value decomposition method in the multi-agent scenario, it is found that the conclusion still holds. Assume that the target approximation error of each agent is is a random variable uniformly distributed in the interval [-ε,ε], and At the same time, if all c actions of each agent share the same target Q value when it is in state s′, then On this basis, it is assumed that Make Then the expectation of the overestimation error caused by the maximization operator in the multi-agent scenario is: Upper bound on overestimation error The value of Q is proportional to the number of agents, indicating that the maximization operator is not affected by the zero-mean noise characteristic of the function approximation process when selecting actions, resulting in a positive bias in the results. To reduce this bias, some research has replaced the Max operator with the Mellowmax or Softmax operators. While this replacement can alleviate overestimation, it is no longer the best approximation of Q-values ​​and can easily lead to suboptimal solutions. The Max operator is highly exploratory of the policy, while the Mellowmax operator can reduce overestimation and focus on policy utilization.

[0046] Replace the Max operator with the Mellowmax operator, and according to the calculation process of Θ, we can get Then we can deduce that It can be found that Θ and Θ mm The difference between Θ and Θ is always positive, that is, the error caused by alternating the Mellowmax operator is always smaller than the error caused by using the Max operator alone. mm The difference between the two operators decreases as ω increases, that is, as the value of ω increases, the behavior of the Mellowmax operator becomes closer and closer to that of the Max operator. Therefore, alternating between the two operators can effectively reduce the overestimation error.

[0047] Step 108: Each robot adopts the corresponding optimal strategy to complete the multi-robot cooperative obstacle avoidance task.

[0048] The aforementioned method for enhancing the stability of multi-agent policy learning extends the concept of averaging in single-agent algorithms to a multi-agent collaborative environment. Based on the multi-agent value decomposition algorithm, it switches between the Max and Mellowmax operators during the iterative optimization process, effectively leveraging the strong exploratory power of the Max operator and the good stability of the Mellowmax operator. When switching to the Max operator, the variance of the target approximation error is reduced by averaging multiple target network estimates, thereby alleviating learning instability caused by overestimation. In a mobile robot autonomous obstacle avoidance task, this method significantly reduced the overestimation error of the original value decomposition method and the variance of the target approximation error, enhancing policy stability and significantly reducing the instability of policy learning, which can lead to persistent erroneous actions or robot-induced hardware damage during collaborative obstacle avoidance.

[0049] In one embodiment, step 106 includes: setting a maximum number of iterations and setting the current number of iterations to 1; during the current iteration, collecting the global state; inputting the current local observation information and the action of each robot at the previous moment into their respective policy networks to obtain the current action state value of each robot; randomly selecting the next action according to the ε-greedy strategy and calculating L target action values; executing the joint action u, transitioning to the next state, and collecting samples to update the experience replay pool; if the experience replay pool is full, sampling a subset from the experience replay pool and calculating the average value of the current iteration process; inputting the average value of the current iteration process corresponding to each robot into the hybrid network to obtain the global action value and the output of the K target hybrid networks; calculating the TD target by switching between the Max operator and the Mellowmax operator based on the global action value and the output of the K target hybrid networks; optimizing the parameters of the policy network and the hybrid network by minimizing a preset loss function; updating the parameters of the target policy network and the target hybrid network according to a preset frequency, and attenuating the exploration rate, increasing the current number of iterations by 1, and performing the next round of iterative optimization until the current number of iterations is greater than the maximum number of iterations, thereby obtaining the optimal strategy for each robot.

[0050] Specifically, as a preferred approach, the QMIX algorithm, based on the QMIX algorithm, employs a hybrid network to merge the local value functions of individual agents and incorporates global state information during training to improve algorithm performance. QMIX is a multi-agent reinforcement learning algorithm with the following characteristics: 1) It learns a distributed policy. 2) It is essentially a value function approximation algorithm. 3) Because a single total reward is assigned to a joint action-state, rather than individual rewards for each agent, it is suitable for cooperative environments. 4) The QMIX algorithm utilizes a centralized learning and distributed execution framework. Through centralized information learning, distributed policies are derived for each agent. 5) Global state information is leveraged during training to improve algorithm performance. This is an improvement on the VDN method discussed later. 6) Continuing from the previous point, QMIX designs a neural network to integrate the local value functions of each agent to derive a joint action-value function, whereas VDN performs direct summation. 7) Each agent's local value function requires only its own local observations, resulting in a distributed execution of the entire system. The action with the highest cumulative expected reward is selected based on the local value function. 8) The algorithm makes the joint action-value function identical to the monotonicity of each local value function, so taking the maximum action for each local value function maximizes the joint action-value function. 9) The algorithm targets a distributed multi-agent partially observable Markov decision process (Dec-POMDP).

[0051] The global action value Q obtained by averaging the local action value Q output by the target policy network and the target hybrid network tot Reduce overestimation errors. During training, the TD target is calculated using an operator switching method, effectively combining the advantages of the Mellowmax operator and the Max operator. Simultaneously, a target network architecture is adopted, and the target policy network and target hybrid network are updated at a fixed frequency. This ensures that the robot obtains the most consistent strategy possible when receiving similar observations. At the same time, the hybrid network is expected to obtain a stable decomposition strategy, thereby minimizing the error caused by neural network approximation. Therefore, the VAOS-QMIX algorithm uses K previously learned target hybrid networks and L target policy networks for training to stabilize the algorithm learning process.

[0052] The pseudo code of the VAOS-QMIX algorithm is shown in Algorithm 1, which contains two average calculations. One is the average of the target action value Q of each agent at different stages. i , and the other is the average global target action value Q tot By averaging the target Q i The value method makes the action value of each agent input into the hybrid network more accurate, and the average global Q totThe goal is to reduce the TAE variance during network training and thus stabilize learning. The target networks involved in the average calculation all use an interval update method to save network models at different stages. That is, only one target network is updated at a certain interval, thereby saving the target network parameters of multiple different training stages.

[0053] Algorithm 1:

[0054] 1: Initialize the current policy network θ and the hybrid network parameters μ as random parameters;

[0055] 2: Initialize L target strategy network parameters θ - =θ, and K target hybrid network parameters μ - =μ;

[0056] 3: Initialize the Experience Replay (ER) buffer D, exploration rate ε, and discount factor γ;

[0057] 5: for j=1,2,...,maxiterations do

[0058] 6: Collect global state s t ;

[0059] 7: For i=1,2,...,N do

[0060] 8: Each robot calculates the current action state value Q based on its own strategy network i (o i ,a i θ j );

[0061] 9: Randomly select the next action according to the ε greedy strategy and calculate L target action values

[0062] 10: End for

[0063] 11: Execute the joint action u, transfer to the next state, and collect samples to update the buffer area D;

[0064] 12: If replay buffer D is full then

[0065] 13: Sample minibatch from D;

[0066] 14: Calculation And use it as the input of the hybrid network to get the global action value Q tot (μ j ) and the output of the K target hybrid network

[0067] 15: Calculate TD target y tot , and optimize the network parameters by minimizing the loss function of formula (4);

[0068] 16: End if

[0069] 17: Update the target network parameters at a fixed frequency so that different target networks maintain network parameters at different training stages;

[0070] 18: decay exploration rate ε;

[0071] 19: End for

[0072] In one embodiment, a subset is sampled from the experience replay pool, and the average action value of each robot in the current iteration is calculated as:

[0073]

[0074] Among them, maxQ i is the maximum action value of the ith robot after averaging, is the maximum value of the j-th target strategy network of the i-th robot; are the parameters of the j-th target policy network.

[0075] maxQ i The value averaging method is used to calculate and stabilize the input data of the hybrid network, further enhancing the stability of the value decomposition algorithm.

[0076] In one embodiment, the TD target is calculated using an operator switching method based on the global action value and the output of the K target hybrid network. The calculation formula of the TD target is:

[0077]

[0078]

[0079]

[0080] Among them, y tot For TD target, is the parameter of the target hybrid network at the jth iteration, r is the discounted reward at the tth time step, γ is the discount factor, λ is the regularization coefficient, maxQ i is the average maximum action value of the i-th robot, i = 1, 2, ... N, is the maximum value of the global target action value of all robots under the maximization operator Max, is the value of the global target action value of all robots under the Mellowmax operator mm, s′,u′; μ - are the state at the next moment, the joint action of all robots at the next moment, and the network parameters of the target hybrid network; is the average of the maximum values ​​of the global target action values ​​of all robots under the maximization operator Max, and fre is the frequency of switching between the Max operator and the Mellowmax operator.

[0081] Specifically, the strategy network of the agent in value averaging is based on the current observation o t i And action, output the action value Q corresponding to each agent i The design of the hybrid network is the most important in the value decomposition method. tot The decomposition of is used to assign credit, thereby measuring the relative strengths and weaknesses of each agent's policy network in the current global state. However, the focus of the value averaging method is not entirely on designing the neural network structure, but rather on stabilizing learning by using the averaging concept in value decomposition algorithms such as QMIX that use the Mix operator. Taking the QMIX algorithm as an example, when combined with value averaging, the calculation formulas for the TD target are obtained as shown in Equations (2) to (4).

[0082] The upper bound of the overestimation error is given in operator switching, where it is assumed that However, it is worth noting that the bias of the overestimation error is non-uniform, especially in the MDP process with longer trajectories, where the bias is larger due to the accumulation of TAE variance, because the variance needs to be considered. The variance can be effectively reduced by the value averaging method, thus stabilizing the learning process. Assume that in a Dec-POMDP model with T states, the terminal state value (The initial state is ).random variable are independent of each other, and

[0083] make in The discrete Fourier transform of the rectangular pulse is combined with the average value to estimate the variance of the Q value:

[0084]

[0085] Formula (5) shows that the value decomposition method can effectively reduce the TAE variance after adopting the value average, and is smaller than the TAE variance of the original value decomposition algorithm. times, and is proportional to the square of the number of agents. The use of a value decomposition algorithm after value averaging reduces TAE variance, effectively stabilizing the algorithm's learning process. Furthermore, the Q-values ​​output by the target policy network for each agent are also averaged.

[0086] In one embodiment, the parameters of the policy network and the hybrid network are optimized by minimizing a preset loss function, wherein the preset loss function is:

[0087]

[0088] in, is the loss, y tot For TD target, Q tot (s,u;μ) is the global action value, Q tot (s,u) is the output of the target policy network, λ is the regularization coefficient, r is the discounted reward at the tth time step, μ is the parameter of the hybrid network, s,u are the system state and action respectively, γ is the discount factor, is the experience buffer of the sample, b is from 1 to , v is each step in a task, and the value of v is from the first step to the end of the task.

[0089] Specifically, the cause of learning instability in the DQN algorithm is attributed to the sum of three errors, namely target approximation error, overestimation error, and optimality difference, which are defined as:

[0090]

[0091] The target approximation error is expressed as represents the overestimation error. is the TD target in the DQN algorithm, is the true target value. The optimality difference can be viewed as the error of standard form Q-learning and is an error inherent to the algorithm.

[0092] The value decomposition method based on the CTDE architecture allows a single agent to have access to global information by relaxing the training conditions, that is, it allows the agent to obtain global information during training, but during the execution process, the strategy of each agent can only be based on the current observation. The most important difference between different types of value decomposition methods is the Q tot The decomposition of , that is, how to distribute the contribution of different agents according to their local Q values. The various decomposition methods are described as a generalized function:

[0093]

[0094] in, represents a set of functions containing various mapping relationships, μ is the mapping network parameter, and θ is the agent strategy network parameter. For example, in VDN is defined as a linear function, i.e. QMIX algorithm It is a nonlinear function and is represented by a multi-layer neural network. It is worth noting that in order to obtain a consistent decomposition strategy, the neural network (super network structure) used by the QMIX algorithm needs to meet the monotonicity condition, that is, This ensures that the following Individual-Global-Max (IGM) assumption holds:

[0095]

[0096] In fact, the VDN algorithm is the QMIX algorithm. Both VDN and QMIX algorithms adopt an end-to-end training method, aiming to minimize the following loss:

[0097]

[0098] Among them, y tot =r+γmax u′ Q tot (s′,u′;μ - ), μ - Indicates the target network parameters.

[0099] when (that is, the global action Q value can be obtained by summing the local Q values ​​of all agents), the mapping function The global Q value is expressed by linear accumulation. In the multi-agent scenario, the total error of the multi-agent value decomposition method at the jth iteration is:

[0100]

[0101] in, is the target approximation error, is the overestimation error, is the optimality difference, Q tot (s,u;μ j ) represents the global action state value at the jth iteration, is the global target action value estimated by the hybrid network, The global target action estimate estimated by the hybrid network; is the local target action value estimated by the policy network of agent i, is the true target (usually replaced by the average value of the network test results after j steps of iteration).

[0102] Both value averaging and operator switching stabilize learning by changing the TD (temporal difference (TD) method is the core content of reinforcement learning theory and the most important achievement in the field of reinforcement learning) target during the optimization process. This has been able to effectively solve the serious overestimation problem in the value decomposition method. As shown in formula (5), value averaging significantly reduces the variance of the target approximation error, but the error still exists and gradually accumulates. In order to further change the trend of error increase, a regularization method is introduced to directly penalize the global estimate Q that deviates too much from the true value. tot The specific method is to add a penalty term to the optimization objective. Discount rewards As a baseline for learning, r t represents the discounted reward at the t-th time step, and λ represents the regularization coefficient.

[0103] The method of this invention is based on the framework of the value decomposition algorithm and incorporates the two techniques of value decomposition and operator switching. Therefore, the iterative optimization steps during training are the same as those of the value decomposition algorithm. Both use the TD update method to minimize the mean square error between the TD target value and the current global Q value. Taking the QMIX algorithm as an example, the final optimization objective of the VAOS-QMIX algorithm is shown in Equation (6).

[0104] In one embodiment, the value decomposition multi-agent reinforcement learning algorithm is: VDN algorithm, QMIX algorithm, QPLEX algorithm, QATTEN algorithm, QTRAN algorithm, WQMIX algorithm or other value decomposition algorithms based on the principles of reinforcement learning algorithms.

[0105] In one embodiment, the calculation formula of the Mellowmax operator is:

[0106]

[0107] Among them, ω is the temperature parameter, mmωQ i (o i ,:) is the target action value calculated by the algorithm using Mellowmax operator for robot i, c is the number of actions that the robot can choose, Q i (o i ,a′) is the action value of each robot, o i is the local observation information of robot i, a′ is the specific action of the robot, and the calculation formula of the Mellowmax operator is shown in Equation (12). Existing research results have proved that under any temperature parameter setting, Mellowmax is convex and monotonically non-decreasing with respect to its temperature parameter.

[0108] In one embodiment, local observation information includes: the current position and speed of the robot itself, and the position and speed of teammates; actions include: the movement of each robot in the four directions of forward, backward, left, and right, as well as other directional movements composed of these four directions; the system state is the observation information and action information of all robots, as well as the state of the environment in which the robots are located.

[0109] In a validation example, we experimentally analyze the performance of the VAOS method. We first describe the environment setup, experimental details, and briefly introduce the baseline algorithm. We then conduct comparative and ablation experiments to verify the effectiveness of the VAOS method in reducing overestimation errors and improving the learning stability of the value decomposition method.

[0110] (1) Experimental environment

[0111] We test and study our algorithms on the multi-agent benchmark environment, the Particle Environment. The MPE environment is based on a popular open-source implementation that abstracts real-world cooperation and competition into interactions between particles, each with continuous observations and discrete actions. In Predator-prey, prey and predators compete with each other. Three slower predators are rewarded for collaborating to capture prey, while faster prey are rewarded for staying away from predators. In Physical deception, two red agents quickly reach a target landmark without their opponents knowing the target, and their rewards are inversely proportional to the distance of their opponents from the target landmark. The World common environment is more complex, with four slower agents collaborating to capture two faster opponents. The opponents are rewarded for eating food, and the agents are rewarded for capturing their opponents.

[0112] (2) Baseline Model

[0113] The VAOS method is compared with the overestimation reduction algorithms such as QMIX (DR) and RES. QMIX (DR) is a method that combines the Double Q technique with the QMIX algorithm to estimate the global Q. totThe RES algorithm is currently the best algorithm for reducing overestimation in multi-agent value decomposition algorithms. It replaces the Max operator with the Softmax operator and penalizes Q values ​​that deviate too far through regularization. The RES algorithm uses the default parameters of the original algorithm in the experiment. Specifically, the VAOS method sets the number of target policy networks, L, and the number of target hybrid networks, K, to 5 and 10, respectively, in different value decomposition algorithms. Furthermore, during operator switching, to fully leverage the advantages of both operators and minimize the number of artificially introduced hyperparameters in the algorithm, the aforementioned algorithms in the experiment all set fre to 2, meaning that operator switching is performed once per iteration. The temperature parameter ω is set to 10. The ε-greedy strategy is used as the exploration strategy.

[0114] In the MPE environment, average reward is used as a metric to evaluate the learning quality of algorithms. Each algorithm was run using five random seeds in the experiments, and the results are presented as mean ± standard deviation. The experiments were conducted for 5 million or 10 million steps for specific demonstrations, with evaluation metrics such as average reward and learning variance reported every 2000 steps. The adversary in each of the four MPE tasks was pre-trained with the MADDPG algorithm for 10e4 episodes and exhibited a strong adversarial strategy. The regularization coefficient λ was set to {0.05, 0.5, 0.05} in all three MPE tasks. All experiments were performed on a Linux system with an i9-12900H CPU and 32GB of main memory.

[0115] (3) Performance analysis

[0116] First, let’s answer the first question: how does the value decomposition algorithm after combining the VAOS method perform in terms of learning stability? Let’s take the QMIX algorithm as an example. Figure 3 The performance of the VAOS-QMIX algorithm and the QMIX algorithm are demonstrated in this paper. The results show that the QMIX algorithm integrated with the VAOS method has significantly improved performance, achieved stable learning, and eliminated the performance degradation caused by overestimation.

[0117] like Figure 3 As shown in (a), the reward curve of the QMIX algorithm suddenly drops until the algorithm fails. This is because the QMIX algorithm overestimates some wrong actions during training, thus destroying the entire learning process. Figure 3 The action value estimated by QMIX in (d) deviates sharply from the true value, which is also the cause Figure 3 The reason for the sudden drop in the QMIX algorithm in (a). After using the VAOS method, the motion estimation value is obviously closer to the true value, achieving a smaller deviation. Figure 3This is particularly evident in (c), where the logarithmic scale curve of the action value estimation by the QMIX algorithm becomes larger and larger, while the curve becomes flatter after the VAOS method is used. This also shows the effectiveness of the VAOS method in reducing the overestimation error. The VAOS method stabilizes learning by reducing the variance of TAE by averaging the value. Figure 3 This is also demonstrated in Figure (b), which compares the variance of the returns during the learning process of the two algorithms. The VAOS-QMIX algorithm is significantly superior to the QMIX algorithm in terms of learning stability. The fact that the variance of the QMIX algorithm approaches zero in the later stages of the figure is due to the failure of the QMIX algorithm to learn.

[0118] At the same time, we conducted comparative experiments in two other MPE environments, such as Figure 4 The figure shows the performance parameter comparison between the VAOS-QMIX algorithm and the QMIX algorithm under the Physical deception environment, where (a) is the return and (b) is the variance comparison. Figure 5 Figure 2 shows a performance parameter comparison between the VAOS-QMIX and QMIX algorithms under the World Common environment, where (a) shows the reward and (b) shows the variance. The results show that the VAOS-QMIX algorithm far outperforms the QMIX algorithm in both performance and stability, demonstrating the significant advantages of value averaging and operator switching in mitigating overestimation and enhancing algorithm learning stability.

[0119] The principle of VAOS, which enhances algorithm stability through value averaging and operator switching, is universal and can be easily extended to other value decomposition algorithms based on Q-learning. Therefore, we combined it with four other popular value decomposition methods to obtain the VAOS-VDN, VAOS-WQMIX, VAOS-QPLEX, and VAOS-QTRAN algorithms. Figure 6 Figure 1 shows a comparison of the complete learning curves of four algorithms combined with the VAOS method in the Predator-prey environment: (a) VAOS-VDN, (b) VAOS-WQMIX, (c) VAOS-QTRAN, and (d) VAOS-QPLEX. The results show that the value decomposition algorithm implemented with the VAOS method outperforms the original baseline algorithm in both performance and learning stability.

[0120] (4) Comparative experiment

[0121] What are the advantages of the VAOS method compared to other solutions for reducing overestimation errors in the value decomposition algorithm? The RES algorithm addresses overestimation in the value decomposition algorithm. QMIX (DR) combines the double Q and regularization techniques in QMIX. Figure 7 (a) to (c) show the average return results of the three algorithms in each MPE task. Figure 7 (d) to (f) represent the corresponding return variances. Figure 7 As shown in the figure, both RES and QMIX(DR) effectively address the learning degradation problem of QMIX, but the VAOS-QMIX algorithm outperforms in terms of final performance. This is because RES completely replaces the Max operator with the Softmax operator, which, while effective in reducing overestimation, makes the algorithm more susceptible to falling into local optima. While QMIX(DR) only uses the Max operator, its regularization term forcibly reduces overestimation, limiting the Max operator's exploratory power. The VAOS method, on the other hand, combines the advantages of both the Max and Mellowmax operators, maintaining the Max operator's exploratory power while effectively mitigating overestimation.

[0122] (5) Ablation experiment

[0123] Using the Predator-prey scenario as an example, we analyzed the sensitivity of the VAOS-QMIX algorithm to hyperparameters during implementation and the impact of various components on algorithm performance. Hyperparameters primarily include the number of networks in value averaging, the temperature parameter ω, and the regularization coefficient λ. Components primarily include value averaging and operator switching.

[0124] Effect of the number of target networks K and L. We fix other factors unchanged and vary L and K to illustrate the sensitivity of the VAOS method to the number of networks. Figure 8 (a) and (b) show the performance of the VAOS method when L and K change, respectively. These two parameters have a significant impact on the variance of the target approximation error and the learning rate. Small values ​​cannot effectively reduce the variance of TAE, but large values ​​seriously affect the efficiency of the algorithm. However, in fact, as long as the parameters L and K are set reasonably, it is possible to strike a good balance between stability and efficiency. Figure 8 The results in (a) and (b) illustrate this point well. It also proves that the VAOS method is not sensitive to the settings of L and K.

[0125] The effect of the temperature parameter ω. The temperature parameter ω is crucial for the Mellowmax operator. Theoretically, when ω approaches positive infinity, the Mellowmax operator degenerates into the Max operator, which is equivalent to the VAOS method that only uses the Max operator. When ω approaches negative infinity, the Mellowmax operator degenerates into the mean operator. Therefore, the setting of the temperature parameter ω is also a trade-off between exploration and utilization of the algorithm. When ω is small, the advantage of the good stability of the Mellowmax operator is fully utilized, while when ω is large, more emphasis is placed on exploration. Figure 9 As shown in (a), the VAOS method is not sensitive to the hyperparameter ω, and the performance of the algorithm can remain competitive in a wide range of ω.

[0126] The influence of the regularization coefficient λ. Figure 9 (b) shows the sensitivity of the VAOS-QMIX algorithm to the hyperparameter λ. The regularization term forces the penalty to overestimate the action value of the wrong action. When λ is 0, the VAOS method can stably learn through value averaging and operator switching, but the overestimation error of the true value is still accumulating. When λ becomes larger, the algorithm focuses on learning the regularization target, affecting the efficiency of the algorithm. In fact, the existence of the regularization term in the VAOS method has little effect on the final performance of the algorithm. Figure 9 This conclusion can also be clearly drawn from (b).

[0127] The impact of components. We conducted ablation analysis on the VA (value averaging) and OS (operator switching) components, and the results are as follows: Figure 10 As shown in (a). Figure 10 Figure (b) shows the impact of the two components on the variance of the average return. Clearly, using only the VA method effectively reduces variance and stabilizes learning, but this also impacts algorithm efficiency. When using only the OS method, the algorithm converges quickly in the early stages of training, but due to the larger variance, the algorithm's ultimate performance is inferior to that of the VAOS method. Therefore, the two components of value averaging and operator switching are crucial for reducing the algorithm's overestimation error and stabilizing the algorithm's learning process, further demonstrating the effectiveness of the proposed method.

[0128] We analyzed the learning instability caused by overestimation in the multi-agent value decomposition algorithm and proposed a VAOS method based on value averaging and operator switching, which effectively alleviated the overestimation phenomenon and thus enhanced the learning stability of the value decomposition algorithm. In addition, the focus of the method of the present invention is to fine-tune the network structure of the existing value decomposition algorithm to improve its stability, and to provide theoretical support and explanation. The VAOS method is versatile and can effectively reduce the overestimation error caused by the maximization operator, so it is also applicable to other multi-agent reinforcement learning algorithms that contain these two components. A large number of experiments have shown that the VAOS method significantly reduces the overestimation error of the original value decomposition method and reduces the TAE variance, effectively improving the stability of the algorithm while ensuring the learning effect. Future actions should focus more on algorithm efficiency, that is, studying how to improve the training speed while enhancing stability, so that the VAOS method can be better transferred to other MARL.

[0129] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0130] In one embodiment, Figure 11 As shown, a device for enhancing the stability of multi-agent strategy learning is provided, comprising: a task modeling module, a multi-agent reinforcement learning model initialization module, an optimal strategy determination module and a multi-robot collaboration module, wherein:

[0131] The task modeling module is used to model the multi-robot cooperative obstacle avoidance task as a multi-agent decentralized partially observable Markov decision process. The information of each robot includes: local observation information, action and system state;

[0132] The multi-agent reinforcement learning model initialization module selects any value decomposition multi-agent reinforcement learning algorithm that uses the Max operator in the optimization process and randomly initializes the parameters of the hybrid network and the policy network of each robot; sets the initial values ​​of the K target hybrid network parameters to the hybrid network parameters; sets the initial values ​​of the L target policy network parameters to the policy network parameters; and initializes the experience replay pool, exploration rate, and discount factor.

[0133] The optimal strategy determination module is used to train the value decomposition multi-agent reinforcement learning algorithm based on the experience replay pool, exploration rate, and discount factor, with the current local observation information and the action of each robot at the previous moment as input. During the iterative optimization process, the TD target is calculated by switching between the Max operator and the Mellowmax operator. When switching to the Max operator, the TD target is optimized using the average value method until the preset conditions are met, at which time the iterative optimization is stopped to obtain the optimal strategy for each robot.

[0134] The multi-robot collaboration module is used for each robot to adopt the corresponding optimal strategy to complete the multi-robot cooperative obstacle avoidance task.

[0135] In one embodiment, the optimal strategy determination module is further used to set the maximum number of iterations and set the current number of iterations to 1; during the current iteration process, collect the global state; input the current local observation information and the action of each robot at the previous moment into their respective policy networks to obtain the current action state value of each robot; randomly select the next action according to the ε-greedy strategy and calculate L target action values; execute the joint action u, transfer to the next state, and collect samples to update the experience replay pool; if the experience replay pool is full, sample a subset from the experience replay pool and calculate the average value of the current iteration process; input the average value of the current iteration process corresponding to each robot into the hybrid network to obtain the global action value and the output of the K target hybrid network; based on the global action value and the output of the K target hybrid network, calculate the TD target by switching between the Max operator and the Mellowmax operator; optimize the parameters of the policy network and the hybrid network by minimizing a preset loss function; update the parameters of the target policy network and the target hybrid network according to a preset frequency, and attenuate the exploration rate, increase the current number of iterations by 1, and perform the next round of iterative optimization until the current number of iterations is greater than the maximum number of iterations, and obtain the optimal strategy for each robot.

[0136] In one embodiment, the optimal strategy determination module is further configured to sample a subset from the experience replay pool and calculate the average action value of each robot in the current iteration using formula (1).

[0137] In one embodiment, the optimal strategy determination module is further configured to calculate the TD target using an operator switching method based on the global action value and the output of the K target hybrid networks. The calculation expressions of the TD target are shown in Equations (2) to (4).

[0138] In one embodiment, the preset loss function in the optimal strategy determination module is shown in formula (6).

[0139] In one embodiment, the value decomposition multi-agent reinforcement learning algorithm in the multi-agent reinforcement learning model initialization module is: VDN algorithm, QMIX algorithm, QPLEX algorithm, QATTEN algorithm, QTRAN algorithm, WQMIX algorithm or other value decomposition algorithms based on the principles of Q learning algorithm.

[0140] In one embodiment, the Mellowmax operator is shown in formula (12).

[0141] In one embodiment, the local observation information in the task modeling module includes: the current position and speed of the robot itself, the position and speed of teammates; the actions include: the movement of each robot in the four directions of forward, backward, left, and right, and other directional movements composed of these four directions; the system state is the observation information and action information of all robots, as well as the state of the environment in which the robots are located.

[0142] Regarding the specific definition of the device for enhancing the stability of multi-agent strategy learning, please refer to the definition of the method for enhancing the stability of multi-agent strategy learning above, which will not be repeated here. The various modules in the above-mentioned device for enhancing the stability of multi-agent strategy learning can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0143] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for enhancing the stability of multi-agent strategy learning, characterized in that: The method comprises: The multi-robot cooperative obstacle avoidance task is modeled as a multi-agent decentralized partially observable Markov decision process. The information of each robot includes: local observation information, action and system state. Choose any value decomposition multi-agent reinforcement learning algorithm that uses the Max operator in the optimization process and randomly initialize the parameters of the hybrid network and the policy network of each robot; The initial values ​​of the K target hybrid network parameters are set as the hybrid network parameters; the initial values ​​of the L target policy network parameters are set as the policy network parameters; Initialize the experience replay pool, exploration rate, and discount factor; According to the experience replay pool, the exploration rate, and the discount factor, the value decomposition multi-agent reinforcement learning algorithm is trained with the current local observation information and the action of each robot at the previous moment as input. During the iterative optimization process, the TD target is calculated by switching between the Max operator and the Mellowmax operator. When switching to the Max operator, the TD target is optimized using the average value method until a preset condition is met, and the iterative optimization is stopped to obtain the optimal strategy for each robot. Each of the robots adopts the corresponding optimal strategy to complete the multi-robot cooperative obstacle avoidance task.

2. The method according to claim 1, characterized in that According to the experience replay pool, the exploration rate, and the discount factor, the value decomposition multi-agent reinforcement learning algorithm is trained with the current local observation information and the action of each robot at the previous moment as input. During the iterative optimization process, the TD target is calculated by switching between the Max operator and the Mellowmax operator. When switching to the Max operator, the TD target is optimized using the average value method until the preset condition is met, and the iterative optimization is stopped to obtain the optimal strategy for each robot, including: Set the maximum number of iterations and set the current number of iterations to 1; During the current iteration, collect the global state; Input the current local observation information and the action of each robot at the previous moment into their respective strategy networks to obtain the current action state value of each robot; Randomly select the next action according to the ε greedy strategy and calculate L target action values; Execute the joint action u, transfer to the next state, and collect samples to update the experience replay pool; If the experience replay pool is full, a subset is sampled from the experience replay pool to calculate the average action value of the current iteration process; the average action value of the current iteration process corresponding to each robot is input into the hybrid network to obtain the global action value and the output of the K target hybrid network; based on the global action value and the output of the K target hybrid network, the TD target is calculated by switching between the Max operator and the Mellowmax operator; the parameters of the policy network and the hybrid network are optimized by minimizing the preset loss function; The parameters of the target policy network and the target hybrid network are updated according to the preset frequency, and the exploration rate is attenuated. The current number of iterations is increased by 1, and the next round of iterative optimization is performed until the current number of iterations is greater than the maximum number of iterations, and the optimal strategy for each robot is obtained.

3. The method according to claim 2, characterized in that Sample a subset from the experience replay pool and calculate the average value of the current iteration process, including: Sample a subset from the experience replay pool and calculate the average action value of each robot in the current iteration as: Among them, maxQ i is the maximum action value of the ith robot after averaging, The maximum action value output by the j-th target policy network of the i-th robot; are the parameters of the j-th target policy network.

4. The method according to claim 2, characterized in that According to the global action value and the output of the K target hybrid network, the TD target is calculated using the operator switching method, including: According to the global action value and the output of the K target hybrid network, the operator switching method is used to calculate the TD target. The calculation formula of the TD target is: Among them, y tot For TD target, is the parameter of the target hybrid network at the jth iteration, r is the discounted reward at the tth time step, γ is the discount factor, λ is the regularization coefficient, and maxQ i is the average maximum action value of the i-th robot, i = 1, 2, ... N, The global target action value calculated by the algorithm using the maximization operator Max, mm u′ Q tot (s′,u′;μ - ) represents the global target action value calculated by the algorithm using the Mellowmax operator, s′,u′; μ - are the state at the next moment, the joint action of all robots at the next moment, and the network parameters of the target hybrid network, is the average of the maximum values ​​of the global target action values ​​of all robots under the maximization operator Max, and fre is the frequency of switching between the Max operator and the Mellowmax operator.

5. The method according to claim 2, characterized in that The parameters of the policy network and hybrid network are optimized by minimizing the preset loss function. The preset loss function in the step is: in, is the loss, y tot For TD target, Q tot (s,u;μ) is the global action value, Q tot (s,u) is the output of the target policy network, λ is the regularization coefficient, r is the discounted reward at the tth time step, μ is the parameter of the hybrid network, s,u are the system state and action respectively, γ is the discount factor, is the sample experience buffer size, b is from 1 to v is each step in a task, and the value of v is from the first step to the end of the task.

6. The method according to claim 1, wherein The value decomposition multi-agent reinforcement learning algorithm is: VDN algorithm, QMIX algorithm, QPLEX algorithm, QATTEN algorithm, QTRAN algorithm, WQMIX algorithm or other value decomposition algorithms based on the principles of Q learning algorithms.

7. The method according to claim 1, characterized in that The calculation formula of the Mellowmax operator is: Among them, ω is the temperature parameter, mmωQ i (o i ,:) is the target action value calculated by the algorithm using Mellowmax operator for robot i, c is the number of actions that the robot can choose, Q i (o i ,a′) is the action value of each robot, o i is the local observation information of robot i, and a′ is the action of the robot.

8. The method according to claim 1, characterized in that The local observation information includes: the current position and speed of the robot itself, and the position and speed of teammates; The actions include: the movement of each robot in the four directions of forward, backward, left and right, and other directional movements composed of the movements in these four directions; The system state is the observation information and action information of all robots, as well as the state of the environment in which the robots are located.

9. A device for enhancing the stability of multi-agent strategy learning, characterized in that: The device comprises: The task modeling module is used to model the multi-robot cooperative obstacle avoidance task as a multi-agent decentralized partially observable Markov decision process. The information of each robot includes: local observation information, action and system state; The multi-agent reinforcement learning model initialization module selects any value decomposition multi-agent reinforcement learning algorithm that uses the Max operator in the optimization process and randomly initializes the parameters of the hybrid network and the policy network of each robot; sets the initial values ​​of the K target hybrid network parameters to the hybrid network parameters; sets the initial values ​​of the L target policy network parameters to the policy network parameters; and initializes the experience replay pool, exploration rate, and discount factor. an optimal strategy determination module for training the value decomposition multi-agent reinforcement learning algorithm based on the experience replay pool, the exploration rate, and the discount factor, taking the current local observation information and the action of each robot at the previous moment as input, calculating the TD target by switching between the Max operator and the Mellowmax operator during the iterative optimization process, optimizing the TD target by the average value method when switching to the Max operator until a preset condition is met, stopping the iterative optimization, and obtaining the optimal strategy for each robot; The multi-robot collaboration module is used for each of the robots to adopt the corresponding optimal strategy to complete the multi-robot cooperative obstacle avoidance task.

10. The device according to claim 9, characterized in that The optimal strategy determination module is also used to set the maximum number of iterations and the current number of iterations to 1. During the current iteration, the global state is collected. The current local observation information and the action of each robot at the previous moment are input into their respective strategy networks to obtain the current action state value of each robot. The next action is randomly selected according to the ε-greedy strategy and L target action values ​​are calculated. The joint action u is executed, the next state is transferred, and samples are collected to update the experience replay pool. If the experience replay pool is full, a subset is sampled from the experience replay pool to calculate the average value of the current iterative process; the average value of the current iterative process corresponding to each robot is input into the hybrid network to obtain the global action value and the output of the K target hybrid network; according to the global action value and the output of the K target hybrid network, the TD target is calculated by switching between the Max operator and the Mellowmax operator; the parameters of the policy network and the hybrid network are optimized by minimizing the preset loss function; the parameters of the target policy network and the target hybrid network are updated according to the preset frequency, and the exploration rate is attenuated, the current number of iterations is increased by 1, and the next round of iterative optimization is performed until the current number of iterations is greater than the maximum number of iterations, and the optimal strategy for each robot is obtained.

Images