Multi-agent cooperative confrontation method with off-line policy reuse

Abstract

The application discloses a multi-agent cooperative confrontation method of offline strategy reuse, relates to the technical field of multi-agent, and solves the problems of large training sample demand and long training time in the prior art. The method comprises the following steps: obtaining a current strategy, all offline strategies in a strategy replay pool and initial parameters, respectively calculating divergence values of the current strategy and the offline strategies, saving a minimum divergence value and a corresponding nearest offline strategy; judging whether the minimum divergence value is greater than or equal to a first threshold value, and determining a decision strategy; judging whether the decision strategy is the same as the nearest offline strategy, and calculating a loss value; judging whether the loss value is less than or equal to a second threshold value, if yes, outputting a strategy in the strategy replay pool; if not, putting a new strategy into the strategy replay pool, updating the initial parameters and continuing to calculate until the loss value is less than or equal to the second threshold value; and the reuse of offline strategy data is realized, and the training time of the current strategy interacting with the environment is saved.

Description

Technical Field

[0001] This invention relates to the field of multi-agent technology, and in particular to a multi-agent cooperative adversarial method that reuses offline strategies. Background Technology

[0002] Many complex tasks today require the collaboration of multiple agents, such as autonomous driving, intelligent transportation, and logistics delivery. Multi-agent reinforcement learning (MARL) is an effective method for solving such tasks. As an important branch of reinforcement learning, MARL involves multiple agents learning optimal policies in an interactive environment. Compared to single-agent reinforcement learning, MARL faces more complex problems, such as game theory, cooperation, and competition among agents. However, MARL also faces many challenges, such as policy convergence and the nonlinearity of the Q-value function. Therefore, researching the algorithms and applications of MARL is a challenging and urgent topic.

[0003] Multi-agent reinforcement learning (MAL) refers to the collaboration or competition among multiple agents in a shared environment. Each agent learns how to choose the optimal behavior based on its own state and reward information. Compared to single-agent reinforcement learning, MML faces more complex problems, such as policy coordination, policy competition, and reward sparsity. Traditional MML algorithms include game theory, collective intelligence, and distributed learning, but these algorithms struggle to solve problems in large-scale MML systems. Therefore, in recent years, many deep learning-based MML algorithms have emerged, such as MADDPG, COMA, and QMIX. These algorithms leverage the powerful fitting capabilities of deep neural networks to achieve efficient learning and collaboration in large-scale MML systems.

[0004] In multi-agent reinforcement learning algorithms, based on how the value function and policy are handled, methods can be divided into two categories: value-based and policy-based. Value-based methods are typically based on Nash equilibrium in game theory, achieving cooperation by training a set of global value functions. QMIX and VDN (Value-Decomposition Networks) are two value-based algorithms. QMIX represents the global Q-value by decomposing it into the product of the Q-values ​​of subtasks, while VDN uses a Q-value function with a maximum operation to select actions. Unlike value-based methods that first learn a value function and then calculate the policy based on that function, policy-based multi-agent reinforcement learning algorithms directly learn the policy of each agent. These algorithms learn and directly optimize policies, enabling all agents to achieve Nash equilibrium in cooperative tasks. The REINFORCE algorithm, a policy-based algorithm, achieves cooperation by directly optimizing the policy of each agent. The A2C (Advantage Actor-Critic) algorithm is a policy-based algorithm that uses both Actor and Critic networks. The Actor network directly outputs actions, while the Critic network estimates the value function of each state and calculates the merits of the actions. COMA, as the first policy-based algorithm to introduce a counterfactual baseline collective action value function, achieves cooperation between agents by using a global value function and independent Actor networks. This allows it to adapt to different numbers of agents and dynamic environments, exhibiting high flexibility and adaptability. However, because policy-based algorithms such as COMA require a lot of training time and sample support to model the behavioral probability distribution of each agent, and the direction of gradient update may not be correct, which can cause policy oscillation and training instability, their performance is generally worse than value-based multi-agent reinforcement learning methods.

[0005] Currently, some researchers have made improvements to the COMA algorithm. Christianos et al. proposed an improved version based on COMA called SMAC. SMAC uses a distributed architecture to improve the algorithm's scalability and introduces a hierarchical policy architecture to reduce the complexity of the action space. Park et al. introduced an attention mechanism to model the behavior of agents, thus obtaining an algorithm AAC that is more suitable for dynamic environments than COMA. Ruan et al. introduced a representation called Coordination Graph to describe the relationships between agents, accelerating the training speed of single agents and thus improving the performance of the algorithm, which is called CGCOMA. The common point of these improved algorithms is that they improve the performance and scalability of the algorithm by introducing different mechanisms, improving the COMA algorithm from the perspective of architecture or single agent behavior patterns. However, none of these methods have considered that the fundamental reason for the low efficiency of policy algorithms is the low sample utilization rate. Summary of the Invention

[0006] This invention provides a multi-agent cooperative adversarial method for offline policy reuse, which solves the problems of low utilization rate of training samples, large demand, and long training time in the prior art. It realizes the reuse of offline policy data and saves the training time of the current policy and environment interaction.

[0007] This invention provides a multi-agent cooperative adversarial method for offline policy reuse, the method comprising:

[0008] Obtain the current policy and all offline policies in the policy replay pool, as well as the initial parameters. Calculate the divergence value between the current policy and the offline policies respectively. Obtain and save the minimum divergence value and the nearest offline policy corresponding to the minimum divergence value.

[0009] If the minimum divergence value is greater than or equal to a first threshold, then the current strategy is interacted with the environmental data to generate a new strategy, and the new strategy is determined to be the decision strategy; otherwise, the nearest offline strategy is determined to be the decision strategy.

[0010] Determine whether the decision strategy is the same as the nearest offline strategy. If they are different, first use the fusion formula to fuse the nearest offline strategy with the new strategy to obtain the parameters of the fused strategy, and then use the loss function to calculate the parameters of the fused strategy to obtain the loss value. If they are the same, use the loss function to calculate the loss value between the nearest offline strategy and the environmental data.

[0011] Determine whether the loss value is less than or equal to the second threshold. If yes, output the strategy in the strategy replay pool. If no, put the new strategy into the strategy replay pool, update the initial parameters and continue to calculate until the loss value is less than or equal to the second threshold.

[0012] In one possible implementation, the specific formula for calculating the divergence value is expressed as:

[0013]

[0014] Among them, D KL This represents the divergence value; old_p represents the offline policy; current_p represents the current policy.

[0015] In one possible implementation, the decision-making strategy is specifically expressed as:

[0016]

[0017] Wherein, use_p represents the decision strategy, and τ represents the first threshold.

[0018] In one possible implementation, the process of fusing the nearest offline strategy with the new strategy using a fusion formula to obtain the parameters of the fusion strategy includes:

[0019] The nearest offline strategy and the decision strategy are fused using a gradient to determine the fusion gradient.

[0020] The parameters of the nearest offline strategy and the parameters of the decision strategy are fused to determine the fusion parameters;

[0021] The direction of the fusion parameters is determined based on the fusion gradient, thereby obtaining the parameters of the fusion strategy. In one possible implementation, the gradient fusion is specifically represented as:

[0022] g t '=α·g t +(1-α)·h t

[0023] Among them, g t h represents the gradient of the nearest offline strategy at the current time. t The gradient of the decision strategy at the current time is represented by α; α represents a hyperparameter used to balance the ratio of the gradient of the nearest offline strategy to the gradient of the decision strategy.

[0024] In one possible implementation, the parameter fusion is specifically represented as:

[0025] π(a|s;θ)=β·π offline(a|s;θ offline )+(1-β)·π offline (a|s;θ online )

[0026] Where π(a|s;θ) represents the parameters of the fusion strategy; π offline Represents the nearest offline strategy; θ offline The parameter θ represents the nearest offline strategy. online The parameters represent the decision-making strategy; α and β both represent hyperparameters, which are used to balance the ratio of the nearest offline strategy and the decision-making strategy, and s represents the current state.

[0027] In one possible implementation, updating the initial parameters specifically includes:

[0028] Calculate the gradient value of the nearest offline policy and the gradient value of the decision policy respectively;

[0029] The initial parameters are updated based on the ratio between the gradient value of the nearest offline policy and the gradient value of the decision policy.

[0030] In one possible implementation, the loss function is specifically expressed as:

[0031]

[0032] Where b represents the number of samples sampled from the policy replay pool; Q tot (Γ,u,s;θ) represents an update method consistent with traditional DQN; The update method is indicated as follows:

[0033]

[0034] Among them, Q tot (Γ',u',s';θ - ) represents the target network, θ - Indicates the parameter value.

[0035] In one possible implementation, the offline strategy is specifically expressed as:

[0036] P={policy,episode,gred,parameters}

[0037] Where, policy represents the policy set {p1, p2, ... p} in the policy replay pool. n}, p i This represents the state and action distribution of the i-th offline policy; episode = {d1, d2, ..., d...} n}, di Indicates offline strategy p i Offline data; gred = {g1, g2, ..., g n}, g i Indicates offline strategy p i The gradient value of the loss function calculated from the parameters; parameters = {θ1, θ2, ..., θ n}, θ i Indicates offline strategy p i The initial parameters.

[0038] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0039] This invention employs a multi-agent cooperative adversarial method using offline policy reuse. This method includes: acquiring the current policy and all offline policies in the policy replay pool, along with initial parameters; calculating the divergence values ​​of the current policy and the offline policies; acquiring and saving the minimum divergence value and the nearest offline policy corresponding to the minimum divergence value; reusing offline policy data by calculating the degree of difference between offline policies; determining whether the minimum divergence value is greater than or equal to a first threshold; if so, interacting the current policy with environmental data to generate a new policy, identifying the new policy as the decision policy; otherwise, identifying the nearest offline policy as the decision policy. Policy reuse saves time in the interaction between the current policy and the environment and reduces computational load. It also involves determining whether the decision policy and the nearest offline policy are the same; if different, first fusing the nearest offline policy with the new policy using a fusion formula to obtain the parameters of the fused policy, and then calculating the loss value using a loss function; if the same, calculating the loss value using the loss function on the nearest offline policy and environmental data. Finally, it determines whether the decision policy and the nearest offline policy are the same; if different, proceeding to the policy fusion stage improves the learning efficiency of the policy. This method uses an offline policy parameter fusion approach to combine the offline policy with the current policy to generate a new policy, improving the policy's robustness and generalization ability. It checks if the loss value is less than or equal to a second threshold; if so, it outputs the policy from the policy replay pool; otherwise, it adds the new policy to the replay pool, updates the initial parameters, and continues calculation until the loss value is less than or equal to the second threshold. This effectively solves the problems of low training sample utilization, high demand, and long training time in existing technologies, thereby achieving the reuse of offline policy data and saving training time for the current policy's interaction with the environment. Furthermore, when policy reuse is not used, policy parameters are fused by fusing the current gradient information and network parameters with the gradient information of the policy with the smallest difference in the replay pool, thus improving the policy's learning efficiency. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating the steps of a multi-agent cooperative adversarial method for strategy reuse provided in this embodiment of the invention;

[0042] Figure 2 A flowchart illustrating the application of the method provided by this invention in an embodiment of the invention;

[0043] Figure 3 This is a 3m schematic diagram of a micro-operation scenario provided in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of a micro-operation scenario 2s3z provided in an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of a micro-operation scenario of 2s vs 1sc provided in an embodiment of the present invention;

[0046] Figure 6 This is a comparison chart of the variance values ​​of the returns in a 3m scenario provided in an embodiment of the present invention;

[0047] Figure 7 This is a comparison chart of returns in a 3m scene provided in an embodiment of the present invention;

[0048] Figure 8 This is a comparison chart of win rates in a 3m scenario provided by an embodiment of the present invention;

[0049] Figure 9 This is a comparison chart of the variance values ​​of returns in a 2s3z scenario provided in an embodiment of the present invention;

[0050] Figure 10 This is a comparison chart of returns in a 2s3z scenario provided in an embodiment of the present invention;

[0051] Figure 11 This is a win rate comparison chart for the 2s3z scenario provided in an embodiment of the present invention;

[0052] Figure 12 This is a comparison chart of the variance values ​​of the returns in the 2s_vs_1sc scenario provided in this embodiment of the invention;

[0053] Figure 13 This is a comparison chart of returns in the 2s_vs_1sc scenario provided in an embodiment of the present invention;

[0054] Figure 14 This is a win rate comparison chart for the 2s_vs_1sc scenario provided in an embodiment of the present invention;

[0055] Figure 15 This is a comparison chart of the win rates of various algorithms in the StarCraft II environment 3m provided in this embodiment of the invention;

[0056] Figure 16 This is a comparison chart of the 3m algorithm rewards in the StarCraft II environment provided in this embodiment of the invention;

[0057] Figure 17 This is a comparison chart of the win rates of various algorithms in the StarCraft II environment 2s3z provided in this embodiment of the invention;

[0058] Figure 18 This is a comparison chart of the rewards of various algorithms in the StarCraft II environment 2s3z provided in this embodiment of the invention;

[0059] Figure 19 This is a comparison chart of the win rates of various algorithms in the StarCraft II environment 2s_vs_1sc provided in this embodiment of the invention;

[0060] Figure 20 This is a comparison chart of the algorithm returns for each algorithm in the StarCraft II environment 2s_vs_1sc provided in this embodiment of the invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0062] Because the COMA algorithm assumes that agent policies are independent, each Actor network only considers its own state information when guiding a single agent to calculate its policy, ignoring the state information of other agents. This assumption leads to agents' policies not fully utilizing the state information of other agents in cooperative tasks, thus affecting the policy's performance. Furthermore, in each round of policy evaluation, each agent first uses the Critic network to evaluate the quality of its current policy, and then uses this evaluation value to update the Actor network. In this process, each agent's policy is not independent, and the Critic network's updates are affected by the policies of other agents, resulting in unstable Critic network training and impacting the final policy performance.

[0063] Furthermore, the COMA algorithm requires the Q-value of each agent to calculate the advantage function, thus necessitating a large number of samples to update each agent's policy. Each agent needs to interact with other agents to obtain sufficient samples, leading to a high training sample requirement. In this situation, the algorithm's training efficiency becomes extremely low, causing not only instability but also potential algorithm crashes. Therefore, addressing the high sample requirement issue in the COMA algorithm is crucial.

[0064] This invention provides a multi-agent cooperative adversarial method for offline policy reuse, such as... Figure 1 As shown, the method includes the following steps S101 to S104.

[0065] In a specific embodiment provided by the present invention, steps S101 to S102 occur in the Actor network.

[0066] S101: Obtain the current policy and all offline policies in the policy replay pool, along with their initial parameters. Calculate the divergence values ​​of the current policy and the offline policies respectively. Obtain and save the minimum divergence value and the nearest offline policy corresponding to the minimum divergence value. The offline policy is specifically represented as follows:

[0067] P={policy,episode,gred,parameters}

[0068] Where, policy represents the policy set {p1, p2, ... p} in the policy replay pool. n}, p i This represents the state and action distribution of the i-th offline policy; episode = {d1, d2, ..., d...} n}, d i Indicates offline strategy p i Offline data; gred = {g1, g2, ..., g n}, g i Indicates offline strategy p i The gradient value of the loss function calculated from the parameters; parameters = {θ1, θ2, ..., θ n}, θ i Indicates offline strategy p i The initial parameters.

[0069] In step S101, the specific formula for calculating the divergence value is expressed as follows:

[0070]

[0071] Among them, D KLThis represents the divergence value; old_p represents the offline policy; current_p represents the current policy.

[0072] S102, determine whether the minimum divergence value is greater than or equal to the first threshold. If yes, then interact with the current policy and environmental data to generate a new policy, and determine the new policy as the decision policy; otherwise, determine the nearest offline policy as the decision policy. The specific decision policy is expressed as follows:

[0073]

[0074] Where use_p represents the decision strategy and τ represents the first threshold.

[0075] When training can continue using policies similar to the current policy from the policy replay pool, the time required for the current policy to interact with the environment and generate a new policy is significantly reduced. Furthermore, the similarity can be set according to the training objective, achieving the training goal while reducing computational load and the time required for interaction between the current policy and the environment. Applying the already trained policy to a new agent also reduces the number of samples required.

[0076] Step S103 occurs in the Critic network.

[0077] S103. Determine whether the decision strategy is the same as the nearest offline strategy. If they are different, first use the fusion formula to fuse the nearest offline strategy with the new strategy to obtain the parameters of the fused strategy. Then use the loss function to calculate the parameters of the fused strategy to obtain the loss value. If they are the same, use the loss function to calculate the loss value between the nearest offline strategy and the environmental data.

[0078] In step S103, the nearest offline policy and the new policy are fused using a fusion formula to obtain the parameters of the fused policy, specifically including:

[0079] (1) Perform gradient fusion between the nearest offline policy and the decision policy to determine the fusion gradient. Gradient fusion is specifically expressed as:

[0080] g t '=α·g t +(1-α)·h t

[0081] Among them, g t h represents the gradient of the nearest offline policy at the current time. t α represents the gradient of the decision policy at the current time; α represents a hyperparameter used to balance the gradient of the nearest offline policy and the gradient of the decision policy.

[0082] (2) The parameters of the nearest offline strategy are fused with the parameters of the decision strategy to determine the fusion parameters. Parameter fusion is specifically expressed as follows:

[0083] π(a|s;θ)=β·π offline (a|s;θ offline )+(1-β)·π offline (a|s;θ online )

[0084] Where π(a|s;θ) represents the parameters of the fusion strategy; π offline Indicates the nearest offline strategy; θ offline The parameter θ represents the nearest offline strategy. online The parameters represent the decision-making strategy; α and β are both hyperparameters, which are used to balance the proportion of the nearest offline strategy and the decision-making strategy; s represents the current state.

[0085] (3) Determine the direction of the fusion parameters based on the fusion gradient to obtain the parameters of the fusion strategy.

[0086] In step S103, the loss function is specifically expressed as follows:

[0087]

[0088] Where b represents the number of samples sampled from the policy replay pool; Q tot (Γ,u,s;θ) represents an update method consistent with traditional DQN; The update method is indicated as follows:

[0089]

[0090] Among them, Q tot (Γ',u',s';θ - ) represents the target network, θ - Indicates the parameter value.

[0091] In step S103, without using data reuse, the Critic network determines whether the decision-making policy is the same as the nearest offline policy. If they are different, the process proceeds to the policy fusion stage, thereby improving the learning efficiency of the policy. The offline policy parameter fusion method is used to combine the offline policy with the current policy to generate a new policy, improving the policy's robustness and generalization ability.

[0092] S104. Determine whether the loss value is less than or equal to the second threshold. If yes, output the strategy in the strategy replay pool. If no, put the new strategy into the strategy replay pool, update the initial parameters and continue to calculate until the loss value is less than or equal to the second threshold.

[0093] In step S104, the initial parameters are updated, specifically including:

[0094] (1) Calculate the gradient values ​​of the nearest offline policy and the decision policy respectively.

[0095] (2) The initial parameters are updated according to the ratio between the gradient value of the nearest offline policy and the gradient value of the decision policy.

[0096] In a specific embodiment provided by the present invention, such as Figure 2 The specific embodiment of the method proposed in this invention shown involves the following steps when training the Actor network.

[0097] (1) Initialize environment data, Actor network and Critic network, policy replay pool and initial parameters.

[0098] (2) Obtain the current policy of all single agents in the Actor network and Critic network, and put the current policies into the policy replay pool.

[0099] (3) Determine if the policy replay pool is full. If so, start calculating the divergence between the current policy of a single agent and all offline policies in the policy replay pool. The nearest offline policy corresponding to the smallest divergence value.

[0100] (4) A specific first threshold provided by this invention is 0.05. It is determined whether the divergence value of all offline policies in the policy replay pool is less than the first threshold. If so, the nearest offline policy is determined as the decision policy. If not, the current policy is interacted with the environmental data to generate a new policy, which is then used as the decision policy and saved.

[0101] (5) Evaluate the incoming decision strategy in the Critic network, calculate the counterfactual baseline, and return the actions and Q values ​​of each agent.

[0102] (6) Determine whether the decision strategy is the nearest offline strategy. If yes, calculate the loss value according to the loss function. If not, merge the nearest offline strategy in step (3) with the new strategy.

[0103] (7) Calculate the fusion gradient and fusion parameters of the nearest offline policy and the new policy, and determine the direction of the fusion parameters based on the fusion gradient to obtain the parameters of the fusion policy.

[0104] (8) Determine whether the loss value obtained by the loss function meets the termination condition, that is, determine whether the loss value is less than or equal to the second threshold. If yes, output the strategy in the strategy replay pool; if no, put the new strategy into the strategy replay pool, update the initial parameters and continue to calculate until the loss value is less than or equal to the second threshold.

[0105] The counterfactual baseline in step (5) is represented as follows:

[0106]

[0107] in, This represents the expected Q-value of the agent. This formula requires traversing all actions in the agent's action space and keeping other agent actions u in mind. -a The counterfactual baseline A remains unchanged. a The calculation of (s,u) can distinguish the quality of each agent's policy, that is, the contribution of a single policy to the overall policy. a The update of (s,u) is performed periodically using a temporal difference method. Using the counterfactual baseline as the objective function, the COMA policy gradient is calculated as follows:

[0108]

[0109] Where k represents the number of iterations of the algorithm, θ k This represents the parameters of the network at the k-th iteration.

[0110] In step (8), the loss function is expressed as:

[0111]

[0112] Where b represents the number of samples sampled from the policy replay pool; Q tot (Γ,u,s;θ) represents an update method consistent with traditional DQN. The update method is indicated as follows:

[0113]

[0114] Among them, Q tot (Γ',u',s';θ - ) represents the target network, θ - This represents the target network parameters.

[0115] In a specific embodiment of this invention, the multi-agent experimental environment (full name: StarCraft Multi-Agent Challenge, abbreviated as SMAC) is a game-based multi-agent reinforcement learning environment. In this environment, players control a team of multiple units to fight against an opponent. Players need to make various tactical decisions, including exploring the map, collecting resources, building bases and barracks, training troops, and attacking enemies. The opponent will also make similar decisions and constantly adjust tactics to counter the player. The high complexity and randomness of this environment pose a significant challenge to multi-agent reinforcement learning algorithms. Therefore, the SMAC environment has become an important benchmark for evaluating the performance of multi-agent reinforcement learning algorithms. Experimenting in this environment using the method provided by this invention allows for a more objective evaluation of the performance of the method.

[0116] SMAC is a micro-operational environment built on PySC2 (StarCraft II Learning Environment) and the StarCraft II API. It incorporates advanced multi-agent algorithms such as COMA, QTRAN, and MAVEN.

[0117] The experimental environment of this invention employs a microscopic manipulation map, in such a way as... Figure 3 , Figure 4 and Figure 5 Experiments were conducted in the scenario shown, with the following operating environment: 16GB RAM, 512GB hard drive, 2.60GHz 64-bit processor, Windows 10 operating system, and Python 3.7. The experiments first compared the proposed method with the original COMA algorithm using environments 3m, 2s3z, and 2s vs 1sc. Finally, the method was compared and validated with current advanced multi-agent reinforcement learning algorithms such as MAVEN, QMIX, QTRAN, and VDN.

[0118] In the method provided by this invention, the global reward of the multi-agent system obtained in each complete policy iteration during the training process is recorded. The stationarity of the algorithm during training is reflected by statistically analyzing the variance between the global rewards of each round. This indicator is calculated using the following formula:

[0119]

[0120] Where, r t ' ot This represents the average global reward across the entire training generation.

[0121] This invention uses the global reward returned by the environment as another metric for algorithm comparison. This reward is set based on the average reward of the algorithm-guided multi-agent system over n training steps, expressed as:

[0122]

[0123] Where reward = del_en + del_de - del_ally, del_en represents the cumulative damage reward inflicted on the enemy by our units, del_de represents the reward for eliminating the enemy, and del_ally represents the reward for our agent's death. In addition, the win rate of the multi-agent system performing adversarial tasks is calculated at intervals of evaluation training steps (evaluate_per_epoch). The win rate (win_rate) is calculated as follows:

[0124]

[0125] `win_flag` represents the number of wins within the `evaluate_per_epoch` training step. The specific parameter settings for the method provided in this invention compared to other algorithms are shown in Table 1.

[0126] Table 1 Experimental Algorithm Parameter Table

[0127]

[0128] First, the method provided by this invention is compared with the original COMA algorithm. Figure 6 , Figure 9 , Figure 12 Statistical analysis was performed on the return variance values ​​of the two algorithms after 20,000 generations of training in the 3m, 2s3z, and 2s_vs_1sc scenarios. It can be seen that in all three different scenarios, the return variance value of the method provided in this invention is smaller than that of the original COMA algorithm, indicating that the policy stability of the improved algorithm has been effectively improved. Figure 7 , Figure 10 , Figure 13 The cumulative returns of the two algorithms are compared in three different scenarios. The method provided by this invention can improve the cumulative returns on the basis of COMA and enhance the performance of the algorithm, especially in the 2s3z scenario. Figure 8 , Figure 11 , Figure 14The comparison of win rates is shown. The improved algorithm proposed in this invention still has a higher task completion rate than the original COMA algorithm. This indicates that the method provided in this invention not only improves the stability of policy training but also accelerates the training efficiency. Specifically, the performance of the method provided in this invention is not significantly different from the original algorithm in the first 3000 generations. This is because the offline replay pool is initially empty, requiring expansion of the offline policies in the early stages. Table 2 shows a comparison of the time taken by the two algorithms to run for 20,000 generations and the maximum variance of the reward. The training time of the method provided in this invention is significantly less than that of the original COMA algorithm. This demonstrates that the policy replay pool saves a considerable amount of time for agent-environment interaction, further highlighting the effectiveness of the improved algorithm.

[0129] Table 2. Maximum Variance of Algorithm Training Time and Cumulative Reward

[0130]

[0131] Figure 15 , Figure 16 The table shows the win rate and global reward of the method provided in this invention compared with QMIX, VDN, QTRAN, MAVEN, and COMA algorithms after 20,000 training steps in a 3m environment. Table 3 compares the detailed experimental data in the 3m scenario. While the win rate of the method provided in this invention still lags behind value-based algorithms such as QMIX and VDN, the cumulative reward of the method provided in this invention is not significantly different from other advanced algorithms. Table 3 shows that although the method provided in this invention does not achieve optimal results in any single aspect, it narrows the gap with VDN and MAVEN algorithms compared to the original COMA algorithm, and even outperforms QMIX in terms of maximum reward, average reward, and highest win rate. Figure 17 , Figure 18 This paper presents a comparison of the win rate and global reward of the method provided in this invention with the original QMIX, VDN, QTRAN, MAVEN, and COMA algorithms after 20,000 training steps in a 2s3z environment. Table 4 compares the detailed experimental data in the 2s3z scenario. Due to the poor performance of the COMA algorithm in this scenario, the method provided in this invention still lags significantly behind algorithms such as QMIX, VDN, and MAVEN. However, the win rate of the algorithm still exceeds that of QTRAN, and according to... Figure 18 As can be seen from the specific data in Table 4, the average return and convergence of the method provided by this invention are better than other algorithms, and the maximum return is second only to the VDN algorithm.

[0132] Table 3 Comparison of experimental results for each algorithm in a 3m environment.

[0133]

[0134] Table 4 Comparison of experimental results for each algorithm in environment 2s3z

[0135]

[0136]

[0137] and Figure 19 , Figure 20 The table presents a comparison of the win rate and global reward obtained by the method provided in this invention with the original QMIX, VDN, QTRAN, MAVEN, and COMA algorithms after 20,000 training steps in the 2s_vs_1sc environment. Table 5 compares the detailed experimental data in the 2s_vs_1sc scenario. It can be seen that the maximum reward and average win rate of the method provided in this invention are second only to VDN, and the highest win rate is on par with other optimal algorithms, while the convergence speed is the fastest.

[0138] Table 5 Comparison of experimental results for each algorithm in environment 2s_vs_1sc

[0139]

[0140] This invention proposes a multi-agent cooperative adversarial method for offline policy reuse, aiming to address the problems of low policy utilization and unstable policy training in the COMA algorithm. The innovation of this invention lies in combining policy reuse and policy fusion, which can fully utilize offline policy information to improve sample utilization while avoiding instability issues during new policy training. Experimental results in the SMAC environment show that the method provided by this invention significantly improves policy utilization, learning speed, and convergence performance compared to the COMA algorithm, and narrows the performance gap with other advanced multi-agent reinforcement learning algorithms, even outperforming them in some aspects. This indicates that the offline policy reuse method proposed in this invention has high application value and broad development prospects in multi-agent reinforcement learning. Finally, the algorithm proposed in this invention was validated in the SMAC environment, and this validation mode can be extended to other multi-agent reinforcement learning environments for further experiments and applications, while also providing a reference for subsequent related research.

[0141] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this invention can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0142] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A multi-agent cooperative adversarial method for offline strategy reuse, characterized in that, include: Obtain the current policy and all offline policies in the policy replay pool, as well as the initial parameters. Calculate the divergence value between the current policy and the offline policies respectively. Obtain and save the minimum divergence value and the nearest offline policy corresponding to the minimum divergence value. Determine whether the minimum divergence value is greater than or equal to the first threshold. If so, interact the current strategy with the environmental data to generate a new strategy, and determine the new strategy as the decision strategy. If not, then the nearest offline strategy is determined as the decision strategy; Determine whether the decision strategy is the same as the nearest offline strategy. If they are different, perform gradient fusion between the nearest offline strategy and the decision strategy to determine the fusion gradient. The parameters of the nearest offline strategy and the parameters of the decision strategy are fused to determine the fusion parameters; the direction of the fusion parameters is determined according to the fusion gradient to obtain the parameters of the fusion strategy; and then the loss function is used to calculate the parameters of the fusion strategy to obtain the loss value. If they are the same, the loss function is used to calculate the loss value between the nearest offline policy and the environmental data; wherein, the decision policy is specifically expressed as: ,in, This indicates the decision-making strategy. This represents the first threshold; Determine whether the loss value is less than or equal to the second threshold. If yes, output the strategy in the strategy replay pool. If no, put the new strategy into the strategy replay pool, update the initial parameters and continue to calculate until the loss value is less than or equal to the second threshold.

2. The method according to claim 1, characterized in that, The specific formula for calculating the divergence value is expressed as follows: in, Represents the divergence value; Indicates the offline strategy; Indicates the current strategy.

3. The method according to claim 1, characterized in that, The gradient fusion is specifically represented as follows: in, This represents the gradient of the nearest offline strategy at the current time. This represents the gradient of the decision-making strategy at the current time. This represents a hyperparameter used to balance the ratio of the gradient of the nearest offline policy to the gradient of the decision policy.

4. The method according to claim 1, characterized in that, The parameter fusion is specifically represented as follows: in, Parameters representing the fusion strategy; This refers to the nearest offline strategy; The parameter represents the nearest offline strategy; The parameters representing the decision-making strategy; as well as All represent hyperparameters, which are used to balance the proportion of the nearest offline strategy and the decision strategy. Indicates the current state.

5. The method according to claim 1, characterized in that, The update of the initial parameters specifically includes: Calculate the gradient value of the nearest offline policy and the gradient value of the decision policy respectively; The initial parameters are updated based on the ratio between the gradient value of the nearest offline policy and the gradient value of the decision policy.

6. The method according to claim 1, characterized in that, The loss function is specifically expressed as follows: in, This indicates the number of samples sampled from the replay pool of the stated strategy; The update method is consistent with traditional DQN; The update method is indicated as follows: in, Indicates the target network. Indicates the parameter value.

7. The method according to claim 1, characterized in that, The offline strategy is specifically expressed as follows: in, This represents the set of policies in the policy replay pool. , Indicates the first The state and action distribution of each offline strategy; , Indicates offline strategy Offline data; , Indicates offline strategy The gradient value of the loss function calculated from the parameters; , Indicates offline strategy The initial parameters.

Images