A strategy decision method in multi-agent cooperation and confrontation

Abstract

The application discloses a strategy decision-making method in multi-agent cooperation and confrontation, comprising the following steps: S1, using different reinforcement learning strategies to train a single agent to complete the divide-and-conquer of a single agent strategy, and obtaining a divide-and-conquer decision-making strategy most suitable for a single scene; S2, according to the advantages and disadvantages of each decision-making algorithm under different scenes, completing the fusion of the single agent strategy based on a fusion algorithm of different reinforcement learning strategies, and dynamically selecting the fusion decision-making strategy obtained in the divide-and-conquer in step S1 most suitable for the current scene under the real-time changing scene; S3, using different reinforcement learning strategies to intelligently decompose the task of multi-agent cooperation or confrontation, and completing the divide-and-conquer fusion of the multi-agent cooperation or confrontation, and obtaining the optimal decision-making strategy of the multi-agent cooperation or confrontation. The optimal decision-making strategy of the multi-agent cooperation or confrontation is obtained, and the dynamic fusion of multiple algorithms and the flexible switching of multiple models are realized in the multi-agent cooperation or confrontation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a strategy decision method in multi-agent cooperation and confrontation. BACKGROUND

[0002] In recent years, with the continuous development of science and technology, intelligent robots have been applied in many fields, including industrial production, military, disaster rescue and other aspects, which involve environment perception, dynamic decision and planning, automatic control and other technologies. At the same time, in recent years, there have been various social problems, such as the decrease in the number of labor force, the increase in production cost, the low efficiency of automated production, the incomplete industrial transformation, and the increasingly serious social aging. The application of intelligent agents in the market can effectively alleviate the above problems, therefore, it is necessary to deeply study the related difficulties of intelligent agent technology and further develop the intelligent agent application market. Moreover, intelligent agent application technology is an important indicator to measure the level of national science and technology and industrial automation. Some developed countries in the West have successively promoted the robot industry as a national strategy; in 2017, the Ministry of Science and Technology of China issued the "Intelligent Robot" key special guide, which will further implement the requirements of the "National Medium and Long-term Science and Technology Development Plan Outline (2006-2020)", "Made in China 2025" and other planning.

[0003] For the problem of multi-agent reinforcement learning, a direct solution is to directly apply the single-agent reinforcement learning method to the multi-agent system, that is, each agent regards other agents as factors in the environment, and still updates the strategy through interaction with the environment in the same way as single-agent learning. This learning method is simple and easy to implement, but it ignores the fact that other agents also have decision-making capabilities and the actions of all individuals jointly affect the state of the environment, making it difficult to learn and achieve good results stably.

[0004] In general, there may be competitive relationships (non-cooperative relationships), semi-competitive semi-cooperative relationships (mixed type) or complete cooperative relationships between agents, and in these relationship modes, individuals need to consider the influence of other agent decision behaviors. Because of these influences, the problem of multi-agent cooperation and confrontation becomes extremely complex, and the following four problems need to be overcome in specific algorithms:

[0005] 1) Instability of the environment: while agents are making decisions, other agents are also taking actions; the change of the environment state is related to the joint action of all agents;

[0006] 2) Limitation of information acquisition by agents: global information may not be available, and agents can only obtain local observation information, but cannot obtain observation information, actions and rewards of other agents;

[0007] 3) Individual goal consistency: the goal of each agent may be the optimal global return; or the optimal of each local return;

[0008] 4) Scalability: in a large-scale multi-agent system, high-dimensional state space and action space are involved, which has certain requirements for model expression ability and hardware computing power in real scenarios.

[0009] Multi-agent reinforcement learning is an important research direction combining reinforcement learning and multi-agent learning, focusing on the sequential decision-making problem of multiple agents. In the future, research on MARL (multi-agent reinforcement learning) (including theoretical and application levels) still needs to solve many problems, including the supplement and perfection of the theoretical system, the reproducibility of the method, the training and computing amount of the model parameters, the safety and robustness of the model, etc.

[0010] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0011] In order to solve the technical problem that the existing single-agent reinforcement learning method is directly applied to the multi-agent system, ignoring the decision-making ability of other agents, and the actions of all individuals jointly affect the state of the environment, making it difficult to stably learn and achieve good results, the purpose of the present application is to provide a strategy decision method in multi-agent cooperation and confrontation.

[0012] The present application is realized by the following technical solutions:

[0013] A strategy decision method in multi-agent cooperation and confrontation, comprising the following steps:

[0014] S1, training a single agent using different reinforcement learning strategies to complete the divide-and-conquer of single-agent strategies, and obtaining a divide-and-conquer decision strategy most suitable for a single scenario;

[0015] S2, based on the advantages and disadvantages of each decision algorithm under different scenarios, completing the fusion of single-agent strategies based on the fusion algorithm of different reinforcement learning strategies, and dynamically selecting the fusion decision strategy obtained in step S1 that is most suitable for the current scenario under the real-time changing scenario;

[0016] S3, using different reinforcement learning strategies to intelligently decompose the task of multi-agent cooperation or confrontation, and assigning single agents equipped with the divide-and-conquer decision strategy of step S1 and the fusion decision strategy of step S2 to complete the divide-and-conquer fusion of multi-agent cooperation or confrontation, and obtaining the optimal decision strategy of multi-agent cooperation or confrontation.

[0017] In some embodiments, in steps S1, S2 and S3, the different reinforcement learning strategies include an adversarial rule value estimation strategy, a neural network value estimation strategy, and a Monte Carlo tree search strategy.

[0018] In some embodiments, step S2 includes the following steps:

[0019] S2-1, a single agent decision produces an action, which is then applied to the environment, and the environment returns a decision state and a reward to the agent;

[0020] S2-2, after the agent is trained with different reinforcement learning strategies respectively, for the same decision state, the neural networks of different reinforcement learning strategies return the value estimation of the current decision state;

[0021] S2-3, after the value estimation of the sub-algorithm is fused using the value covariance, the optimal fused decision strategy is output.

[0022] In some embodiments, step S2-3 includes the following steps:

[0023] S2-3-1, a neural network is designed to calculate the covariance threshold c T based on the current decision state and the reward δR;

[0024] S2-3-2, according to the value of the adversarial rule value estimation V rule and the value of the neural network value estimation V net , the value covariance of the current two strategies is calculated;

[0025] S2-3-3, according to the covariance threshold C T of step S2-3-1 and the value covariance of step S2-3-2, the value of the fusion factor λ is determined, and the values of the two strategies are fused;

[0026] S2-3-4, according to the fused value obtained in step S2-3-3, and combining the trained Monte Carlo tree search strategy, the optimal decision strategy is output.

[0027] In some embodiments, in step S2-3-1, the covariance threshold c T is obtained based on a policy gradient reinforcement learning algorithm, specifically: at each time step, the policy network μ(S; θ) produces an action a, the agent performs the action to obtain a reward R, and an output end outputs a covariance threshold c T ; wherein S is the state and θ is the parameter of the action network.

[0028] In some embodiments, in step S2-3-1, the reward δR is obtained by multiplying the reward R used by the sub-policy of the divide-and-conquer fusion controller by a conversion factor δ; the function of the conversion factor δ is:

[0029]

[0030] Where μ is the conversion factor in the early stage of training, t s The time required for the basic completion of training the sub-policy controller;

[0031] The reward δR is learned using DQN, and the optimization objective of its state-action value network is as follows:

[0032]

[0033] Q θ (s,a in (This is a state-action value network.) Let θ be the target state action value function. These are the network parameters, γ is the discount factor, s and s′ correspond to the current state and the next state, respectively, and a in R is the action currently taken, and R is the reward obtained after taking action a in state s.

[0034] In some embodiments, in step S2-3-2, the adversarial rule value estimation V rule Let its estimated value when the input state vector is S be V. rule (S);

[0035] The adversarial rules are defined as follows:

[0036]

[0037] in, This represents the set of functions equivalent to adversarial rules. Representing the state space A subset of is The union of the domains of all elements in the set. It is a one-dimensional real number space.

[0038] In some embodiments, in step S3, the fusion of multi-agent cooperation or adversarial cooperation adopts a fusion method based on Nash equilibrium, expressed as:

[0039]

[0040] Where Agents represents all agents, i and k represent the i-th and k-th agents respectively, π represents the policy, and * represents the Nash equilibrium policy. This means that at the Nash equilibrium, only the value function of the policy of the i-th agent is changed. is the value function of the ith agent under the Nash equilibrium strategy; when the Nash equilibrium strategy of the stage game of each state is combined into a strategy of an agent in a dynamic environment, and the Q value function in the stage game of each state is updated by constantly interacting with the environment; in this special case, the Nash strategy using the fusion method based on Nash equilibrium is:

[0041]

[0042] wherein A i (i = 1, …, n) represents the action space of the ith agent, a i (i = 1, …, n) represents the action of the ith agent, is the probability of the ith agent taking action a i under the Nash equilibrium in state s.

[0043] In some embodiments, the fusion method based on Nash equilibrium includes a hard replacement method and a soft replacement method;

[0044] wherein the execution flow of the hard replacement method is: 1) for a certain fixed time t, if t > T max , the new strategy must be executed according to the optimal strategy at this moment, and t is reset to 0, and cnt is also reset to 0, and is returned to a new period; 2) otherwise, it is judged whether π * and π x are equal, if not equal, then cnt = cnt + 1 (π * ≠ π x ) is calculated; if equal, return 1); if cnt = cnt + 1 (π * ≠ π x ) is calculated, cnt > C max , the optimal strategy π * at this moment is immediately executed, and t is reset to 0, and cnt is also reset to 0, and is returned to a new period; if cnt ≤ C max , return 1);

[0045] wherein the sampling time is T; the current execution strategy is π x ; the time when the strategy starts to be executed is t = 0, and the optimal strategy at this moment is π max , and the longest time when it is executed is T max (meeting T * % T = 0); in this execution period, the total number of times when the optimal strategy π x at each moment is different from the current actual execution strategy π max is counted as cnt, and the upper limit of the count is C .

[0046] In some embodiments, the execution flow of the soft replacement method is: at time t, for a certain policy π, the probability of taking the value and the probability p of taking this policy at the moment are related to the probability of taking this policy at the previous 1 / (1-τ) time; t

[0047] For the policy π i The specific expression is:

[0048]

[0049] Thus, at time t,

[0050] The optimal policy of the policy at this moment is

[0051] Wherein, p' i (t) represents the value of the policy π i at the i-th time, which is manually calculated, and the normalized form of p'1(t), p'2(t) is taken, to obtain and

[0052] The beneficial effects of the present application compared with the prior art include:

[0053] The present application trains a single agent by adopting different reinforcement learning strategies, that is, "divide and conquer", and on this basis, analogizes the fusion method in "ensemble learning", and according to the advantages and disadvantages of each decision algorithm in different scenarios, a fusion algorithm based on different reinforcement learning strategies is used to complete the fusion of the single agent strategy, and the decision strategy obtained by the divide and conquer in the most suitable current scenario is dynamically selected in the real-time changing scenario, and then different reinforcement learning strategies are used to intelligently decompose the single agent equipped with the divide and conquer fusion decision strategy to complete the divide and conquer fusion of the multi-agent cooperation or confrontation, so as to obtain the optimal decision strategy of multi-agent cooperation or confrontation, and the dynamic fusion of multiple algorithms and the flexible switching of multiple models can be realized in multi-agent cooperation or confrontation. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of a strategy decision method in multi-agent cooperation and confrontation according to an embodiment of the present application;

[0055] Figure 2 is a single agent divide and conquer decision strategy diagram in an embodiment of the present application;

[0056] Figure 3 is a single agent fusion decision strategy diagram in an embodiment of the present application;

[0057] ​Figure 4 is a single-agent divide-and-conquer fusion decision strategy schematic diagram in an embodiment of the present application;

[0058] Figure 5 is a multi-agent cooperative or antagonistic divide-and-conquer fusion decision strategy schematic diagram in an embodiment of the present application;

[0059] Figure 6 is a threshold value estimation module flowchart of a reinforcement learning algorithm based on a policy gradient in an embodiment of the present application;

[0060] Figure 7 is a conversion factor change over time schematic diagram in an embodiment of the present application;

[0061] Figure 8 is a hard replacement flowchart schematic diagram of a fusion method based on Nash equilibrium in an embodiment of the present application;

[0062] Figure 9 is a schematic diagram of a probability value for a certain policy at time t in an embodiment of the present application;

[0063] Figure 10 is a soft replacement flowchart schematic diagram of a fusion method based on Nash equilibrium in an embodiment of the present application;

[0064] Figure 11 is a main flowchart in a model application module in an embodiment of the present application;

[0065] Figure 12a is an average reward schematic diagram of a single agent using an optimal antagonistic strategy against 5k games throughout the entire process in an embodiment of the present application;

[0066] Figure 12b is an average reward schematic diagram of a single agent using an optimal antagonistic strategy against another 5k game throughout the entire process in an embodiment of the present application;

[0067] Figure 12c is an average reward schematic diagram of a single agent using an optimal antagonistic strategy against another 10k game throughout the entire process in an embodiment of the present application;

[0068] Figure 12d is a win rate schematic diagram of a single agent using an optimal antagonistic strategy throughout the entire process in an embodiment of the present application;

[0069] Figure 13a is a negative rate schematic diagram of a single agent using a plurality of antagonistic strategies fused in different states against 5k games in an embodiment of the present application;

[0070] Figure 13b is a flat rate schematic diagram of a single agent using a plurality of antagonistic strategies fused in different states against 5k games in an embodiment of the present application;

[0071] Figure 13cis a 5k game winning rate diagram of a single agent in different states after the fusion of various countermeasures in the embodiment of the application;

[0072] Figure 13d is a 5k game reward diagram of a single agent in different states after the fusion of various countermeasures in the embodiment of the application;

[0073] Figure 14a is a 3k game negative rate diagram of a plurality of agents in the embodiment of the application after dynamically and optimally selecting the strategy executed at each moment based on the divide-and-conquer fusion algorithm;

[0074] Figure 14b is a 3k game flat rate diagram of a plurality of agents in the embodiment of the application after dynamically and optimally selecting the strategy executed at each moment based on the divide-and-conquer fusion algorithm;

[0075] Figure 14c is a 3k game winning rate diagram of a plurality of agents in the embodiment of the application after dynamically and optimally selecting the strategy executed at each moment based on the divide-and-conquer fusion algorithm;

[0076] Figure 14d is a 3k game average reward diagram of a plurality of agents in the embodiment of the application after dynamically and optimally selecting the strategy executed at each moment based on the divide-and-conquer fusion algorithm. DETAILED DESCRIPTION

[0077] The application will be further described below with reference to the drawings and in conjunction with preferred embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0078] It should be noted that the left, right, up, down, top, bottom and other orientation terms in the embodiments are only relative concepts or are referenced to the normal use state of the product, and should not be considered as limiting.

[0079] The embodiment of the application proposes divide-and-conquer and fusion of intelligent decision-making in a complex scene, in which the divide-and-conquer is used to train the decision-making strategy most suitable for a single scene, and the fusion is to dynamically select the decision-making strategy most suitable for the current scene in the divide-and-conquer under the real-time changing scene. A divide-and-conquer fusion method based on reinforcement learning is also proposed to solve the intelligent decision-making problem in the current application of multi-agent collaboration and confrontation. A threshold estimation based on the policy gradient reinforcement learning algorithm is designed.

[0080] The embodiment of the application also proposes a divide-and-conquer fusion algorithm to solve the decision-making problem of a single agent in a complex confrontation scene. And on the basis of using the divide-and-conquer fusion method to optimize the intelligent decision-making of a single agent to the maximum, the algorithm system is extended to the intelligent decision-making in multi-agent collaboration and confrontation.

[0081] Embodiments of the present application relate to the intelligent decision-making problem of multi-agent in cooperative work and cooperative confrontation. In the process of multi-agent cooperative work and confrontation, the situation is more complex: in this process, it is necessary to adjust which cooperative or confrontation strategy to use according to the state of the opponent and our own agent; it is also necessary to adjust the cooperative or confrontation strategy according to the strategy style and model of our side or the opponent. Therefore, when training by using various algorithms, different algorithms will inevitably have different performance capabilities in different situations, and it is hoped that the advantages of each strategy can be fully played in different models, so that not only a single strategy can play a role in a cooperative or confrontation.

[0082] To achieve the above effects, embodiments of the present application train multiple strategies for a single agent using reinforcement learning algorithms, that is, "divide and conquer", and on this basis, analogize the fusion method in "ensemble learning", according to the advantages and disadvantages of each decision-making algorithm in different scenarios, embodiments of the present application also train a fusion algorithm based on reinforcement learning, so that the algorithm after fusion can dynamically use which strategy in a certain situation according to the scene and situation to achieve the best effect.

[0083] In order to achieve the best fusion effect, so that the performance of each algorithm can be best played, and each algorithm can achieve the maximum complementary advantages, it is necessary to divide the problem into multiple parts at different levels, make control decisions respectively, and finally form a complete decision-making system, so as to achieve the final goal. The intelligent decision-making model used in cooperation or confrontation can achieve dynamic fusion of multiple algorithms and flexible switching of multiple models.

[0084] Embodiments:

[0085] As shown in Figure 1 The strategy decision-making method in multi-agent cooperation and confrontation proposed by the present embodiment includes the following steps:

[0086] S1, training a single agent to complete the division of a single agent strategy using different reinforcement learning strategies, to obtain a division decision-making strategy most suitable for a single scenario;

[0087] S2, according to the advantages and disadvantages of each decision-making algorithm in different scenarios, a fusion algorithm based on different reinforcement learning strategies is used to complete the fusion of a single agent strategy, and dynamically select the fusion decision-making strategy obtained in step S1 that is most suitable for the current scenario in the real-time changing scenario;

[0088] S3, decompose the task of multi-agent cooperation or confrontation by using different reinforcement learning strategies, assign the single agent equipped with the division decision strategy of step S1 and the fusion decision strategy of step S2 to complete the division and fusion of multi-agent cooperation or confrontation, and obtain the optimal decision strategy of multi-agent cooperation or confrontation.

[0089] More specific operation is described as follows:

[0090] 1) Complete the division and fusion of single-agent strategy.

[0091] As shown in Figure 2 , S1, train single agent to complete the division of single-agent strategy by using different reinforcement learning strategies, wherein the division decision strategy most suitable for single scenario is obtained through the interaction of each agent strategy with the simulation environment and the result feedback of the simulation environment to the strategy; wherein the division is reflected in the training of single agent to realize the task by using different reinforcement learning strategies;

[0092] As shown in Figure 3 , S2, complete the fusion of single-agent strategy based on the fusion algorithm of different reinforcement learning strategies according to the advantages and disadvantages of each decision algorithm under different scenarios, dynamically select the fusion decision strategy obtained in step S1 that is most suitable for the current scenario under the real-time changing scenario; wherein the fusion means that based on the fusion algorithm of reinforcement learning, it can autonomously judge which strategy should be used under the current situation to achieve the best effect. So as to achieve the complementary advantages of each algorithm.

[0093] As shown in Figure 3 , the decision state refers to the state of the agent itself and the environment it is in when making decisions, such as the running speed of the agent, its own weight, the relative position with other agents in the environment, etc.; strategy estimation refers to the set of multiple reinforcement learning strategies that the agent can use under the current state; value refers to the feedback that each strategy can bring, including positive feedback and negative feedback; strategy selection is to select the optimal strategy from multiple strategies in strategy estimation according to value feedback, and finally obtain the optimal strategy.

[0094] As shown in Figure 4 , in the sub-algorithm value estimation module, the data samples available to the value estimation scheme based on attack rules are rich, but there are limitations of fuzzy estimation and insufficient accuracy, and the estimation effect of the value estimation scheme based on neural network is accurate, but it is easy to encounter path dependence and dimension disaster challenges. The goal of the fusion strategy scheme is to seek a way to combine the strengths of the two sub-algorithms, so that the performance of the comprehensive scheme can break through the effect limit of the two. Therefore, the fusion strategy scheme adds a covariance calculation module, which uses covariance to estimate the value of the value estimation based on the confrontation rule V rule and the value estimation based on neural network Vnet the difference between the two is quantified, and a more accurate V net as an estimate of the current state value (i.e., the estimated value vector V) is used in the case where the difference between the two is small (i.e., the covariance is small), and a more trustworthy V rule as the current value estimate (i.e., the estimated value vector V) is used in the case where the difference between the two is large (i.e., the covariance is large).

[0095] In order to obtain a better threshold for measuring the size of the covariance, the embodiment uses a method of combining a neural network with a reinforcement learning PG algorithm, and the covariance threshold value is learned and estimated by a neural network fitting covariance threshold value module, and the threshold value and the output variance of the covariance calculation module are fused by a fusion factor calculation module.

[0096] Further, the embodiment uses the value estimate V * (i.e., the output of the sub-algorithm divide-and-conquer fusion value estimate module), uses a Monte Carlo tree search (MCTS) method to make a decision (i.e., select an action), obtains the action that the intelligent fighter should take at the moment, and sets a ladder for the reward according to the attack strategy of our side and the enemy-our attack sequence (i.e., the cooperative confrontation environment). Finally, the action generated by MCTS interacts with the environment to generate the next time state input, and the ladder reward is used to update the neural network parameters in the scheme.

[0097] As to step S2, the following steps are further included:

[0098] S2-1, a single agent decision generates an action, which is then applied to the environment, and the environment returns a decision state and a reward to the agent.

[0099] There are three learning strategies: an adversarial rule value estimation strategy, a neural network value estimation strategy, and a Monte Carlo tree search strategy.

[0100] S2-2, after the agent trains different reinforcement learning strategies respectively, for the same state, the neural networks of different reinforcement learning strategies return value estimates of the current decision state.

[0101] The different reinforcement learning strategies include the three learning strategies in step S2-1.

[0102] S2-3, after the value estimates of the sub-algorithms are fused using the value covariance of the value estimates, the optimal fused decision strategy is output.

[0103] As Figure 4 shown, the embodiment of the application designs a fusion scheme, and as to step S2-3-1, the following steps are further included:

[0104] S2-3-1, design a neural network to calculate the covariance threshold value with the current state and reward;

[0105] S2-3-2, according to the value estimation V rule and neural network value estimation V net The two groups of learning strategies return the value, and the value covariance of the current two groups of strategies is calculated;

[0106] S2-3-3, according to the covariance threshold value C T of step S2-3-1 and the value covariance of step S2-3-2, determine the value of the fusion factor λ (0≤λ≤1), so as to complete the fusion of the values of the two groups of strategies;

[0107] S2-3-4, according to the fused value obtained in step S2-3-3, combine the trained Monte Carlo tree search strategy to output the optimal decision strategy.

[0108] 2) Complete the divide and conquer fusion of multi-agent cooperation or confrontation.

[0109] As Figure 5 shown, S3, in the simulation environment, the task of multi-agent cooperation or confrontation is intelligently divided by using different reinforcement learning strategies, and the single agent equipped with the divide and conquer decision strategy of step S1 and the fusion decision strategy of step S2 is assigned to complete the divide and conquer fusion of multi-agent cooperation or confrontation, and the optimal decision strategy of multi-agent cooperation or confrontation is obtained. The specific operation of step S3 is described as follows:

[0110] The divide and conquer is reflected in the intelligent decomposition of the current task by using different reinforcement learning strategies, and the single agent equipped with the strategy in 1) is assigned to complete the corresponding subtask; The fusion is reflected in the overall framework, which decomposes the problem into multiple parts from different aspects, makes control decisions respectively, and finally forms a complete decision system, so as to achieve the final goal. The intelligent decision model used in cooperation or confrontation can achieve dynamic fusion of multiple algorithms and flexible switching of multiple models.

[0111] The covariance threshold value c T in step S2-3-1 is obtained based on the policy gradient reinforcement learning algorithm, which is described as follows:

[0112] Threshold estimation of policy gradient based reinforcement learning algorithm

[0113] For the covariance threshold value c TThe estimation in this embodiment of the invention is accomplished using a reinforcement learning algorithm that uses a neural network, namely the combination algorithm of the Deterministic Policy Gradient (DPG) algorithm and a deep neural network mentioned in the previous section: the Deep Deterministic Policy Gradient (DDPG) algorithm.

[0114] like Figure 6 As shown, the Deep Deterministic Policy Gradient Algorithm (DDPG) generates an action 'a' at each time step, where the policy network p(S; θ) (where S is the state and θ is the parameter of the action network). The agent then receives a reward R after executing the action. The reward R is the final reward, which consists of a base reward and an additional reward from the reward mechanism. The allocation method for the base reward based on the final reward is relatively simple. This embodiment adopts a more intuitive amortization method, distributing the total value evenly across each time step.

[0115] The additional reward uses a tiered design and can be linear or a two-focus function, such as the sigmoid function. The additional reward incentivizes our attack to gain the upper hand in the confrontation, meaning our attack time is prioritized over the opponent's.

[0116] The difference here is that the output is no longer a value network, but rather a covariance threshold c. T The magnitude of the covariance threshold is related to a set of data input states S obtained every 5 sampling time intervals t in air combat and the size of the distributed reward allocated to each time step in the final reward, where the distributed reward is the reward allocated to each time step in the final reward.

[0117] Therefore, the flowchart of the entire neural network fitting and output is given, as follows: Figure 6 As shown, the reward value in the calculation of the state-action value Q is the reward value at each time step of the previous decision, which is then distributed to the final reward in the previous part. The gradient descent algorithm is then used to minimize the loss function, and finally the covariance threshold c of the final output result is obtained. T .

[0118] This invention also includes two fusion methods: fusion of adversarial strategies in one-on-one agent confrontation and divide-and-conquer fusion in multi-agent cooperation and adversarial situations. Detailed explanations are as follows:

[0119] (1) A method for fusing adversarial strategies in one-on-one agent confrontation:

[0120] The fusion of adversarial strategies in the 1-on-1 adversarial process of this invention embodiment is to fuse the value estimates of the sub-algorithms using the covariance of value estimation. It needs to focus on three issues: obtaining the value estimates V of the two sub-algorithms, calculating the covariance and fusing the V values, and choosing between divide and conquer.

[0121] For convenience of discussion, assume that the adversarial rule value estimate generated by the adversarial rule-based value estimation module is V rule , the neural network value estimate generated by the neural network-based value estimation module is V net , the covariance of V rule and V net is C, and the covariance threshold estimated based on the PG (policy gradient) algorithm is C T (wherein, for the same state, the decision output by the PG algorithm is random, and the decision output by the DPG algorithm is fixed).

[0122] The description of the adversarial rule value estimate in step S2-3-2 is as follows:

[0123] Assume that the adversarial rule value estimate obtained based on the attack rule is V rule , and the estimate value thereof when the input state vector is S is V rule (S).

[0124] The adversarial rule is a reference for guiding intelligent decision-making artificially summarized and refined through human actual adversarial experience or simulated adversarial records, which is mathematically equivalent to a series of deterministic or fuzzy functions that map a point or point set in the state space to a deterministic value estimate, so the domain of these functions is a certain subset of the state space, and the value range is the real number field, that is, the adversarial rule can be defined as:

[0125]

[0126] wherein, represents the function set equivalent to the adversarial rule, represents a subset of the state space , is the union set of the domains of all elements in , and is a one-dimensional real number space.

[0127] The description of the reward δR in step S2-3-2 is as follows:

[0128] The reward used by the divide-and-rule fusion controller is obtained by multiplying the reward R used by the sub-policy by a conversion factor δ. Since a certain fault tolerance space and sufficient training time need to be given to each sub-policy controller in the early stage, the conversion factor δ of the divide-and-rule fusion training controller in the early stage of training should be small, that is, a sub-policy controller should not be completely denied because of the success or failure of the exploration period. The conversion factor changes with time, as shown in Figure 7 .

[0129] But when coming to the training later stage of the divide-and-conquer fusion controller, it should be considered that each sub-strategy controller has obtained sufficient training, and its decision-making ability will not have greater improvement in a short time. At this time, if a sub-strategy controller performs poorly, another sub-strategy controller should be switched to try immediately. Therefore, the conversion factor can be selected as the hyperbolic tangent function δ=tanht.

[0130] The original hyperbolic tangent function is as shown by the dotted line in Figure 7 , and the value before t=0 is δ=-1, which is inconsistent with the idea of the embodiment of the present application. The specific description is as follows:

[0131] Firstly, the embodiment of the present application does not hope that the reward obtained by the sub-strategy controller has no effect on the divide-and-conquer fusion controller in the training early stage, because even from the perspective of giving sufficient training time, when a sub-strategy controller still cannot obtain a positive reward after a period of training, the embodiment of the present application hopes to give another sub-strategy controller an opportunity to train, thereby avoiding the situation that only one sub-strategy controller is trained. Therefore, the conversion factor in the training early stage should be smaller but not equal to zero, so as to realize the slow switching of the two sub-strategies, thereby enabling both controllers to be trained.

[0132] Meanwhile, the conversion factor in the early stage should obviously not be negative, otherwise the divide-and-conquer fusion controller will switch the sub-strategy with poor performance to make a decision, which deviates from the goal expected to be achieved by the divide-and-conquer fusion.

[0133] Secondly, the original hyperbolic tangent function switches at t=0, which is also inconsistent with the actual situation.

[0134] Therefore, the embodiment of the present application makes some transformations to the original hyperbolic tangent function to obtain the function actually used by the conversion factor:

[0135]

[0136] Wherein μ is the conversion factor in the training early stage, t s is the time when the training of the sub-strategy controller is basically completed.

[0137] The actually used δ function is as shown by the solid line in Figure 7 , which maintains δ=μ in the training early stage, and then gradually increases the conversion factor to increase the sub-strategy switching frequency after the sub-strategy controller basically completes the training, that is, near t=t s . Finally, the conversion factor will reach δ=1, and the reward used by the divide-and-conquer fusion controller is completely the same as the reward used by the sub-strategy controller.

[0138] As shown in Figure 7 , wherein μ=0.2 and t s= 10 For example, the specific super parameters μ and t s The selection of the specific super parameters μ and t

[0139] The divide-and-conquer fusion controller will use the obtained reward δR to learn using DQN (Deep Q Network, an algorithm that combines neural networks and Q-learning), and the optimization objective of the state-action value network thereof is as follows:

[0140]

[0141] Where Q θ (s,a in ) is the state-action value network, is the target state-action value function, θ and are network parameters, γ is a discount factor, s and s' correspond to the current state and the next state respectively, a in is the action taken at present, and R is the reward obtained after taking action a in state s.

[0142] (2) Divide-and-conquer fusion method in multi-agent cooperation and confrontation:

[0143] A multi-agent system (MAS) is composed of a group of agents with certain autonomous ability, and is used to solve distributed interactive decision problems with large scale in reality. Each agent in the multi-agent system has the ability of perception, cognition, decision-making and execution, and the multi-agent system formed thereby often has the characteristics of autonomy, distribution, coordination, etc., and has self-organizing ability and learning reasoning ability.

[0144] According to whether the agent itself has a cooperative intention, the multi-agent system can be generally divided into two categories: cooperative multi-agent system and non-cooperative multi-agent system.

[0145] In the multi-agent scene, the embodiment of the present application designs a fusion method based on Nash equilibrium, which is specifically explained as follows:

[0146] In game theory, Nash equilibrium (English: Nash equilibrium, or Nash equilibrium point) is a concept solution in a non-cooperative game (Non-cooperative game) involving two or more participants, assuming that each participant knows the equilibrium strategy of other participants, and no participant can benefit from changing their own strategy.

[0147] A strategy is said to be dominant if, regardless of the strategies chosen by the other players, a player will always choose that strategy. A set of strategies is defined as a Nash equilibrium if no player can benefit by changing his strategy while the other players keep theirs unchanged. A Nash equilibrium is named after the mathematician John Forbes Nash Jr.

[0148] In game theory, if each participant chooses his or her own strategy and no player can benefit by changing his or her strategy while the other participants keep theirs unchanged, then the current set of strategies and its consequences is called a Nash equilibrium. That is, if Then s is called a Nash equilibrium, where p i is the payoff of player i, r i represents a possible strategy for player i, (s; r i ) denotes the change in payoff to player i if he changes his strategy to r i .

[0149] A strategy combination is called a Nash equilibrium when no player can benefit by unilaterally changing strategy. In other words, every player's strategy is the best response to the strategies of the other players.

[0150] Nash equilibrium is an important concept in reinforcement learning games, which refers to a fixed point reached by multiple agents. In general, Nash equilibrium is a joint strategy of all agents.

[0151] At Nash equilibrium, no agent can obtain a higher reward by changing his strategy alone. For any agent, he cannot obtain a higher cumulative reward by adopting other strategies.

[0152] The fusion of multi-agent cooperation or confrontation in step S3 in the embodiment of the application adopts a fusion method based on Nash equilibrium, which can be expressed in mathematical form as:

[0153]

[0154] Where Agents is all agents, i and k represent the i th and k th agents respectively, π represents a strategy, * represents a Nash equilibrium strategy, represents the value function of changing the strategy of the i th agent at Nash equilibrium, is the value function of the i th agent under the Nash equilibrium strategy.

[0155] It is worth noting that the Nash equilibrium is not necessarily globally optimal, but it is the most likely result in probability, and it is the state that is easier to converge to when learning, especially if the current agent cannot know what strategy the other agents will take.

[0156] Multi-agent reinforcement learning is a stochastic game, which combines the Nash strategy of each state stage game into an agent's strategy in a dynamic environment. And constantly interact with the environment to update the Q value function (game reward) in each state stage game. In this special case, the Nash strategy of the fusion method based on Nash equilibrium can be rewritten as:

[0157]

[0158] Where, A i (i = 1, …, n) represents the action space of the ith agent, a i (i = 1, …, n) represents the action of the ith agent, is the probability of the ith agent taking action a i under state s in the Nash equilibrium.

[0159] According to the reward function of each agent, the stochastic game can be classified. If the reward function of the agent is the same, it is called a complete cooperation game or a team game. If the reward function of the agent is inverse, it is called a complete competition game or a zero-sum game.

[0160] In the fusion method based on Nash equilibrium, the present embodiment designs two methods, namely hard replacement method and soft replacement method.

[0161] (1) Hard replacement method

[0162] Premise: The algorithm simultaneously parallel two strategies.

[0163] Assumed condition: The sampling time is T; the current execution strategy is π x ; the time when the strategy starts to be executed is t = 0, and it is the optimal strategy of this, and the longest time of being executed is T max (satisfying T max % T = 0); In this execution cycle, the total number of times that the optimal strategy π * at each time is different from the actual execution strategy π x is counted as cnt, and the upper bound of the count is C max .

[0164] The execution process of the hard replacement method is shown in Figure 8 , and the specific description is as follows:

[0165] 1) For a certain fixed time t, if t > T maxThe new strategy must be executed according to the optimal strategy at this moment, and t and cnt should be reset to 0, returning to a new cycle;

[0166] 2) Otherwise, based on the time t, recommend a time period T forward, that is, determine π at time t = t + T. * and π x Check if they are equal. If not, calculate cnt = cnt + 1 (π). * ≠π x If they are equal, then determine whether the current time t > T. max And repeat the above steps.

[0167] 3) When calculating cnt = cnt + 1 (π) * ≠π x ) Get cnt>C max Immediately execute the optimal strategy π at this moment. * And set t back to 0, and cnt also to 0, returning to a new cycle;

[0168] If cnt≤C max Then determine whether the current time t > T. max And repeat the above steps.

[0169] (2) Soft substitution method

[0170] Prerequisite: The algorithm operates on two strategies simultaneously.

[0171] Assumption: There exists a decay coefficient τ, used to decay the effect of a certain strategy π taken at the previous time step. i (i = 1, 2) probability p i The impact. For example... Figure 9 As is evident, before time 1 / (1-τ), the policy π i The probability p i It will not have any effect.

[0172] The soft substitution method execution process is as follows: Figure 10 As shown, the specific description is as follows:

[0173] At time t, the probability of a certain policy π taking a certain value is compared with the probability p of taking this policy at the present moment. t And it relates to the probability of adopting this strategy in the first 1 / (1-τ) time steps, as described in detail below:

[0174] Where * = int(1 / (1-τ)), int represents integer division, p i This represents the probability that the strategy will be selected at time i.

[0175] Furthermore, for strategy π i For (i = 1, 2), such asFigure 10 The specific expression is as follows:

[0176]

[0177] At time t,

[0178] The optimal strategy of the strategy at this moment is

[0179] Where p' i (t) represents the value of the strategy pi i adopted at the i-th moment by manual calculation, since the value is not normalized, the embodiment of the application adopts the normalized form of p'1(t), p'2(t) to obtain and

[0180] Since the final decision is not determined only by the probability at time t, the instability of switching back and forth is alleviated to some extent.

[0181] As Figure 10 shown is the optimal strategy that should be selected at time t. The probability of selecting a certain strategy is not simply determined by the instantaneous probability at time t, but is obtained by the cumulative sum and superposition of all previous probabilities. Since the final number is not a probability, it needs to be further normalized to convert it into a probability. Where tau is the decay coefficient, tau t-1 , tau t-2 , …, tau 0 are the decay coefficients at the corresponding moments, and p ij represents the probability of using the strategy pi i at time j. Finally, the value with the larger probability is obtained to obtain the current optimal strategy.

[0182] The main flow in the model application module Figure 11 : The divide-and-conquer fusion algorithm based on reinforcement learning in the embodiment of the application has a process of improving the effect in steps for the interaction and antagonism of multiple intelligent agents.

[0183] First, the interaction of multiple single reinforcement learning algorithms with extreme complex environments and the improvement of efficiency and antagonistic success rate; second, the multi-algorithm fusion of single function modules in a single intelligent agent, and the antagonistic success rate is improved by 5%-10% compared with a single algorithm; then, multi-algorithm collaborative fusion of multiple function modules in a single intelligent agent, and there is a significant effect improvement compared with before fusion; finally, multi-algorithm hierarchical fusion in multiple intelligent agent collaborative work, and there is a significant effect improvement compared with before fusion.

[0184] As Figure 12aAs shown in -d, the result of a single agent using the optimal countermeasures throughout the game, the win rate is about 50%, and the average reward is -0.9;

[0185] As shown in Figure 13a As shown in -d, the result of a single agent using the optimal countermeasures throughout the game, the win rate is about 50%, and the average reward is -0.9;

[0186] As shown in Figure 14a As shown in -d, the result of a single agent using the optimal countermeasures throughout the game, the win rate is about 50%, and the average reward is -0.9;

[0187] The above is a further detailed description of the present application in combination with specific preferred embodiments, and cannot be regarded as limiting the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent substitutions or obvious modifications can be made, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present application.

Claims

1. A method for strategy decision in multi-agent cooperation and confrontation, characterized in that, The method comprises the following steps: S1, training a single agent to complete the divide-and-conquer of a single agent strategy by using different reinforcement learning strategies, to obtain a divide-and-conquer decision strategy most suitable for a single scenario; S2, completing the fusion of the single agent strategy based on a fusion algorithm of different reinforcement learning strategies according to the advantages and disadvantages of each decision algorithm in different scenarios, to dynamically select the fusion decision strategy obtained in the divide-and-conquer in step S1 most suitable for the current scenario in a real-time changing scenario; S3, intelligently decomposing a multi-agent cooperation or confrontation task by using different reinforcement learning strategies, and assigning the single agent equipped with the divide-and-conquer decision strategy in step S1 and the fusion decision strategy in step S2 to complete the divide-and-conquer fusion of the multi-agent cooperation or confrontation, to obtain an optimal decision strategy of the multi-agent cooperation or confrontation; In step S3, the fusion of the multi-agent cooperation or confrontation adopts a fusion method based on Nash equilibrium, expressed as: ； where Agents is all the agents, i, k represent the ith, kth agent respectively, denotes the strategy, denotes the Nash equilibrium strategy, denotes the value function of only changing the ith agent strategy at the Nash equilibrium, is the value function of the ith agent under the Nash equilibrium strategy; when the Nash equilibrium strategy of the stage game of each state is combined into an agent's strategy in the dynamic environment, and the Q value function in the stage game of each state is updated by constantly interacting with the environment; in this special case, the Nash strategy using the fusion method based on the Nash equilibrium is: wherein, represents the action space of the i-th agent, represents the action of the i-th agent, is the probability that the i-th agent takes action at state s in the Nash equilibrium, represents the state value function for agent i that n agents take actions at state s in the Nash equilibrium.

2. The method of claim 1, wherein, In steps S1, S2 and S3, the different reinforcement learning strategies include an adversarial rule value estimation strategy, a neural network value estimation strategy and a Monte Carlo tree search strategy.

3. The method of claim 1 or 2, wherein, Step S2 comprises the following steps: S2-1, a single agent decision generates an action, which is then applied to the environment, and the environment returns a decision state and a reward to the agent; S2-2, after the agent is trained with different reinforcement learning strategies, for the same decision state, the neural networks of different reinforcement learning strategies return value estimates of the current decision state; S2-3, after the value estimates of the sub-algorithms are fused by using the value covariance of the value estimates, an optimal fusion decision strategy is output.

4. The method of claim 3, wherein, Step S2-3 comprises the following steps: S2-3-1, using the current decision state and the reward designing a neural network to compute a covariance threshold ; S2-3-2, estimating the value against the rule and neural network value estimation The two groups of learning strategies return the value, respectively, and the value covariance of the current two groups of strategies is calculated. S2-3-3. determining the fusion factor according to the covariance threshold of step S2-3-1 and the value covariance of step S2-3-2 and step S2-3-2, the value of the fusion factor is determined, and the values of the two groups of strategies are fused. and step S2-3-2, the value of the fusion factor is determined, and the values of the two groups of strategies are fused. S2-3-4, according to the completed fusion value obtained in step S2-3-3, combining the Monte Carlo tree search strategy that has been trained, an optimal decision strategy is output.

5. The method of claim 4, wherein, In step S2-3-1, the covariance threshold value is obtained by a policy gradient-based reinforcement learning algorithm, specifically: at each time step, a policy network generates an action a, the agent performs the action to obtain a reward R, and an output end outputs a covariance threshold value ; wherein S is a state, is a parameter of the action network.

6. The method of claim 5, wherein, In step S2-3-1, the reward is obtained by multiplying the reward R used by the sub-policy of the divide-and-conquer controller by a conversion factor that is a function of: the number of steps taken by the sub-policy of the divide-and-conquer controller wherein is a conversion factor for the pre-training period, is the time at which the sub-policy controller training is substantially complete; The rewards Learning using DQN, the optimization objective of the state-action value network is as follows: wherein is a state-action value network, is a target state-action value function, and are network parameters, is a discount factor, s and correspond to a current state and a next state, respectively, is a currently taken action, and R is a reward obtained after taking action a in state s.

7. The method of claim 4, wherein, In step S2-3-2, the counter-rule value estimate is set to when the input state vector is ; The adversarial rule is defined as: wherein, represents a set of functions equivalent to the adversarial rule, represents a state space is a subset of is the union of the domains of all elements in is a one-dimensional real number space.

8. The method of claim 1, wherein, The fusion method based on Nash equilibrium includes a hard replacement method and a soft replacement method; Wherein, the execution flow of the hard replacement method is: 1) for a certain fixed time , if , the new strategy must be executed according to the optimal strategy at this moment, and is reset to 0, is also reset to 0, and is brought into a new cycle; 2) otherwise, it is judged whether and are equal, if not equal, then is calculated; if equal, then return to 1); if is obtained , the optimal strategy at this moment is executed immediately , and is reset to 0, is also reset to 0, and is brought into a new cycle; if , then return to 1); Wherein, the sampling time is T; the current execution strategy is ; the time when the strategy starts to be executed is , and the optimal strategy for this is , which is executed for the longest time ; in this execution cycle, the total number of times that the optimal strategy is different from the current actual execution strategy at each time is , and the upper bound of the count is .

9. The method of claim 8, wherein, The execution flow of the soft replacement method is as follows: at a certain time point, for a certain strategy , the value probability is related to the probability of taking this strategy at the moment ; the value probability is related to the probability of taking this strategy at the moment ; and the value probability is related to the probability of taking this strategy at the previous time point. For the policy The specific expression is: ； Thus, at time, , ; The optimal strategy of the moment is ; where, denotes the value of the policy computed manually at the i-th time instant, taking into account the normalized form of the reward , obtaining and , denotes the decay coefficient at time k, denotes the probability of choosing the policy at time t - k. ​

Images