A multi-agent autonomous decision-making method based on deep reinforcement learning

Abstract

The application discloses a kind of multi-agent autonomous decision-making methods based on deep reinforcement learning, applied to multi-agent deep reinforcement learning technical field, for the problem that prior art has not been fully used experience data in decision-making process;The application collects experience generated by interaction of agent and environment, and the immediate reward of environment is optimized by reward remodeling module, and is cached in local experience replay pool;Then merge the historical experience data in local experience replay pool into global experience replay pool PT-Buffer, and use two-level priority experience queue to maintain;Collect training sample data from PT-Buffer by probability summation tree to solve target policy network and global Critic network model;Finally, using the updated autonomous decision-making model, according to the input environment local observation information, directly realize the autonomous decision-making of multi-agent, to complete relevant multi-agent task.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of war game AI, and particularly relates to a multi-agent autonomous decision-making technology. BACKGROUND

[0002] Deep reinforcement learning technology has made breakthroughs in natural language processing, game AI and robot control, etc. From OpenAI-Five that defeats professional game teams, AlphaFold that helps biologists discover unknown proteins, to ChatGPT that chats with people and answers questions, deep reinforcement learning represents the artificial intelligence technology that makes autonomous decision-making and control of agents possible.

[0003] Currently, deep reinforcement learning algorithms can be mainly divided into the following three categories: value function method, policy gradient method and policy value function method. The representative algorithms in the value function method include IQL, VDN, QMIX, etc., which train the policy network of each agent by using the joint action value function through different decomposition methods. The algorithms in the policy gradient method such as REINFORCE, IPPO, PPO, COMA, etc. reduce the policy variance and speed up the solution of the policy model through importance sampling, gradient clipping, etc. The algorithms in the policy value function method such as Actor-Critic, TRPO, MAPPO, etc. improve the efficiency of using online data and the ability of autonomous decision-making of multi-agent through clipping factors, distributed data collection, etc.

[0004] How to use deep reinforcement learning technology to construct an autonomous decision-making model of multi-agent in a real-time confrontation environment such as war game is one of the current research hotspots, which mainly focuses on data usage efficiency and environment reward remodeling. In terms of data usage efficiency, priority experience replay methods such as PER, DPER, double experience replay pool, etc. are proposed, which improve the data usage efficiency of reinforcement learning algorithms by quantifying the importance of experience through time difference error. In view of the problems of sparse and lagging immediate rewards in real-time multi-agent environment, policy model is not easy to solve. Random network, expert knowledge transfer, imitation learning and other reward remodeling methods are proposed to guide the learning of decision-making model.

[0005] Although there have been many breakthroughs in the research of deep reinforcement learning, there are still some problems in the field of war game AI:

[0006] (1) How to fully use experience data in a multi-agent environment to improve the exploration ability of agents to the environment and the usage efficiency of historical experience.

[0007] (2) In the multi-agent environment with sparse rewards, how to design a reward function with certain robustness to reduce the negative impact of sparse rewards in real-time environment on the convergence speed of decision-making model and improve its generalization ability. SUMMARY

[0008] To solve the above technical problems, the present application provides a multi-agent autonomous decision-making method based on deep reinforcement learning,

[0009] The technical scheme adopted by the present application is: a multi-agent autonomous decision-making method based on deep reinforcement learning, the application scenarios include: a capture point in the center of the map, an own agent, and an enemy agent; the own agent and the enemy agent have the same type and number of tanks, battle tanks, and infantry to capture the control point as the task; the decision-making method includes the following steps:

[0010] S1, using a common neural network as the target policy network Target-Actor, the policy network Actor of each agent, and providing the number and type of the agent as the environmental information to distinguish the agent to the target policy network Target-Actor for training and learning, the target policy network Target-Actor generates different countermeasures for different agents;

[0011] The policy network Actor of each agent makes decisions through local observation information o i of the environment, outputs the corresponding action distribution and action a i , and the environment produces an immediate reward for the action of the agent

[0012] S2, the reward remodeling module remodels the immediate reward to obtain R t ;

[0013] S3, according to the remodeled R t , the experience data generated by the interaction between the agent and the environment is obtained; and the historical experience data of each agent is stored and maintained based on the global experience replay pool PT-Buffer of the two-level experience queue;

[0014] S4, collect training sample data from the global experience replay pool PT-Buffer through the probability summation tree;

[0015] S5, train the target policy network and the global evaluation network according to the training sample data obtained in step S4;

[0016] S6, synchronize the target policy network parameters obtained by training to the policy network Actor of each agent.

[0017] The experience data is represented as [S, O, A, R, S', A'], wherein S represents a global state space set of a current time step, R represents a reshaped reward set of the agent at the current time step, O represents a local observation information set of the agent at the current time step, A represents an action information set of the agent at the current time step, S' represents a global state space set of a next time step, and A' represents an action information set of the agent at the next time step.

[0018] The global state space includes capture point information, map information and real-time decision information, wherein the capture point information specifically refers to positions, scores and control states of all capture points; the map information specifically refers to sizes, terrains and heights of each position of the map; and the real-time decision information specifically refers to a current time step, a net score and a number of agents of the own side.

[0019] The local observation information of the agent includes capture point positions, map sizes, enemy and own net scores and a current time step, own positions, blood volumes and ammunition volumes, and enemy positions and blood volumes.

[0020] The global evaluation network Critic is composed of an action value function network and a hybrid network; wherein the action value function network is composed of a backbone network for feature extraction and fusion, two 256-dimensional and 1-dimensional fully connected layers, wherein the backbone network is composed of a 32-kernel 5x5 convolution layer, a 64-kernel 3x3 convolution layer and a 128-kernel 1x1 convolution layer; and the hybrid network is mainly composed of multiple feature mapping modules H, wherein each feature mapping module is composed of a 256-dimensional fully connected layer and a 64-dimensional fully connected layer.

[0021] The reward reshaping module specifically refers to that the reward reshaping module uses an internal reward function module constructed by environment factors to reshape the immediate reward of the environment; wherein the internal reward function module reshapes the immediate reward of the environment in the following process: the number of enemy agents, the number of ammunition and the distance between the agent and the capture point are used to constitute an internal reward function to adjust the distribution of the immediate reward of the environment in real time.

[0022] The beneficial effects of the present application: the multi-agent autonomous decision algorithm based on deep reinforcement learning of the present application adopts the improved time difference error TD-N and the probability summation tree to sample favorable training data, so that the data utilization efficiency of the multi-agent decision model is improved. At the same time, by using the adaptive reward function constructed by the environmental factors to adjust the distribution of the environmental immediate reward, the adaptability of the deep reinforcement learning algorithm to the sparse reward in the multi-agent environment is improved, and the learning speed of the decision model is accelerated. The method of the present application enables the war game AI to have the ability to fully explore the environment and optimize the strategy using historical experience data in the deduction environment, so that the model can converge faster in the training process and obtain higher average return. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a schematic diagram of the multi-agent autonomous decision algorithm based on deep reinforcement learning in an embodiment of the present application;

[0024] Figure 2 is a network framework structure diagram of PTAIR-MAPPO in an embodiment of the present application;

[0025] Figure 3 is a flowchart of immediate reward remodeling in an embodiment of the present application;

[0026] Figure 4 is a structure schematic diagram of the global experience replay pool in an embodiment of the present application;

[0027] Figure 5 is a structure schematic diagram of the probability summation tree in an embodiment of the present application;

[0028] Figure 6 is a mountain and forest 3v3 environment schematic diagram in an embodiment of the present application;

[0029] Figure 7 is a composition diagram of the global state space S of the environment and the local observation information O of the agent in an embodiment of the present application;

[0030] Figure 8 is a raw environment information diagram for constructing the state space in an embodiment of the present application;

[0031] Figure 9 is a training flowchart of the PTAIR-MAPPO algorithm in the war game deduction in an embodiment of the present application;

[0032] Figure 10 is an effect comparison diagram of two algorithms and rule agent confrontation training in an embodiment of the present application, taking the average return per round as an index;

[0033] Figure 11 is an effect comparison diagram of two algorithms and rule agent confrontation training in an embodiment of the present application, taking the average immediate reward as an index. DETAILED DESCRIPTION

[0034] To facilitate the understanding of the technical content of the present application for those skilled in the art, the content of the present application is further explained below in combination with the drawings.

[0035] The application discloses a kind of multi-agent autonomous decision algorithm based on deep reinforcement learning, its flow as shown in Figure 1 Specifically, the following steps are implemented:

[0036] S1, the experience generated by the interaction of the agent policy network Actor in the data collector Worker and global evaluation network Critic and environment, with the help of reward remodeling module optimization environment immediate reward, is cached in local experience replay pool.

[0037] S2, merge the historical experience data in local experience replay pool into global experience replay pool PT-Buffer, and use two-level priority experience queue to maintain.

[0038] S3, in the process of solving target policy network and global Critic network model, PT-Buffer will collect training sample data in PT2 by probability summation tree, Learner uses training sample data to calculate the corresponding advantage function and action probability logarithm value, and combines the proximal policy gradient descent method to optimize the objective function of target policy network and global Critic network.

[0039] S4, using the updated autonomous decision model, the local observation information of environment can be directly input into the trained agent policy network to realize the autonomous decision of multi-agent, so as to complete the related multi-agent task.

[0040] S1 is specifically:

[0041] When collecting data, Worker will copy the parameters of initialized target policy network Target-Actor to the policy network Actor of each agent i in distribution i , Actor i Then make a decision through the local observation information O i Of environment, output corresponding action distribution and action a iAfter the environment and the agent interact, the environment will make a state transition and reward or punish the decision action of the agent. The worker will repeatedly repeat the above process during the simulation run until the end of the deduction, which is set to run 20000 deduction rounds in this embodiment, so as to store a large amount of experience data [S, O, A, R, S', A'] generated by the interaction in the local experience cache pool. At the same time, the reward remodeling module will remodel the immediate reward R. Secondly, the global evaluation network Critic will use the global state S of the current time step of the environment, the local observation information set O = (o i , o i+1 , o i+2 ,..., o i+n ) of the agent at the current time step and the action information set A = (a i , a i+1 , a i+2 ,..., a i+n ) of the agent at the current time step to predict the expected return Q(S, O, A; ω c ) of the agent action under the current state.

[0042] As shown in the worker in Figure 2 , the structure of the data collector worker is: including a plurality of agents, cloned environments and local experience cache pools.

[0043] The strategy network of the plurality of agents is: using a common neural network as the strategy network Actor of each agent, and providing the number and type of the agent as the environment information for distinguishing the agent to the target strategy network Target-Actor through hard coding to train and learn the target strategy network Target-Actor, thereby helping the target strategy network Target-Actor to generate different countermeasures for different agents.

[0044] The strategy network Actor of the agent and the target strategy network Target-Actor are both from the common neural network , that is, the strategy network Actor of the agent and the target strategy network Target-Actor have the same network structure.

[0045] But the strategy network Actor of the agent is used to make actual actions for each agent in the board game environment, so the state of the environment changes with the action, and a series of data is generated, and after a certain amount of data is stored, some of them are taken out to update the parameters of the target strategy network Target-Actor network. After the target strategy network Target-Actor is updated for several times, the parameters are synchronized to the strategy network Actor of the agent. At this time, the two networks not only have the same structure, but also have the same parameters. Then repeat the above operation, and the strategy network Actor of the agent remains unchanged, and the target strategy network Target-Actor changes.

[0046] The common neural network The structure is: containing a backbone network composed of 32-core 5*5, 64-core 3*3 convolutional layer, 128-core 1*1 convolutional layer, a 256-dimensional fully connected layer and a 12-dimensional fully connected layer.

[0047] The global evaluation network Critic is composed of an action value function network and a hybrid network. The action value function network is composed of a backbone network for feature extraction and fusion, two 256-dimensional and 1-dimensional fully connected layers, and the backbone network is composed of 32-core 5*5, 64-core 3*3 convolutional layer, 128-core 1*1 convolutional layer. The hybrid network is mainly composed of multiple feature mapping modules H, wherein the feature mapping module is composed of a 256-dimensional fully connected layer and a 64-dimensional fully connected layer.

[0048] As shown in Figure 3 , the reward remodeling module is specifically: the reward remodeling module uses the internal reward function module constructed by the environment factor to remodel the immediate reward of the environment. The process of the internal reward function module remodeling the immediate reward of the environment is that the internal reward function is constructed by the number of enemy, the number of ammunition and the distance between the agent and the control point to adjust the distribution of the immediate reward of the environment in real time, and reduce the negative impact of the sparse reward of the environment on the model training. The calculation formula of remodeling the reward of the environment is:

[0049]

[0050] Where R t is the remodeled immediate reward of the environment at time t; is the immediate reward of the environment at time t, which comes from the confrontation score of the board game environment, including the control score, the remaining operator score and the battle score, etc.; is the internal immediate reward of the agent at time t, and its calculation formula can be expressed as:

[0051]

[0052] Where is the internal immediate reward of the agent i at time t, and its calculation formula is:

[0053]

[0054] wherein is the number of remaining ammunition of the agent i at time t; is the number of enemy agents at time t; η is a hyperparameter for controlling the influence of the distance factor on the internal immediate reward; d i，t is the distance between the agent i and the capture point at time t; ε' is a minimum value greater than 0. From the above three formulas, the immediate reward of the agent at time t is:

[0055]

[0056] Through the above reward function remodeling method, in the early and middle stages of confrontation, the real-time decision model considers the influence of the distance between the agent and the capture point as much as possible, which becomes the main driving force for the agent to explore the capture point. In the middle and late stages of confrontation, the number of ammunition of the agent itself and the survival number of enemy agents will become the main influencing factors of the internal immediate reward, encouraging each agent to generate an autonomous confrontation strategy with the largest number of enemy kills and the least consumption of its own resources.

[0057] S2 is specifically:

[0058] In view of the better exploration and utilization of experience data by the priority experience replay method, the application uses a global experience replay pool PT-Buffer based on a two-level experience queue to store and maintain the historical experience data of each agent, and uses a TD-N quantization method and a probability summation tree to improve the way in which the multi-agent decision model utilizes historical data. The global experience replay pool is composed of two-level experience queues PT1 and PT2, as shown in Figure 2 PT-Buffer.

[0059] After each simulation round, the local experience cache pool in all workers will synchronize and merge the local experience trajectory into the global experience replay pool PT-Buffer, so that the reinforcement learning algorithm can use the historical experience data.

[0060] As shown in Figure 4 , the global experience replay pool PT-Buffer is specifically as follows:

[0061] The global experience replay pool PT-Buffer is based on a two-level experience queue and is composed of two-level experience queues PT1 and PT2, and has a good ability to explore and utilize experience data.

[0062] In the process of data maintenance in the global experience replay pool, the historical experience data [S, O, A, R, S', A'] generated by the interaction between each agent and the environment is first cached in PT1 queue, and then all the data in PT1 are transferred to PT2 after the agent uses each data in PT1. When the agent obtains data from the global experience replay pool again, the experience replay pool samples the priority experience data generated by the previous strategy in PT2 using the method of probability summation number, and merges the sample data into PT1. Finally, the data in PT1 is taken out for training of the algorithm model. In the process of data collection and sampling in the global experience replay pool, PT1 is constantly updated, and PT2 discards the experience with too long storage time and resets at fixed time steps when full, so as to ensure the timeliness of the data in PT2.

[0063] In the process of using PT-Buffer to store historical experience data, first, the experience τ i is updated as follows: i , that is, the difference between the current Q value and the target Q value of the evaluation network and the number of times of using the experience to construct the priority TD-N of itself. Thus, the historical experience τ i = [S t , A t , R t , S' t+1 , A' t+1 ]| t＝i is updated as follows: i :

[0064]

[0065] wherein i∈[1, k]; for the experience τ i with larger δ i , its priority can be effectively reduced with the increase of the number of times of using ; the experience with smaller δ i can also moderately improve its priority. The sampling probability of the experience τ i can be obtained by combining the above formula and the idea of probability sampling:

[0066]

[0067] wherein α is the adjustment factor of the sampling probability. When α=0, the experience is uniformly sampled; when α=1, the experience is preferentially sampled according to the size of the sampling probability P samp (i).

[0068] To further balance the frequency of use of historical experience and ensure the diversity of sampled data, a probability summation tree method is used to ensure that each piece of historical experience data can be fully used by the agent, wherein the structure of the probability summation tree is as shown in Figure 5 First, the historical experience data in PT2 is sorted and grouped according to the sampling probability P samp samp The reciprocal of the sampling probability P i The sum of each pair of adjacent leaf node values is taken as the value of the corresponding parent node, and the probability summation tree is generated step by step. Then, the historical experience in PT2 is evenly divided into N intervals according to the sample size N, and a value is randomly selected in each interval as the sampling probability P sample Then, starting from the root node, when P sample is greater than the node value P node , P sample is updated to P sample and P node , when P sample is less than P node , P sample is updated to P node , and the traversal of the subtree of the node is continued until the leaf node is reached. Finally, the experience data in the leaf node is uniformly sampled to obtain the corresponding sample data, and the above process is repeated to obtain N pieces of training sample data.

[0069] S3 is specifically:

[0070] The target function of the policy network is as follows:

[0071]

[0072] Where A t θ′ (s t , a t ) is the advantage function of the agent policy network Actor at time t; ε represents the size of the gradient clipping factor; θ and θ' represent the parameters of the target policy network and the parameters of the agent policy network; P θ and P θ′ are the probabilities of the target policy network and the agent policy network selecting action a t in state S t .

[0073] The target function of the global Critic network is as follows:

[0074]

[0075] Where T is the number of time steps within a round, and t is the time step in the environment.​samp is the sampling probability of the sample samp (i) the size of the sampled experience data from the experience pool;P samp (i) the sampling probability of the sample; N is the number of agents; β is the sampling weight adjustment factor;R t is the reshaped environment immediate reward at time t;Q i is the action expected return of agent i; ω c is the Critic network parameter; γ is the decay factor of the expected return.

[0076] In order to reduce the Q value deviation caused by the use of probability summation tree, an annealing factor w i The target function of the global Critic network is corrected as:

[0077]

[0078] where N is the sample size; β is a hyperparameter that controls the algorithm's preference for the degree of sampling.

[0079] In order to ensure the stability of the target policy network update process during the training process of the algorithm, the parameters of all agent policy networks Actor are fixed, and the network parameters of Target-Actor are updated to Actor after the Target-Actor is trained a certain number of times.

[0080] Learner is a module containing some calculation formulas, which is used to update the network calculation. The main function is to update the network parameters through the two formulas of the policy network target function and the Critic network target function in front of S3, and then synchronize Actor and Target-Actor.

[0081] As Figure 2 shown, in one specific embodiment:

[0082] First, a micro environment is defined using the open-source wargame simulation environment engine of the Chinese Academy of Sciences, and a multi-agent confrontation wargame scene Shanshun3v3 is set up in a 23*13 hexagonal grid mountainous forest wargame map. In the Shanshun3v3 environment, it mainly contains the mountainous terrain of the mountainous forest, two control points (main / secondary control points) and 6 agents of the red and blue confrontation parties. The Shanshun3v3 environment used for algorithm training and testing is as shown in Figure 6The initial situation of the mountain 3v3 environment is shown in Table 1.

[0083] Table 1 Initial situation of the mountain 3v3 environment

[0084] Environmental attributes Attribute values Main control point position-score (12，5)-80 Secondary control point position-score (12，7)-50 Number of friendly agents 3 Number of enemy agents 3 Friendly troop position (2，5) Enemy troop position (26，5) Tank position-blood volume-score (3，12)-3-10 Number of large direct-fire cannons on tanks 30 Number of light weapons on tanks 30 Tank movement speed (seconds / meter) 20 Tank observation distance (meters) 25 Tank position-blood volume-score (3，13)-3-8 Number of light weapons on tanks 40 Number of small direct-fire cannons on tanks 40 Number of rapid-fire cannons on tanks 4 Number of missiles on tanks 4 Tank movement speed (seconds / meter) 20 Tank observation distance (meters) 25 Infantry position-blood volume-score (3，13)-3-4 Number of infantry light weapons 10 Number of rocket launchers 10 Number of portable missiles 4 Infantry movement speed (seconds / meter) 144 Infantry observation distance (meters) 10 Net score of friendly forces 0

[0085] The original confrontation situation in the war game deduction environment is preprocessed, and the state space of the war game agent is designed from the global state space of the environment and the local observation information of the agent. The composition of the global state space S of the environment and the local observation information O of the agent is shown in Figure 7 , and the original environment information for constructing the state space is shown in Figure 8 .

[0086] In the state space S, the global state space of the environment is mainly composed of three parts: capture point information, map information and real-time decision information, so as to regard the global state space as an abstract set of the entire environment information. The information of the capture point is mainly composed of the position, score and whether it is controlled of all capture points. The information of the capture point is the target of the confrontation between the two parties, and is the key factor affecting the decision of the multi-agent. The map information is mainly composed of the size of the confrontation map, the terrain and height of each position of the map, etc., which is the state set of the inherent information of the deduction environment. The real-time decision information mainly includes the current time step of the environment, the net score of the own side and the number of agents of the own side, etc.

[0087] In the design of the local state space of the agent, due to the influence of environmental factors such as war fog and special terrain, each agent can only obtain the local environmental state information within a certain observation range. However, the local environment state obtained by the agent is not only closely related to the state of the agent itself, but also has a certain implicit relationship with the global state of the current environment. Therefore, in order to help each distributed agent make effective real-time decisions under the limited local information and improve its decision-making level, the local state space contains the position, ammunition, blood volume and observed enemy agent type and position of each agent of the current own side, so as to completely reflect the local state characteristics of the environment, so that the agent can make autonomous decisions to a certain extent with the help of the local state.

[0088] As shown in Figure 7As shown in the global state space, the capture point information includes the position, score, and control status of all capture points; the map information includes the size, terrain, and height of the map; and the real-time decision information includes the current time step, net score, and number of friendly agents. The local observation information of the agent includes the position of the capture point, the size of the map, the net score, and the current time step; the position, blood volume, and ammunition of the friendly agent; and the position, blood volume, and ammunition of the enemy agent.

[0089] As shown in the original situation information, the map information includes the position of the capture point, the score of the capture point, the height of the map, and the terrain of the map; and the agent information includes the position, category, score, blood volume, and ammunition of the friendly agent, and the position, category, score, blood volume, and ammunition of the enemy agent. Figure 8 To reduce the negative impact of the mixed action space of the Kriegspiel environment and improve the performance of the algorithm, the action space of the agent is simplified, and the action of the agent is simplified into 12 basic actions, mainly including continuous actions such as getting off, moving, and hiding, and discrete actions such as shooting and stopping moving. The simplified action space is shown in Table 2.

[0090] Table 2 Simplified action space

[0091]

[0092]

[0093] ​After initializing the simulation training environment of the PTAIR-MAPPO algorithm, the situation information in the training environment is encoded and normalized in a hot encoding manner, and then the current local observation information is mapped to a decision action by using a policy network Actor constructed by a deep neural network. Secondly, the evaluation network is used to estimate the action reward of the current agent, and the hybrid network is used to adjust the estimated value of the evaluation network for each agent action reward. The environment state changes under the action of the agent and generates an immediate reward, and the agent remolds the immediate reward of the environment through its own reward remolding module. The agent and the environment in the war game environment repeatedly interact the above process until the game ends, and experience data [S, O, A, R, S', A'] is generated in the process. Finally, the PTAIR-MAPPO algorithm collects training sample data by using the probability summation tree, and uses the advantage function and action probability logarithmic value generated by the obtained historical experience data to solve the optimization target policy network and evaluation network model. According to the network framework of the PTAIR-MAPPO algorithm, the process of decision-making and network training using the priority experience replay mechanism and the adaptive reward function based on TD-N and the probability summation tree in the war game can be obtained, as shown in Figure 9

[0094] Through the above training process, the decision-making model is trained for 20,000 rounds, and each round is simulated for 1,800 steps. The hyperparameters involved in the training are shown in Table 3.

[0095] Table 3 Hyperparameters in training

[0096] Training hyperparameters Parameter values Number of steps / rounds 1800 steps / rounds Experience pool size 10000 Training sample size 256 Learning rate λ 0.99 Return decay rate γ 0.97 Number of sample training times 5 times Clipping factor ε 0.2 α 0.99 β 1 η 0.3

[0097] In view of the strong decision-making ability and excellent performance of the MAPPO algorithm in real-time multi-agent environments such as Dota2 and StarCraft, the present application uses the MAPPO algorithm as the baseline algorithm for the war game environment to evaluate the performance of the present algorithm.

[0098] The present application first trains the PTAIR-MAPPO algorithm and the MAPPO algorithm in the mountain forest 3v3 environment for 20,000 rounds, and uses the average reward to evaluate the performance of the two algorithms in the process. The average reward of the two algorithms in this training process is shown in Figure 10 The average reward in each round is the average of the net points obtained by the agent at each environment time step. The size of the average reward indirectly reflects the real-time confrontation ability of the agent. The larger the average reward, the stronger the confrontation ability of the agent, and the more effective the decision-making, and vice versa.

[0099] As shown in Figure 10 ​As shown, with the increase of the number of training, the average return of PTAIR-MAPPO algorithm and MAPPO algorithm is gradually improved, but the average return of MAPPO algorithm after stabilization is always lower than that of PTAIR-MAPPO algorithm. Compared with MAPPO algorithm, the convergence speed of PTAIR-MAPPO algorithm is faster, and the average return is improved by about 10 points.

[0100] In the mountain 3v3 environment, the two algorithms were trained for 10000 rounds using training data with batch sizes of 64, 256 and 512 respectively, and the data usage capabilities of the two algorithms under different batch size training data were tested. The average return of the two algorithms is shown in Table 4, from which it can be seen that the average return score obtained by PTAIR-MAPPO algorithm on different batch size training data is higher than that of MAPPO algorithm. It can be seen that the PTAIR-MAPPO algorithm based on TD-N priority experience and probability summation tree improves the efficiency of multi-agent using online historical data by adjusting the sampling probability of different historical experience.

[0101] Table 4 Average return score in mountain 3v3 scenario

[0102]

[0103] Using PTAIR-MAPPO algorithm and MAPPO algorithm with 256 batch size data respectively, the two algorithms were used in the mountain 3v3 environment for 1000 rounds of confrontation with the built-in rule AI in the environment, and the average instant reward obtained by the two algorithms in the round is shown in Figure 11 .

[0104] As shown in Figure 11 , in the board game multi-agent real-time confrontation environment, the MAPPO algorithm without remodeling the environment reward obtains very sparse instant reward in the early stage of one round of training, and the low sensitivity of action state value change makes the optimization speed of global Critic network slow, which requires longer training time to help the agent search for the optimal strategy. The PTAIR-MAPPO algorithm can effectively adjust the distribution of instant rewards in the environment in one round by using the internal reward function constructed by the environment factor. When the agent outputs beneficial actions, it gives a larger instant reward to encourage the agent's behavior, and when it outputs wrong actions, it gives a larger punishment to reduce the probability of the agent selecting the action again. In the later stage of one round, PTAIR-MAPPO algorithm effectively adjusts the influence of net score of the two sides in confrontation, ensures the variance and update range of target policy network parameters within a certain range, and ensures the stability of the training process of the policy model.

[0105] Table 5 Average return score under different instant rewards

[0106] Model Early stage Late stage MAPPO 7.3 -55.4 PTAIR-MAPPO 39.7 -17.9

[0107] The average return of the two algorithms is shown in Table 5. It can be seen that the PTAIR-MAPPO algorithm has a higher average return than the MAPPO algorithm by 32.4 points in the early stage of confrontation (0-750 steps) and by 37.5 points in the late stage of confrontation (1200-1800 steps). Figure 11 The results of the comparative experiments in Table 5 show that the adaptive reward function in the PTAIR-MAPPO algorithm can effectively adjust the distribution of immediate rewards in the multi-agent environment and accelerate the training of the confrontation decision algorithm, achieving higher average returns in the early and late stages of confrontation.

[0108] Finally, the PTAIR-MAPPO decision model and the MAPPO decision model trained in the Mountain Range 3v3 environment were tested in the Water Network 3v3 environment for 2000 rounds to verify the robustness and generalization ability of the algorithm in the multi-agent confrontation environment. The average score, win rate, and average reasoning time per round of the two decision models are shown in Table 6.

[0109] Table 6 Performance indicators in the Water Network 3v3 scenario

[0110] Model Average return / score Win rate / % Round average reasoning time / s MAPPO 40.7 51 17.2 PTAIR-MAPPO 51.3 63 16.9

[0111] The experimental results in Table 6 show that the convergence speed and score ability of the PTAIR-MAPPO algorithm in the Water Network 3v3 environment still surpass the MAPPO algorithm. Compared with the MAPPO algorithm, the win rate of the PTAIR-MAPPO algorithm increases by 12%, the average return increases by 10.6 points, and the average reasoning time per round decreases by 0.3s. It can be seen that the multi-agent decision model constructed by the PTAIR-MAPPO algorithm has the ability to adapt to different environments and can achieve good results in different confrontation scenarios.

[0112] To sum up, the multi-agent autonomous decision algorithm based on deep reinforcement learning of the application adopts the improved time difference error TD-N and the probability summation tree to sample favorable training data, so that the data use efficiency of the multi-agent decision model is improved. Meanwhile, by using the adaptive reward function constructed by the environmental factors to adjust the distribution of the instant reward of the environment, the adaptability of the deep reinforcement learning algorithm to the sparse reward in the multi-agent environment is improved, and the learning speed of the decision model is accelerated. The method of the application enables the war game AI to have the ability of fully exploring the environment and optimizing the strategy by using the historical experience data in the deduction environment, so that the model can converge faster in the training process and obtain higher average return. Those skilled in the art will realize that the embodiments described herein are to help the reader understand the principles of the application and should be understood as the protection scope of the application is not limited to such specific statements and embodiments. The application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application should be included in the protection scope of the claims of the application.

Claims

1. A multi-agent autonomous decision-making method based on deep reinforcement learning, characterized in that, The application scenarios include a map, a capture point in the center of the map, an own agent, and an enemy agent; the own agent and the enemy agent have the same type and number of tanks, battle cars and infantry to capture the capture point; the decision method comprises the following steps: S1, using a common neural network As the target policy network Target-Actor, the policy network Actor of each agent, and by hard coding the number and type of the agent as the environmental information distinguishing the agent to provide the target policy network Target-Actor for training and learning, the target policy network Target-Actor generates different countermeasures for different agents. The policy network Actor of each agent makes a decision by taking the local observation information o i of the environment, and outputs the corresponding action distribution and action a i , while the environment produces an immediate reward for the action of the agent S2, reward reshaping module reshapes the immediate reward to obtain R t The calculation formula of the reward reshaping module is wherein R t is the reshaped environment immediate reward at time t; is the environment immediate reward at time t; is the agent's internal immediate reward at time t, The calculation formula is represented as: wherein is the internal instant reward of agent i at time t, and its calculation formula is: wherein is the number of remaining ammunition of agent i at time t; is the number of enemy agents at time t; η is a hyperparameter controlling the influence of the distance factor on the internal immediate reward; d i,t is the distance between agent i and the capture point at time t; ε' is a small positive value; S3、According to the remodeled R t , obtain experience data generated by the interaction of the agent and the environment; and store and maintain the historical experience data of each agent based on the global experience replay pool PT-Buffer of the secondary experience queue; the global experience replay pool PT-Buffer includes secondary experience queues PT1 and PT2, and when storing the historical experience data using the global experience replay pool PT-Buffer, the experience τ i The time difference error δ i of the experience τ i , that is, the difference between the current Q value and the target Q value of the evaluation network and the number of times of using the experience The priority TD-N of the agent itself is constituted; thereby obtaining the priority P i of the historical experience τ t =[S t ,A t ,R t+1 ,S′ t+1 ,A′ t＝i | i : Wherein i∈[1,k]; S4, collecting training sample data from the global experience replay pool PT-Buffer through a probability summation tree; S5, training the target policy network and the global evaluation network according to the training sample data obtained in step S4; S6, synchronizing the target policy network parameters obtained by training to the policy network Actor of each agent.

2. The multi-agent autonomous decision-making method based on deep reinforcement learning according to claim 1, characterized in that, The experience data is represented as [S, O, A, R, S', A'], wherein S represents a global state space set at a current time step, R represents a reward set of the agent at the current time step, O represents a local observation information set of the agent at the current time step, A represents an action information set of the agent at the current time step, S' represents a global state space set at a next time step, and A' represents an action information set of the agent at the next time step.

3. The multi-agent autonomous decision-making method based on deep reinforcement learning according to claim 2, characterized in that, The global state space includes capture point information, map information and real-time decision information, wherein the capture point information specifically includes the positions, scores and control states of all capture points; the map information specifically includes the size of the map, the terrain and height of each position of the map; and the real-time decision information specifically includes the current time step, the net score of the own side, and the number of own agents.

4. The multi-agent autonomous decision-making method based on deep reinforcement learning according to claim 3, characterized in that, The local observation information of the agent includes the positions of capture points, the size of the map, the net score of the enemy and the own side, the current time step, the positions, blood volume and ammunition volume of the own side, and the positions and blood volume of the enemy.

5. The multi-agent autonomous decision-making method based on deep reinforcement learning according to claim 4, characterized in that, Common neural network The structure is: including a backbone network composed of 32-core 5x5 convolution layer, 64-core 3x3 convolution layer, 128-core 1x1 convolution layer, a 256-dimensional fully connected layer and a 12-dimensional fully connected layer.

6. The multi-agent autonomous decision-making method based on deep reinforcement learning according to claim 5, characterized in that, The global evaluation network structure comprises an action value function network and a hybrid network; The action value function network comprises a backbone network for feature extraction and fusion, a 256-dimensional full connection layer and a 1-dimensional full connection layer, wherein the backbone network comprises a 32-kernel 5*5 convolution layer, a 64-kernel 3*3 convolution layer and a 128-kernel 1*1 convolution layer. The hybrid network comprises a plurality of feature mapping modules, each of which comprises a 256-dimensional full connection layer and a 64-dimensional full connection layer.

7. The multi-agent autonomous decision-making method based on deep reinforcement learning according to claim 6, characterized in that, Step S4 is specifically: The sampling probability calculation formula for collecting training sample data from the global experience replay pool PT-Buffer is: where P samp (i) is the sampling probability of the experience τ i , and α is a tuning factor for the sampling probability. When α = 0, uniform sampling is performed on the experiences; when α = 1, biased sampling is performed according to the size of the sampling probability P samp (i) of the experiences. According to the sampling probability P samp (i) sorting and grouping the historical experience data in PT2, and taking the reciprocal of the sampling probability P samp (i) as the experience τ i At the leaf node value on the probability summation tree, taking the sum of each pair of adjacent leaf node values as the value of the corresponding parent node to generate the probability summation tree step by step; Then, the record is updated according to the sampling probability P samp (i) The training sample size N is collected from the global experience replay pool PT-Buffer, the historical experience in PT2 is divided into N intervals, and a value is randomly selected in each interval as the sampling probability P sample Then, the sum tree is traversed from the root node, and when P sample is greater than the node value P node , P sample is updated to the difference between P sample and P node , when P sample is less than P node , P sample is updated to P node , and the subtree of the node is continuously traversed until the leaf node is reached. Finally, the experience data in the leaf nodes is uniformly sampled to obtain corresponding sample data; The above process is repeated to obtain N training sample data.

Images