Multi-agent cooperative and confrontation decision method and device based on reinforcement learning

Abstract

The present disclosure relates to the field of deep reinforcement learning, and provides a multi-agent cooperative and antagonistic decision-making method and device based on reinforcement learning. The method comprises: obtaining current state information; inputting the current state information into a decision-making agent trained by a preset reinforcement learning algorithm to obtain a decision result; the reinforcement learning algorithm comprises: obtaining observation information of the decision-making agent; encoding the observation information and a weight vector to obtain weight and state feature collaborative encoding, designing a target reward function; multi-state value resampling, determining an optimal augmented tuple sequence in combination with overall trajectory Q values; determining a loss function of a policy network and a total asymptotic envelope loss function of a value network based on the optimal tuple sequence, and updating parameters of the policy network and the value network; and based on the target reward function, using the network updated by the parameters to perform decision decoding on the weight and state feature collaborative encoding to output a decision result. The present disclosure can enable the agent to make decisions in real time and effectively improve the accuracy of the decision result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of deep reinforcement learning, in particular to a multi-agent cooperative and adversarial decision-making method and device based on reinforcement learning. BACKGROUND

[0002] In real-time strategy games, an agent needs to make a corresponding decision based on the environment it is in to complete the task of confrontation between the enemy and itself, and perform the corresponding action according to the decision. In order to enable the agent to make the optimal decision, the existing technology usually adopts a multi-objective deep reinforcement learning algorithm. In a multi-objective problem, reinforcement learning needs to handle multiple objective functions, for example, in a game, it needs to consider maximizing the score and minimizing the game time at the same time, and therefore, in the reinforcement learning algorithm, a suitable reward function needs to be designed to evaluate the behavior of the agent, thereby guiding its behavior. Generally, a linear weighting method or a Pareto optimization method can be used to handle the reward function in multi-objective reinforcement learning.

[0003] The linear weighting method is to linearly weight the reward values of multiple objective functions to obtain a weighted sum as the reward value of the agent, for example, the reward value can be represented as R = ω1 x R1 + ω2 x R2, wherein ω1 and ω2 are the weights of different objective functions R1 and R2, which can be adjusted according to specific problems. The advantage of the linear weighting method is that it is simple and easy to implement, and the disadvantage is that the setting of the weights between different objective functions is very dependent on specific application scenarios, it is difficult to fix specific weights in actual dynamic optimization target problems, the weights are variable, and it may cause the accuracy of the reward function to be inaccurate in some cases, thereby causing the final decision made by the agent to be biased.

[0004] The Pareto optimization method is to solve the multi-objective reinforcement learning problem by finding the Pareto optimal solution. In the Pareto optimal solution, no objective function can reach the optimal solution in all solutions. In this method, we need to determine a set of different weight vectors, and then optimize each weight vector. By repeating this process, we can get a set of Pareto optimal solutions. The advantage of the Pareto optimization method is that it can consider multiple objective functions at the same time and can get a set of optimal solutions, and the disadvantage is that it usually requires a large amount of calculation to find a set of effective weight vectors, and the number of Pareto optimal solutions can be large, it is difficult to find the optimal vector in actual dynamic optimization target problems, the network training efficiency is low, and therefore the agent cannot make a corresponding decision in real time. SUMMARY

[0005] The present disclosure aims to at least solve one of the problems existing in the prior art, and provides a multi-agent cooperative and adversarial decision-making method and device based on reinforcement learning.

[0006] In one aspect of the present disclosure, a reinforcement learning-based multi-agent cooperative and antagonistic decision-making method is provided, which comprises:

[0007] acquiring current state information, the current state information comprising self state information, ally state information and enemy state information;

[0008] inputting the current state information into a decision-making agent trained by a preset reinforcement learning algorithm to obtain a decision-making result, the decision-making result comprising any one of a moving action, an attacking action and a stationary action;

[0009] the preset reinforcement learning algorithm comprising the following steps:

[0010] acquiring observation information of the decision-making agent, the observation information comprising self state information, ally state information and enemy state information of the decision-making agent;

[0011] encoding the observation information and a weight vector by using an encoding weight matrix to obtain weight and state feature collaborative encoding, and designing a target reward function;

[0012] based on a weight space in which the weight vector is located, performing multi-state value resampling, combining an overall trajectory Q value corresponding to the weight and state feature collaborative encoding, and determining an optimal augmented tuple sequence corresponding to the weight and state feature collaborative encoding;

[0013] determining a loss function of a policy network and a total asymptotic envelope loss function of a value network based on the optimal tuple sequence, and performing parameter updating on the policy network and the value network based on the loss function of the policy network and the total asymptotic envelope loss function;

[0014] based on the target reward function, performing decision decoding on the weight and state feature collaborative encoding by using the policy network and the value network after parameter updating, and outputting a decision-making result.

[0015] Optionally, the encoding of the observation information and the weight vector by using the encoding weight matrix to obtain the weight and state feature collaborative encoding comprises:

[0016] encoding the observation information according to the following formula (1) to obtain observation information encoding O, wherein O self ∈R OS represents the self state information of the decision-making agent, R represents a reward function, OS represents the information dimension of the decision-making agent, is a vectorized representation of O f O f ∈R OF×NFrepresents the friendly state information, OF represents the friendly information dimension, and NF represents the number of friendly combat units, is the vectorized representation of O e e ∈R OE×NE represents the enemy state information, OE represents the enemy information dimension, and NE represents the number of enemy combat units:

[0017]

[0018] The observation information code O is encoded according to the following formula (2) to obtain the state information code S, wherein F1∈R E1×NO represents the first encoding weight matrix, O∈R NO , NO = OS + OF * NF + OE * NE, S∈R E1 , and E1 represents the state feature dimension corresponding to the state information code S:

[0019] S = F1 x O (2)

[0020] The weight vector is encoded according to the following formula (3) to obtain the weight vector code FW, wherein FW∈R E2 , F2∈R E2 ×NW represents the second encoding weight matrix, ω∈R NW represents the weight vector, NW represents the weight vector dimension, and E2 represents the weight feature dimension corresponding to the weight vector code FW:

[0021] FW = F2 x ω (3)

[0022] The weight and state feature collaborative code C sw is obtained according to the following formula (4):

[0023] C sw = {S, FW} (4).

[0024] Optionally, the target reward function is represented as the following formula (5):

[0025] f ω (R(s, a)) = ω T R(s, a) = ω1R1(s, a) + ω2R2(s, a) (5)

[0026] where f ω ​(R(s, a)) represents the target reward function, s represents a state tensor corresponding to the decision-making agent, a represents an action tensor corresponding to the decision-making agent, R1(s, a) represents a kill enemy combat unit reward, ω1 represents a weight corresponding to R1(s, a), R2(s, a) represents a remaining blood volume reward of the decision-making agent, and ω2 represents a weight corresponding to R2(s, a).

[0027] Optionally, multi-state value resampling is performed based on a weight space in which the weight vector is located, an overall trajectory Q value corresponding to the weight and state feature collaborative coding is determined, and an optimal augmented tuple sequence corresponding to the weight and state feature collaborative coding is determined, including:

[0028] The overall trajectory Q value is calculated according to the following formula (6), wherein Q π (s, a) represents the overall trajectory Q value, π is an action policy distribution, γ is a decay coefficient, t represents a time, R t represents a reward function at time t, s0 represents an initial state at t = 0 when the decision-making agent interacts with the environment, a0 represents an action at t = 0 when the decision-making agent interacts with the environment, represents a mathematical expectation:

[0029]

[0030] n times of uniform random sampling are performed on a weight space Ω in which the weight vector ω is located, to obtain n weight vectors ω, the n weight vectors ω are combined with a state transition sequence respectively, to obtain m x n x K state tuples (s t , a t , R t , s t+1 , ω) as an augmented tuple sequence, wherein m represents a dimension of a reward vector, K represents a number of state transition sequences, s t represents an initial state at time t when the decision-making agent interacts with the environment, a t represents an action at time t when the decision-making agent interacts with the environment, s t+1 represents a state returned after interaction at time t when the decision-making agent interacts with the environment;

[0031] For any given transition tuple sequence (s t , a t , R t , s t+1 ), an optimal weight vector ω * is calculated in the n weight vectors ω sampled, wherein ω ω = argmax T ω t+1 Q(s t, ω), to obtain the corresponding optimal augmented tuple sequence (s t a t R t s t+1 ω * ), where Q(s) t+1 a t ,ω) represents (s t+1 a t The Q value of , ω).

[0032] Optionally, the loss function of the policy network is expressed as equation (7):

[0033]

[0034] Among them, L π (θ) represents the loss function of the policy network, π θ and The distributions of the new and old strategies are θ and θ', respectively. old π θ and The parameters, This indicates that the mean of the sample is taken. To augment the advantage function, it is expressed as equation (8).

[0035]

[0036] Among them, V π For a value network learned according to the current policy π, θ V For V π The parameter, Q π (s t a t ,ω) represents the Q value of the state trajectory corresponding to the current policy π.

[0037] Optionally, the total asymptotic envelope loss function of the value network is expressed as equation (9):

[0038]

[0039] Where L(θ) represents the total asymptotic envelope loss function of the value network. This represents the weighting of the tradeoffs, where k is the number of iterations and τ is the delay factor. Let the loss function be the initial loss function of the value network during training, and expressed as equation (10). Let the loss function for the later training stage of the network be expressed as equation (11).

[0040]

[0041]

[0042] wherein y1 and y2 are both intermediate variables and are represented as the following formula (12) and the following formula (13) respectively

[0043]

[0044]

[0045] wherein γ represents a discount factor.

[0046] Optionally, the policy network and the value network are updated based on the target reward function, and the weight and state feature collaborative coding is used to make a decision decoding, and a decision result is output, comprising:

[0047] The decision action probability is calculated according to the following formula (14):

[0048]

[0049] wherein P represents the decision action probability, softmax represents a normalized exponential function, NA represents the dimension of the decision action, i represents the dimension sequence number of the decision action, output = FC(C sw ) represents and output ∈ R NA , output i represents the i-th element of output, and FC represents a full connection network.

[0050] Another aspect of the present disclosure provides a multi-agent collaborative and antagonistic decision device based on reinforcement learning, comprising:

[0051] An acquisition module is configured to acquire current state information, wherein the current state information comprises self state information, ally state information and enemy state information.

[0052] A decision module is configured to input the current state information into a decision agent trained by a preset reinforcement learning algorithm to obtain a decision result, wherein the decision result comprises any one of a moving action, an attacking action and a keeping still action.

[0053] A training module is configured to train the decision agent according to the preset reinforcement learning algorithm comprising the following steps:

[0054] An observation information of the decision agent is acquired, wherein the observation information comprises self state information, ally state information and enemy state information.

[0055] An encoding weight matrix is used to encode the observation information and a weight vector to obtain a weight and state feature collaborative coding, and a target reward function is designed.

[0056] Based on the weight space where the weight vector is located, multi-state value resampling is performed, and an optimal augmented tuple sequence corresponding to the weight and state feature collaborative coding is determined by combining the overall trajectory Q value corresponding to the weight and state feature collaborative coding.

[0057] Based on the optimal tuple sequence, a loss function of a policy network and a total asymptotic envelope loss function of a value network are determined respectively, and the policy network and the value network are updated based on the loss function of the policy network and the total asymptotic envelope loss function.

[0058] Based on the target reward function, the policy network and the value network after parameter update are used for decision decoding of the weight and state feature collaborative coding, and a decision result is output.

[0059] Another aspect of the present disclosure provides an electronic device comprising:

[0060] at least one processor; and

[0061] a memory in communication connection with the at least one processor; wherein

[0062] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-agent collaborative confrontation decision-making method based on reinforcement learning as described above.

[0063] Another aspect of the present disclosure provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the multi-agent collaborative confrontation decision-making method based on reinforcement learning as described above.

[0064] Compared with the prior art, the present disclosure can make the agent make corresponding decisions based on the current state in the multi-agent collaborative confrontation process and effectively improve the accuracy of the decision result by inputting the current state information into the decision agent trained by the preset reinforcement learning algorithm to obtain the decision result. BRIEF DESCRIPTION OF DRAWINGS

[0065] One or more embodiments are exemplarily illustrated by the pictures in the drawings corresponding thereto, which do not constitute a limitation on the embodiments, and elements with the same reference numerals in the drawings represent similar elements, unless otherwise specified, and the drawings do not constitute a proportional limitation.

[0066] Figure 1 A flowchart of a multi-agent collaborative confrontation decision-making method based on reinforcement learning provided by an embodiment of the present disclosure;

[0067] Figure 2A network framework schematic diagram of a proximal policy optimization algorithm provided for another embodiment of the present disclosure;

[0068] Figure 3 A flowchart of a preset reinforcement learning algorithm provided for another embodiment of the present disclosure;

[0069] Figure 4 A flowchart of a preset reinforcement learning algorithm provided for another embodiment of the present disclosure;

[0070] Figure 5 A structural schematic diagram of a multi-agent collaborative decision-making device based on reinforcement learning provided for another embodiment of the present disclosure;

[0071] Figure 6 A structural schematic diagram of an electronic device provided for another embodiment of the present disclosure. DETAILED DESCRIPTION

[0072] Multi-Objective Reinforcement Learning (MORL) refers to a technique that considers multiple objectives in a reinforcement learning task. Multi-Objective Reinforcement Learning adds objectives as conditions in the standard reinforcement learning Markov Decision Process, and seeks a policy that maximizes cumulative rewards. Compared with the standard reinforcement learning algorithm, the policy trained by the multi-objective reinforcement learning algorithm can make different decisions according to different objectives, and can be used to solve some problems that need to optimize multiple objectives at the same time, such as getting high scores and keeping characters alive in games. Existing multi-objective reinforcement learning algorithms can be roughly divided into two categories, namely single-policy method and multi-policy method.

[0073] Many applications of single-policy algorithms use weighted or unweighted linear scalar functions, which is equivalent to converting multi-objective reinforcement learning into corresponding single-objective reinforcement learning, and thus also applies to the existing convergence proof of reinforcement learning. In some fields (for example, in some scenarios where multiple objectives are directly represented by currency), such linear scalar functions can reasonably represent the potential utility of users. However, in many cases, linear functions are not sufficient to represent the true utility of users. Therefore, it is generally more desirable to use nonlinear functions. Pan et al. implemented a hybrid long-term policy gradient and short-term planning to find a single policy solution, while Siddique et al. developed a multi-objective form of Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C) policy search methods to find a single policy that is fair to all objectives. In summary, the single-policy method mainly solves the fixed-weight multi-objective optimization problem.

[0074] Multi-strategy methods can be further divided into two categories, outer-loop and inner-loop methods. Outer-loop methods construct (approximate) convex coverage sets (CCS) for a series of single-objective problems, while inner-loop methods consist of algorithms designed to generate multiple strategies at once. The simplest outer-loop method iterates over a series of different parameter settings for a utility function and re-runs a single-strategy MORL method for each setting. The efficiency of outer-loop methods can be improved in two ways: on the one hand, re-using information from previous runs instead of discarding it can reduce the time for learning; on the other hand, naive search over the parameter space can re-learn the same strategies multiple times or require small steps to ensure that all optimal strategies are discovered. More efficient adaptive search methods can reduce the number of iterations of the outer loop. Inner-loop methods modify the underlying algorithm to directly and concurrently identify and store multiple strategies, rather than in a certain order. They modify the Q-Learning algorithm to store multiple Pareto-optimal values for each state-action pair and use dominance pruning to eliminate dominated strategies.

[0075] The most widely used MORL algorithm is the preference-based update method, which optimizes the entire preference weight space in a domain, providing a multi-objective version of Q-learning with linear preferences. The key idea is to use a vectorized value function and perform envelope updates, which update the parameters using a convex envelope that bounds the solution. By maintaining the envelope, Q-learning can quickly align a weight vector with the optimal reward and trajectory, but these rewards and trajectories are actually information collected under other weight vectors. Although envelope Q-learning can obtain larger Q values, the Q value network is not stable at the beginning of the algorithm, and the Q value obtained is too large.

[0076] In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure clearer, the embodiments of the present disclosure will be described in detail below with reference to the drawings. However, those skilled in the art can understand that in the embodiments of the present disclosure, many technical details are proposed in order to make the readers better understand the present disclosure. However, the technical solutions claimed by the present disclosure can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present disclosure, and the embodiments can be combined with each other and referred to each other without contradiction.

[0077] One embodiment of the present disclosure relates to a multi-agent cooperative and adversarial decision-making method based on reinforcement learning, which can be applied to a multi-agent cooperative and adversarial scene including our combat units, friendly combat units, and enemy combat units, especially in the enemy and our confrontation tasks in the micro-operation environment of real-time strategy games, to control the actions of our combat units.

[0078] As Figure 1 shown, the multi-agent cooperative and counter-competitive decision-making method based on reinforcement learning involves:

[0079] In step 110, the current state information is obtained, and the current state information includes self state information, friendly state information and enemy state information. The self state information of the self side can include but is not limited to the horizontal position, vertical position, blood volume and the like of the self side combat unit. The friendly state information can include but is not limited to the agent type, distance, horizontal position, vertical position, blood volume, and armor of the friendly combat unit. The enemy state information can include but is not limited to the agent type, distance, horizontal position, vertical position, blood volume, and armor of the enemy combat unit. It should be noted that the specific acquisition method of the current state information is not limited in the embodiment, and those skilled in the art can select according to actual needs.

[0080] In step 120, the current state information is input into the decision-making agent trained by the preset reinforcement learning algorithm to obtain a decision-making result, and the decision-making result includes any one of executing a moving action, an attacking action, and keeping still. Specifically, the decision-making agent herein is an intelligent model trained according to the preset reinforcement learning algorithm, and when used, the corresponding decision-making result can be obtained directly by inputting the current state information. When the application scenario is a multi-agent cooperative and counter-competitive scenario, the current state information is input into the decision-making agent to obtain the corresponding decision-making result, which can guide the self combat unit to obtain the optimal reward by executing the moving action or executing the attacking action or keeping still.

[0081] Next, the various variables involved in the preset reinforcement learning algorithm and their mutual relationships will be introduced.

[0082] Multi-objective Markov decision process (MOMDP) is generally described using the tuple <S, A, T, R, Ω>. S represents the state space of the system, and its elements are state tensors s. A represents the action space of the system, and its elements are action tensors a. T(SxAxS) is the probability transition function of the Markov process. is a reward function vector that describes the immediate rewards of different objectives. Ω is the weight space of the multi-objective, and its elements are weight vectors ω. The reward function R of the multi-objective Markov process is in vector form, where the length of the reward vector is equal to the number of objectives, and the comprehensive reward of the MOMDP is f ω (R(s, a))=ω TR(s, a). Multi-objective reinforcement learning algorithm learns the entire policy by training an agent, and then gives the optimal policy according to any given ω∈Ω when testing.

[0083] (s ω , a T , R t , s t ) represents the initial state, action, reward and returned state after interaction at time t when the agent interacts with the environment, and the Bellman optimal operator TQ under the form of multi-objective reinforcement learning is represented as: where, represents the expectation of (HQ)(s t , ω) on s t+1 , and γ represents the discount factor. The operator HQ is defined as: where, represents the upper bound with a t+1 , ω as the independent variable, and argmax t+1 represents the Q value corresponding to the extreme point. This operator simultaneously completes: (1) obtains the decision vector a that maximizes the Q value in the action space A; (2) adjusts the weight inside the Q function to obtain the maximum Q value given the external weight vector.

[0084] The deep neural network takes s, ω as input and outputs the Q value of |A|×m, where m is the dimension of the reward vector, which is the same as the dimension of ω. Let θ be the network parameter, and further update the network parameter at each step through the loss function, which is expressed as: where s' is the returned value of the next time of state s, and y is the actual value of Q and is expressed as

[0085] Optimizing L t indicates that the algorithm wants to maximize the Q value. However, since the estimation of Q value itself is not accurate at the beginning of the algorithm, the direct use of maximization will lead to overestimation of the Q value of part of the action space. Therefore, at the beginning of the algorithm training, ω used to update the Q value is randomly generated and combined with the action sequence one by one to generate experience tuples; in the later stage of the algorithm training, since the prediction of state-action value is relatively accurate, the weighted Q value time difference loss is further used for updating: Therefore, the final loss function can be expressed as L(θ)=(1-λ)L Q (θ)+λL A (θ), where λ is an L A and L Bbetween the two. A homotopy optimization is used to vary from 0 to 1, so that the loss function gradually changes from L A to L B For each update step, it uses the optimization result of the previous step as the initial guess.

[0086] In addition, for each experience tuple (s t , a t , R t , s t+1 ) in the batch, N ω weight vectors are additionally sampled to reuse the experience. However, under the condition of linear weights, the agent will give similar strategies for similar weights, which can provide more information to help the agent learn the current policy, so limiting the newly sampled preferences to the neighborhood of the actual sampling ω can improve the efficiency of sample utilization.

[0087] In the proximal policy optimization (PPO) algorithm, the network framework of the algorithm is as shown in Figure 2 The spatial graph layer information is first extracted by convolution layer 1 and convolution layer 2, and then the extracted features are merged into a feature vector by flattening and the like. The feature vector is spliced with the weight vector. The weight vector is obtained by the fully connected layer output of the weight vector information activated by the tanh function. The spliced feature vector and weight vector are input into the fully connected layer activated by the ReLU function for processing to obtain the corresponding output result. The output result is processed by the Softmax normalization exponential function to output the action probability, and is processed by the fully connected layer to output the multi-value vector. The multi-value vector is combined with the weight vector output by the fully connected layer activated by the tanh function to obtain the final weight vector.

[0088] As shown in Figure 3 , the preset reinforcement learning algorithm includes the following steps:

[0089] Step 210, obtaining observation information of the decision-making agent, the observation information including self-state information of the decision-making agent, friendly-state information, and enemy-state information. Specifically, the self-state information of the decision-making agent can include but is not limited to the horizontal position, the vertical position, the blood volume, etc. of the friendly combat unit. The friendly-state information can include but is not limited to the agent type, the distance, the horizontal position, the vertical position, the blood volume, the armor, etc. of the friendly combat unit. The enemy-state information can include but is not limited to the agent type, the distance, the horizontal position, the vertical position, the blood volume, the armor, etc. of the enemy combat unit.

[0090] Step 220, the observation information and the weight vector are encoded by using the encoding weight matrix to obtain the weight and state feature collaborative encoding, and a target reward function is designed.

[0091] Specifically, for different forms of observation information, different encoding methods can be used. For example, for observation information whose features are stored in the form of a vector, a matrix splicing method can be directly used to combine it with the weight vector. For observation information in the form of a picture (three-dimensional tensor) or two or three dimensions, the features can be extracted through convolution operation first, and then the extracted features can be merged into a vector through flattening or other means, and then spliced with the weight vector. For observation information with adjacency matrix and node information, the node's own features and neighbor features can be extracted through single word or multiple graph convolution, and the extracted features can be spliced with the weight vector. In combination with Figure 4 , step 220 corresponds to step one in Figure 4 , i.e., weight and state feature collaborative encoding, which specifically includes Markov state definition and feature encoding, agent observation feature, information abstraction extraction, and weight space sampling.

[0092] Illustratively, in step 220, the observation information and the weight vector are encoded by using the encoding weight matrix to obtain the weight and state feature collaborative encoding, including:

[0093] The observation information is encoded according to the following formula (1) to obtain observation information encoding O:

[0094]

[0095] Wherein, O self ∈R OS represents the state information of the decision-making agent itself, including but not limited to the horizontal position, the vertical position, the blood volume, etc. R represents the reward function. OS represents the information dimension of the decision-making agent. represents the vectorization of O f , i.e., the vectorized representation of O f . O f ∈R OF×NF represents the friendly state information, including but not limited to the friendly agent type, the distance, the horizontal position, the vertical position, the blood volume, the armor, etc. OF represents the friendly information dimension. NF represents the number of friendly combat units. represents the vectorization of O e , i.e., the vectorized representation of O e . O e ∈R OE×NE represents the enemy state information, including but not limited to the enemy agent type, the distance, the horizontal position, the vertical position, the blood volume, the armor, etc. OE represents the enemy information dimension, and NE represents the number of enemy combat units.

[0096] The observation information coding O is coded according to the following formula (2) to obtain state information coding S:

[0097] S = F1 x O (2)

[0098] wherein F1 e R E1×NO represents a first coding weight matrix, O e R NO , NO = OS + OF * NF + OE * NE, S e R E1 , E1 represents a state feature dimension corresponding to the state information coding S.

[0099] The weight vector is coded according to the following formula (3) to obtain weight vector coding FW:

[0100] FW = F2 x ω (3)

[0101] wherein FW e R E2 , F2 e R E2×NW represents a second coding weight matrix, ω e R NW represents a weight vector, NW represents a weight vector dimension, and E2 represents a weight feature dimension corresponding to the weight vector coding FW.

[0102] The weight and state feature collaborative coding C is obtained according to the following formula (4): sw

[0103] C sw = {S, FW} (4).

[0104] By coding the observation information and the weight vector using the coding weight matrix, the generality of the algorithm can be effectively enhanced.

[0105] For example, in a task scenario, the target reward function is represented by the following formula (5):

[0106] f ω (R(s, a)) = ω T R(s, a) = ω1R1(s, a) + ω2R2(s, a) (5)

[0107] wherein f ω (R(s, a)) represents the target reward function, s represents a state tensor corresponding to a decision-making agent, a represents an action tensor corresponding to the decision-making agent, R1(s, a) represents a kill enemy combat unit reward, ω1 represents a weight corresponding to R1(s, a), R2(s, a) represents a remaining blood volume reward of the decision-making agent, and ω2 represents a weight corresponding to R2(s, a).

[0108] ​Step 230, based on the weight space where the weight vector is located, multi-state value resampling is performed, the corresponding overall trajectory Q value is determined by combining the weight and state feature collaborative coding, and the optimal augmented tuple sequence corresponding to the weight and state feature collaborative coding is determined. In combination Figure 4 , step 230 corresponds Figure 4 step two in step 230, that is, multi-state value resampling, based on multi-state value decoding, returns a state trajectory containing a weight value.

[0109] Exemplarily, step 230 includes:

[0110] The overall trajectory Q value is calculated according to the following formula (6):

[0111]

[0112] Wherein, Q π (s, a) represents the overall trajectory Q value, π represents the action policy distribution, γ represents the decay coefficient, t represents the time, R t represents the reward function at t time, s0 represents the initial state at t=0 time when the decision-making agent interacts with the environment, a0 represents the action at t=0 time when the decision-making agent interacts with the environment, represents the mathematical expectation.

[0113] n uniform random samplings are performed on the weight space Ω where the weight vector ω is located, n weight vectors ω are obtained, the n weight vectors ω are combined with the state transition sequence respectively, and m×n×K state tuples (s t , a t , R t , s t+1 , ω) are obtained as augmented tuple sequences, wherein m represents the dimension of the reward vector, K represents the number of state transition sequences, s t represents the initial state at t time when the decision-making agent interacts with the environment, a t represents the action at t time when the decision-making agent interacts with the environment, s t+1 represents the state returned after interaction at t time when the decision-making agent interacts with the environment.

[0114] For any given transition tuple sequence (s t , a t , R t , s t+1 ), the optimal weight vector ω * is calculated in the n weight vectors ω sampled, that is, ω ω = argmax T ω t+1 Q(s t , a t , ω), and the corresponding optimal augmented tuple sequence (s t, R t , s t+1 , ω * ), wherein Q(s t+1 , a t , ω) represents the Q value of (s t+1 , a t , ω).

[0115] In step 240, the loss function of the policy network and the total progressive envelope loss function of the value network are determined based on the optimal tuple sequence respectively, and the parameters of the policy network and the value network are updated based on the loss function of the policy network and the total progressive envelope loss function. For details, please refer to the description of step 240 in the first embodiment. Figure 4 Step 240 corresponds to step three in Figure 4 , that is, parameter updating.

[0116] For example, the loss function of the policy network is represented by the following formula (7):

[0117]

[0118] Wherein L π (θ) represents the loss function of the policy network, π θ and are the new and old policy distributions, θ and θ old are the parameters of π θ and , represents the average of the sample, is the augmented advantage function and is represented by the following formula (8),

[0119]

[0120] Wherein V ππ is the value network learned according to the current policy π, θ V is the parameter of V π , Q π (s t , a t , ω) represents the state trajectory Q value corresponding to the current policy π.

[0121] That is, for each fixed weight vector ω and optimal augmented tuple sequence, the augmented advantage function can be calculated in the corresponding state trajectory according to the above formula (8). On this basis, the loss function of the policy network can be represented by the above formula (7).

[0122] For example, the total progressive envelope loss function of the value network is represented by the following formula (9):

[0123]

[0124] wherein L(0) represents the total asymptotic envelope loss function of the value network, denotes the trade-off weight, k is the iteration number, and τ is the delay coefficient, is the initial training loss function of the value network and is represented by the following formula (10), is the later training loss function of the value network and is represented by the following formula (11),

[0125]

[0126]

[0127] wherein y1 and y2 are both intermediate variables and are represented by the following formula (12) and formula (13), respectively

[0128]

[0129]

[0130] wherein γ represents the discount factor.

[0131] Specifically, the estimation loss of the value network is related to the iteration step and is realized by λ to achieve soft conversion, and thus the initial training loss function of the value network is represented by the above formula (10), the later training loss function of the value network is represented by the above formula (11), and the final total asymptotic envelope loss function is represented by the above formula (9).

[0132] Step 250, based on the target reward function, uses the policy network and the value network updated by the parameters to perform decision decoding on the weight and state feature collaborative coding, and outputs a decision result. In combination with Figure 4 , step 250 corresponds to Figure 4 step four in the above-mentioned step four, that is, the decision decoding output based on the decision feature decoding.

[0133] Exemplarily, step 250 includes calculating the decision action probability according to the following formula (14):

[0134]

[0135] wherein P represents the decision action probability, softmax represents the normalized exponential function, NA represents the dimension of the decision action, i represents the dimension sequence number of the decision action, output=FC(C sw ) represents and output∈R NA , output i represents the i-th element of output, and FC represents the full connection network.

[0136] Specifically, the decision action herein includes but is not limited to moving, attacking, keeping still, i.e. not doing any action, etc.

[0137] The preset reinforcement learning algorithm provided by the embodiment of the present disclosure changes the reward from a scalar to a vector in the framework of reinforcement learning, and also extends the output of the value network to a vector, introduces a controllable weight preference vector at the state input end to adaptively and dynamically meet the target requirements, and adapts and optimizes the PPO algorithm for multiple advantage functions, so that the Q-Learning is applicable to the policy gradient method. In order to improve the training effect of reinforcement learning, the reinforcement learning algorithm modifies the sampling process of training, expands the range of the preference vector sampling in the early stage to obtain more sufficient samples, and in the updating process, the network is trained by using a gradual envelope, all augmented trajectory sequences are returned in the early stage of training, more uniform exploration is performed in the whole state space, the problem of deviation of the weight vector caused by insufficient training of the value network when searching for the optimal weight vector is avoided, and the convergence speed of the value network is further accelerated.

[0138] The multi-agent cooperative and antagonistic decision-making method based on reinforcement learning provided by the embodiment of the present disclosure can make the agent make corresponding decisions based on the current state in the process of multi-agent cooperative and antagonistic decision-making, and effectively improve the accuracy of the decision-making result.

[0139] Another embodiment of the present disclosure relates to a multi-agent cooperative and antagonistic decision-making device based on reinforcement learning, as shown in Figure 5 The device comprises:

[0140] The acquisition module 510 is configured to acquire current state information, wherein the current state information comprises self-state information, friend-state information and enemy-state information.

[0141] The decision module 520 is configured to input the current state information into a decision-making agent trained by using a preset reinforcement learning algorithm to obtain a decision-making result, wherein the decision-making result comprises any one of a moving action, an attacking action and a stationary action.

[0142] The training module 530 is configured to train the decision-making agent according to a preset reinforcement learning algorithm including the following steps: obtaining observation information of the decision-making agent, the observation information including self-state information, friendly-state information and enemy-state information of the decision-making agent; encoding the observation information and a weight vector by using an encoding weight matrix to obtain weight and state feature collaborative coding, and designing a target reward function; performing multi-state value resampling based on a weight space in which the weight vector is located, determining an optimal augmented tuple sequence corresponding to the weight and state feature collaborative coding in combination with an overall trajectory Q value corresponding to the weight and state feature collaborative coding; determining a loss function of a policy network and a total asymptotic envelope loss function of a value network based on the optimal tuple sequence, and performing parameter updating on the policy network and the value network based on the loss function of the policy network and the total asymptotic envelope loss function; and performing decision decoding on the weight and state feature collaborative coding by using the policy network and the value network after the parameter updating based on the target reward function, and outputting a decision result.

[0143] The specific implementation method of the multi-agent collaborative and antagonistic decision device based on reinforcement learning provided by the embodiments of the present disclosure can be referred to the method of multi-agent collaborative and antagonistic decision based on reinforcement learning provided by the embodiments of the present disclosure, which will not be described here.

[0144] The multi-agent collaborative and antagonistic decision device based on reinforcement learning provided by the embodiments of the present disclosure can make the agent make corresponding decisions based on the current state in the multi-agent collaborative and antagonistic process in real time, and effectively improve the accuracy of the decision result, compared with the prior art.

[0145] Another embodiment of the present disclosure relates to an electronic device, as shown in the accompanying drawings, comprising: Figure 6 as shown in the accompanying drawings, comprising:

[0146] at least one processor 601; and

[0147] The memory 602 is in communication connection with the at least one processor 601; wherein

[0148] The memory 602 stores instructions executable by the at least one processor 601, and the instructions are executed by the at least one processor 601 to enable the at least one processor 601 to perform the method of multi-agent collaborative and antagonistic decision based on reinforcement learning described in the above embodiments.

[0149] The memory and the processor are connected via a bus. The bus can include any number of interconnecting buses and bridges depending on the specific application of the mobile terminal. The bus connects the various circuits of the memory and the processor together and mediates data communication among different components. The bus can also connect with the various other circuits such as peripheral devices, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described further. A bus interface provides an interface between the bus and a transceiver. The transceiver can be a single device or a plurality of devices such as a plurality of receivers and transmitters that are arranged to transmit and receive data respectively over a transmission medium. The data processed by the processor is transmitted over a wireless medium via an antenna. Further, the antenna also receives data and transmits the data to the processor.

[0150] The processor is responsible for managing the bus and general processing, and can also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory can be used for storing data used by the processor when executing operations.

[0151] Another embodiment of the present disclosure relates to a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the method for multi-agent collaborative and confrontation decision based on reinforcement learning according to the above-mentioned embodiments.

[0152] That is, those skilled in the art can understand that all or part of the steps of the method according to the above-mentioned embodiments can be completed by a program instructing relevant hardware, and the program is stored in a storage medium, including a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method according to the embodiments of the present disclosure. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0153] Those skilled in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present disclosure, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present disclosure.

Claims

1. A multi-agent cooperative and adversarial decision-making method based on reinforcement learning, characterized in that, The decision method comprises: obtaining current state information, the current state information comprising self state information, ally state information and enemy state information; inputting the current state information into a decision agent trained by using a preset reinforcement learning algorithm to obtain a decision result, the decision result comprising any one of a moving action, an attacking action and keeping still; the preset reinforcement learning algorithm comprises the following steps: obtaining observation information of the decision agent, the observation information comprising self state information, ally state information and enemy state information of the decision agent; encoding the observation information and a weight vector by using an encoding weight matrix to obtain weight and state feature collaborative encoding, and designing a target reward function; based on a weight space where the weight vector is located, performing multi-state value resampling, combining an overall trajectory Q value corresponding to the weight and state feature collaborative encoding to determine an optimal augmented tuple sequence corresponding to the weight and state feature collaborative encoding; based on the optimal augmented tuple sequence, determining a loss function of a policy network and a total asymptotic envelope loss function of a value network respectively, and performing parameter updating on the policy network and the value network based on the loss function of the policy network and the total asymptotic envelope loss function; based on the target reward function, performing decision decoding on the weight and state feature collaborative encoding by using the policy network and the value network after parameter updating to output a decision result; the total asymptotic envelope loss function of the value network is represented by the following formula (9): The overall trajectory Q value is calculated according to the following formula (6), wherein Q π (s, a) represents the overall trajectory Q value, π is an action policy distribution, γ is a decay coefficient, t represents a time, R t represents a reward function at time t, s0 represents an initial state at time t = 0 when the decision-making agent interacts with the environment, a0 represents an action at time t = 0 when the decision-making agent interacts with the environment, represents a mathematical expectation: n times uniform random sampling is performed on a weight space Ω in which the weight vector ω is located, to obtain n weight vectors ω, the n weight vectors ω are combined with the state transition sequence respectively, to obtain m×n×K state tuples (s t ,a t ,R t ,s t+1 ,ω) as an augmented tuple sequence, wherein m represents a dimension of a reward vector, K represents a number of state transition sequences, s t represents an initial state at t moment when the decision-making agent interacts with the environment, a t represents an action at t moment when the decision-making agent interacts with the environment, s t+1 represents a returned state after interaction at t moment when the decision-making agent interacts with the environment. For any given sequence of transition tuples (s) t ,a t ,R t ,s t+1 ), calculate the optimal weight vector ω from the n sampled weight vectors ω. * =argmax ω ω T Q(s t+1 ,a t ,ω), to obtain the corresponding optimal augmented tuple sequence (s t ,a t ,R t ,s t+1 ,ω * ), where Q(s) t+1 ,a t ,ω) represents (s t+1 ,a t The Q value of (ω); wherein y1 and y2 are both intermediate variables and are represented by the following formula (12) and formula (13) respectively: where L(θ) represents a total asymptotic envelope loss function of the value network, denotes a trade-off weight, k is the number of iterations, and τ is a delay coefficient, is a loss function at the initial stage of training of the value network and is represented by the following equation (10), is a loss function at the final stage of training of the value network and is represented by the following equation (11), the encoding of the observation information and the weight vector by using the encoding weight matrix to obtain the weight and state feature collaborative encoding comprises:

2. The decision method of claim 1, wherein, S = F1 x O (2) The observation information is encoded according to the following formula (1) to obtain the observation information code O, where O self ∈R OS Let R represent the state information of the decision-making agent, R represent the reward function, and OS represent the information dimension of the decision-making agent. For O f The vectorized representation of O f ∈R OF×NF This represents friendly status information, OF represents the friendly information dimension, and NF represents the number of friendly combat units. For O e The vectorized representation of O e ∈R OE×NE This indicates enemy status information, OE represents the enemy information dimension, and NE represents the number of enemy combat units. The observation information coding O is coded according to the following formula (2) to obtain state information coding S, wherein F1∈R E1×NO represents the first coding weight matrix, O∈R NO , NO= OS+ OF* NF+ OE* NE, S∈R E1 E1 represents the state feature dimension corresponding to the state information coding S: EW = F2 x ω (3) The weight vector is encoded according to the following formula (3) to obtain a weight vector code FW, wherein FW∈R E2 , F2∈R E2×NW , indicates a second encoded weight matrix, ω∈R NW , indicates a weight vector, NW indicates a weight vector dimension, and E2 indicates a weight feature dimension corresponding to the weight vector code FW. the target reward function is represented by the following formula (5): The weight and state feature collaborative coding c is obtained according to the following formula (4) sw : C sw = {S, FW} (4).

3. The decision method of claim 2, wherein, the loss function of the policy network is represented by the following formula (7): f ω (R(s,a)) = ω T R(s,a) = ω1R1(s,a) + ω2R2(s,a) (5) wherein f ω (R(s,a)) represents the target reward function, s represents a state tensor corresponding to the decision-making agent, a represents an action tensor corresponding to the decision-making agent, R1(s,a) represents a reward for killing enemy combat units, ω1 represents a weight corresponding to R1(s,a), R2(s,a) represents a reward for the remaining blood volume of the decision-making agent, and ω2 represents a weight corresponding to R2(s,a).

4. The decision method of claim 3, wherein, based on the target reward function, performing decision decoding on the weight and state feature collaborative encoding by using the policy network and the value network after parameter updating to output a decision result, comprising: where L π (θ) denotes the loss function of the policy network, π θ and are the new and old policy distributions, θ and θ old are the parameters of π θ and respectively, denotes taking the mean over samples, is the augmented advantage function and is denoted as follows (8), where V π is the value network learned according to the current policy π, θ V is the parameter of V π , and Q π (s t , a t , ω) represents the state trajectory q value corresponding to the current policy π.

5. The decision method of claim 4, wherein, calculating a decision action probability according to the following formula (14): the decision device comprises: wherein P denotes the decision action probability, softmax denotes a normalized exponential function, NA denotes a dimension of the decision action, i denotes a dimension serial number of the decision action, output = FC(C sw ) denotes and output ∈ R NA , output i denotes an i-th element of output, and FC denotes a fully connected network.

6. A multi-agent cooperative and adversarial decision-making device based on reinforcement learning, characterized in that, an obtaining module configured to obtain current state information, the current state information comprising self state information, ally state information and enemy state information; a decision module configured to input the current state information into a decision agent trained by using a preset reinforcement learning algorithm to obtain a decision result, the decision result comprising any one of a moving action, an attacking action and keeping still; a training module configured to train the decision agent according to the preset reinforcement learning algorithm comprising the following steps: ​ Obtain observation information of a decision-making agent, the observation information including self-state information, friendly-state information and enemy-state information of the decision-making agent; Encode the observation information and a weight vector by using an encoding weight matrix to obtain weight and state feature collaborative coding, and design a target reward function; Based on a weight space in which the weight vector is located, perform multi-state value resampling, combine an overall trajectory Q value corresponding to the weight and state feature collaborative coding, and determine an optimal augmented tuple sequence corresponding to the weight and state feature collaborative coding; Determine a loss function of a policy network and a total asymptotic envelope loss function of a value network based on the optimal augmented tuple sequence, and perform parameter updating on the policy network and the value network based on the loss function of the policy network and the total asymptotic envelope loss function; Based on the target reward function, perform decision decoding on the weight and state feature collaborative coding by using the policy network and the value network after parameter updating, and output a decision result; The total asymptotic envelope loss function of the value network is expressed as the following formula (9): The overall trajectory Q value is calculated according to the following formula (6), wherein Q π (s, a) represents the overall trajectory Q value, π is an action policy distribution, γ is a decay coefficient, t represents a time, R t represents a reward function at time t, s0 represents an initial state at time t = 0 when the decision-making agent interacts with the environment, a0 represents an action at time t = 0 when the decision-making agent interacts with the environment, represents a mathematical expectation: n times uniform random sampling is performed on a weight space Ω in which the weight vector ω is located, to obtain n weight vectors ω, the n weight vectors ω are combined with the state transition sequence respectively, to obtain m×n×K state tuples (s t ,a t ,r t ,s t+1 ,ω) as an augmented tuple sequence, wherein m represents a dimension of a reward vector, K represents a number of state transition sequences, s t represents an initial state at t time when the decision-making agent interacts with the environment, a t represents an action at t time when the decision-making agent interacts with the environment, s t+1 represents a returned state after interaction at t time when the decision-making agent interacts with the environment. For any given sequence of transition tuples (s) t ,a t ,R t ,s t+1 ), calculate the optimal weight vector ω from the n sampled weight vectors ω. * =argmax ω ω T Q(s t+1 ,a t ,ω), to obtain the corresponding optimal augmented tuple sequence (s t ,a t ,R t ,s t+1 ,ω * ), where Q(s) t+1 ,a t ,ω) represents (s t+1 ,a t The Q value of (ω); Wherein, y1 and y2 are intermediate variables and are expressed as the following formula (12) and the following formula (13) respectively: where L(0) represents a total asymptotic envelope loss function of the value network, denotes a trade-off weight, k is the number of iterations, and τ is a delay coefficient, is a loss function at the initial stage of training of the value network and is represented by the following equation (10), is a loss function at the final stage of training of the value network and is represented by the following equation (11), Including:

7. An electronic device, comprising: At least one processor; And The memory is in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multi-agent collaborative confrontation decision-making method based on reinforcement learning in any one of claims 1 to 5. The computer program is executed by the processor to implement the multi-agent collaborative confrontation decision-making method based on reinforcement learning in any one of claims 1 to 5.

8. A computer readable storage medium storing a computer program, characterized in that, ​

Images