Improved DS-PPO reinforcement learning method for intelligent game deduction

Abstract

The invention discloses an improved DS-PPO reinforcement learning method for intelligent game deduction. In order to enable an intelligent agent to make a more accurate decision in a complex war game deduction environment, the DS-PPO firstly introduces a dual-state input mechanism and an action mask mechanism according to the characteristics of large-scale state-action space and environment asymmetry, high randomness and the like of the war game deduction environment; therefore, the intelligent agent can better perceive global environment information and avoid invalid actions. Secondly, an original loss function is redesigned, masks for processing illegal actions and self-defined entropy loss are introduced, exploratory performance of a strategy is enhanced, an intelligent agent is prevented from falling into a local optimal solution, and instability of strategy updating is reduced;

Description

Technical Field

[0001] The present invention relates to the improvement of algorithms in the field of reinforcement learning, and specifically to an improved DS-PPO reinforcement learning method for intelligent game deduction. Background Art

[0002] With the development of artificial intelligence technology, significant progress has also been made in the field of intelligent game confrontation.

[0003] AlphaGo, AlphaStar and other intelligent agents have defeated humans in the fields of Go and games, promoting the rapid development of intelligent agents. In recent years, reinforcement learning agents have also been applied to the field of military confrontation, such as the war game confrontation simulation platforms such as "Mozi" and "Miaosuan", which have made important contributions to the application of decision-making intelligent agents in the military field.

[0004] War games, as an important tool for simulating wars, play a key role in future battlefields. Traditional rule-based decision-making is difficult to cope with the complex environment in war games, while reinforcement learning agents can better conduct autonomous learning and decision-making during the game process. However, there are also a series of challenges in applying reinforcement learning to war games at this stage. The actions in war games contain extremely complex tactical ideas, which puts higher demands on the cognitive and decision-making capabilities of agents. At the same time, war games are more in line with the development trend of agents because of their characteristics of incomplete information, large-scale state-action space, environmental asymmetry, and high randomness.

[0005] How to improve the autonomous learning and decision-making ability of intelligent agents can be mainly started from three aspects. First, in war game simulation, the environmental information is extremely complex, and its state is usually highly complex and multidimensional. Each state contains a lot of information, such as the position of our operator, the position of the enemy operator, the seizure point, etc. This amount of information is huge and highly correlated, forming a high-dimensional state space. To address this problem, the global state and local state can be received in the Actor and Critic networks, and processed through two GRU networks. Secondly, in war game simulation, the actions of the intelligent agent need to be restricted at certain moments, such as attacking when the enemy operator is not found. These actions are illegal in certain states. By constraining the action space of the intelligent agent, illegal actions are blocked when the policy network is output, and the learning ability of the intelligent agent in the simulation process is improved. Finally, it is necessary to redesign the original loss function to strengthen the exploratory nature of the strategy, effectively avoid the intelligent agent from falling into the local optimal solution, and enhance the exploration ability of the intelligent agent. Summary of the invention

[0006] Aiming at the problems existing in the prior art: large-scale state-action space, asymmetric environment, high randomness and other characteristics of the war game simulation environment, the present invention proposes a DS-PPO reinforcement learning method based on PPO improvement to solve the above-mentioned problems.

[0007] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: an improved DS-PPO reinforcement learning method for intelligent game deduction, the method uses Actor-Critic as a network framework to perform the following steps to realize the interaction between the intelligent agent and the surrounding environment, collect environmental states, make corresponding strategic actions, and act on the environment; the method includes the following steps:

[0008] S1, the agent obtains the current battlefield situation s in the environment t {local current state s1, global history state s2};

[0009] S2, the Actor network in the agent is based on the current battlefield situation s t , action space, using the action mask mechanism to dynamically adjust the optional action set, and applying the strategy to generate an action a t ;

[0010] S3, the agent performs actions a to the environment t , get the battlefield situation s at the next moment of environmental feedback t+1 and reward r t , generate historical trajectory information τ{the battlefield situation at the next moment s t+1 、Current reward r t};

[0011] S4, the Critic network in the agent receives the historical trajectory information τ and calculates the state value and advantage function at the current moment;

[0012] S5, calculate the policy loss function and value loss function by the policy gradient method and mean square error method respectively to update the parameters of the Actor and Critic networks; superimpose the time steps, return to step S1 to iterate and calculate to gradually improve the policy actions, and obtain the ideal intelligent agent model;

[0013] S6. In an unknown environment, obtain one or more unknown battlefield situations and input them into the ideal intelligent agent model, and automatically output actions.

[0014] The Actor network includes: a parallel GRU network, a concatenation layer, a fully connected linear layer, and a mask layer; the two parallel GRU networks are used to capture the time correlation of the environment, respectively extract the features of the local current state s1 and the global historical state s2, and then perform a concatenation operation Concatenate, and generate fused state features through a series of fully connected linear layers to represent the local state and the global state; the mask layer is used to shield the impossible actions in the optional action set, and apply the strategy function to generate an action a t .

[0015] The application strategy function calculates the probability distribution of the action π(a|s t ), generates an action a t ,include:

[0016] f θ ′(s t ,a)=f θ (s t ,a)·m(a)+(1-m(a))·(-∞)

[0017] π(a|s t )=softmax(f θ ′(s t ,a))

[0018] Among them, f θ (s t , a) is the action logits output by the Actor neural network; m(s) is the action mask, m(a) = 0 means that the action cannot occur, and m(a) = 1 means that the action may occur; through the mask operation, the impossible action is set to infinity to shield it.

[0019] The Critic network includes: a parallel GRU network, a concatenation layer, a fully connected linear layer, a state value calculation unit, and an advantage function calculation unit; two parallel GRU networks are used to capture the time correlation of the environment, respectively provide the features of the local current state s1 and the global historical state s2, perform concatenation operations, and generate fused state features through a series of fully connected linear layers to represent the local state and the global state; the state value calculation unit calculates the given state s according to the value function t = the state value at time s; the advantage function calculation unit adopts the weighted temporal difference error GAE method to perform weighted summation on the temporal difference error.

[0020] The value function is:

[0021]

[0022] Among them, π represents the strategy adopted by the agent, γ is the discount factor, and r t+k represents the reward at time step t+k.

[0023] The advantage function

[0024] δ t =r t +γV(s t+1 )-V(s t )

[0025]

[0026] Among them, δ t is the temporal difference error with time step t, (γλ) u is a combination of the discount factor γ and the GAE parameter λ, representing the discount for time step u.

[0027] It uses the strategy loss function L clip (θ), value loss function L value (φ) Update the parameters θ and φ of the Actor network and Critic network respectively;

[0028]

[0029] L value (φ)=E t [v φ (s t )-r t ] 2

[0030] Among them, θ and φ are the parameters of the two networks, Used to calculate the ratio between the current strategy and the old strategy, ∈ is the clipping coefficient;

[0031]

[0032] α and β represent the learning rate of the network, which are used to control the magnitude of each update. Represents the gradient of the two. The former maximizes the strategy loss function L by gradient ascent clip (θ), which minimizes the value loss function L by gradient descent value (φ), ← indicates assignment update.

[0033] Advantages and beneficial effects of the present invention:

[0034] 1. By introducing the dual-state input mechanism, the intelligent agent can more comprehensively perceive the global information in the game deduction environment, so that it can make more accurate decisions when facing large-scale state-action space, enhancing the adaptability of the intelligent agent in complex environments.

[0035] 2. Using the action mask mechanism, the agent can identify and avoid performing invalid or illegal actions, which not only improves the effectiveness of strategy execution, but also reduces invalid attempts during training and improves the decision-making efficiency of the agent.

[0036] 3. By redesigning the loss function, introducing illegal action masks and custom entropy losses, the present invention strengthens the agent's exploration ability and effectively prevents it from falling into a local optimal solution. In addition, the strategy update is more stable, reducing the uncertainty caused by strategy fluctuations during training and improving the overall deduction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the interaction between the DS-PPO and the environment proposed by the present invention;

[0038] Figure 2 This is the structural diagram of the DS-PPO proposed in the present invention;

[0039] Figure 3 It is a diagram of the intelligent agent control operation unit of the present invention; DETAILED DESCRIPTION

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0042] Aiming at the large-scale state-action space, environmental asymmetry, and high randomness of the war game environment, this paper proposes an improved reinforcement learning algorithm DS-PPO, which enables the intelligent agent to make more accurate decisions in a complex war game environment. DS-PPO first introduces a dual-state input mechanism and an action mask mechanism, which enables the intelligent agent to better perceive global environmental information and avoid invalid actions. Secondly, the original loss function is redesigned, and a mask for handling illegal actions and a customized entropy loss are introduced to enhance the exploratory nature of the strategy, prevent the intelligent agent from falling into a local optimal solution, and reduce the instability of the strategy update.

[0043] The improvements of the DS-PPO algorithm of the present invention include:

[0044] Stage 1: Introducing a dual-state input mechanism, obtaining the global state s1 and the local state s2 from the environment. The global state s1 provides complete information about the current scene, while the local state s2 records historical information. By inputting these two states into independent GRU layers, the temporal correlation of the environment is captured;

[0045] Stage 2: The action mask mechanism dynamically adjusts the set of optional actions according to the battlefield situation and the state of the agent, and masks impossible actions. Specifically, by setting the infeasible actions to infinitesimal values, the policy network is ensured to focus on reasonable action selection during the decision-making process;

[0046] Stage 3: Strategy loss function L clip (θ) and the value loss function L value (φ) is optimized. In the policy loss function, the calculation method of dual-state input and action mask is introduced to avoid excessive policy update. At the same time, the weighted temporal difference error (GAE) is used for the calculation of the advantage function, by weighted summing the temporal difference error;

[0047] Stage 4: Dynamic adjustment mechanism of action mask based on the current battlefield situation and unit status. The agent dynamically adjusts the action set according to the current status information at each time step to ensure that each unit can make the best decision at the right time.

[0048] Figure 1 This is the flow chart of the interaction between DS-PPO and the environment proposed by the present invention. Its specific theory and functions are as follows:

[0049] The DS-PPO algorithm uses Actor-Critic as the network framework and introduces the GRU network to process local and global states, thereby constructing an agent network. At each time step t, the agent t (s1, s2) uses the Actor network to generate an action at , where s1 and s2 represent the global state and local state respectively. The environment receives action a t And feedback to the agent's next state s t+1 and reward r t . This information is recorded to form a trajectory τ={(s0,a0,r0),…(s T ,a T ,r T )}, T is the length of the trajectory. The entire trajectory τ is then passed to the Critic network. The Critic network calculates the value of each time step t according to the value function formula.

[0050]

[0051] Among them, π represents the strategy adopted by the agent, γ is the discount factor, and r t+k represents the reward at time step t+k. Specifically, V(s) represents the reward at a given state s t = s, the expected discounted reward V(s) that can be obtained in all future time steps starting from time step t to the end of the deduction process t ).

[0052] like Figure 2 As shown in Figure 1, in the DS-PPO algorithm, the agent receives the global state s1 and the local state s2. The algorithm is designed to effectively utilize the dual input of global state and local state in a complex war game environment to enhance decision-making ability.

[0053] s1 provides the state information of the current scene at time step t. s2 contains the historical information observed in the previous time step. s1 and s2 are input into two independent GRU layers for processing. The GRU layer captures the temporal correlation in the state sequence by gradually updating the hidden state and outputs the hidden state of the last time step. Subsequently, the outputs of the two GRU layers are connected and processed through a series of fully connected layers and activation functions. Connecting s1 and s2 to integrate global and local information allows the agent to consider the current global situation and all past experience when making decisions. The agent has more comprehensive information and a better understanding of the environment through the combination of s1 and s2. In the Actor network, impossible actions are masked by action masks, and the probability distribution of actions π(a|s t ), as shown in the second formula below.

[0054] f θ ′(s t ,a)=f θ (s t ,a)·m(a)+(1-m(a))·(-∞)

[0055] π(a|s t )=softmax(f θ ′(s t ,a))

[0056] Among them, f θ (s t ,a) is the output (logits) of the neural network, that is, the logits probability of each action. The action mask is m(a), m(a) = 0 means that the action cannot happen, and m(a) = 1 means that the action can happen. Through the mask operation, the impossible action is set to infinity. a is the action, and softmax is the classifier.

[0057] The critic outputs the state of the valuation, and calculates the advantage function based on the value function at different time steps This is shown in the second formula below.

[0058] δ t =r t +γV(s t+1 )-V(S t )

[0059]

[0060] δ t is the time series difference error (TD error) with a time step of τ, (γλ) u is a combination of the discount factor γ and the ΓAE parameter λ, which represents the discount for time step u. By weighted summing of TD errors, the advantage function can be estimated more stably and the policy update effect can be improved.

[0061] When the Acor network and the Critic network are updated, the strategy loss function L needs to be calculated first. clip (θ) and the value loss function L value (φ), this paper introduces dual-state input and action mask, which are calculated as shown in the first and second formulas below respectively.

[0062]

[0063] L value (φ)=E t [v φ (s t )-r t ] 2

[0064] θ and φ are the parameters of the two networks, It is used to calculate the ratio between the current strategy and the old strategy, and ∈ is the clipping coefficient. The two are used to limit the amplitude of the strategy update to prevent the difference between the new strategy and the old strategy from being too large. Then, the policy gradient method and the mean square error loss function are used to update the parameters θ and φ of the Actor network and the Critic network respectively, as shown in the following two formulas.

[0065]

[0066] α and β represent the learning rate of the network, which are used to control the magnitude of each update. Represents the gradient of the two. The former maximizes the strategy loss function L by gradient ascent clip (θ), which minimizes the value loss function L by gradient descent value (φ), ← indicates assignment update.

[0067] Figure 3 This is a diagram of the intelligent agent control operation unit of the present invention, as shown in the figure:

[0068] The agent controls multiple combat units, including unmanned ground vehicles, drones, and ground soldiers. During the simulation, each combat unit is faced with a large number of optional actions at each moment. For example, unmanned combat vehicles {move, shoot, ..., support}, drones {fly, bomb, ..., signal interference}, etc. In order to speed up the decision-making process of the agent and reduce the selection of invalid actions, this paper proposes to introduce the concept of action mask when constructing the agent. The action mask can dynamically adjust the set of optional actions according to the current battlefield situation and combat unit status, thereby improving decision-making efficiency and rationality.

[0069] First, we define the set of actions that can be performed for each type of unit A = [a1, a2, a3, ... a n ], where n is the number of actions. Initially, each value in the action mask M is set to 1 (M = [1, 1, 1, ... 1]), that is, it is assumed that all actions are optional (0 means disabled, 1 means optional). The mask value is dynamically adjusted according to the current state and battlefield information. For example, the "fire" action is enabled or disabled based on whether the enemy information is found; the "occupy" action is enabled or disabled based on whether the force unit reaches the target point.

[0070] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. An improved DS-PPO reinforcement learning method for intelligent game deduction, characterized in that: The method uses Actor-Critic as a network framework to perform the following steps to enable the intelligent agent to interact with the surrounding environment, collect environmental states, make corresponding strategic actions, and act on the environment; the method includes the following steps: S1, the agent obtains the current battlefield situation s in the environment T {Local current state s1, global history state s2}; S2, the Actor network in the agent is based on the current battlefield situation s t , action space, using the action mask mechanism to dynamically adjust the optional action set, and applying the strategy to generate an action a t ; S3, the agent performs actions a to the environment t , get the battlefield situation s at the next moment of environmental feedback T+1 and reward r t , generate historical trajectory information τ{the battlefield situation at the next moment s t+1 、Current reward r t }; S4, the Critic network in the agent receives the historical trajectory information τ and calculates the state value and advantage function at the current moment; S5, calculate the policy loss function and value loss function by the policy gradient method and mean square error method respectively to update the parameters of the Actor and Critic networks; superimpose the time steps, return to step S1 to iterate and calculate to gradually improve the policy actions, and obtain the ideal intelligent agent model; S6. In an unknown environment, obtain one or more unknown battlefield situations and input them into the ideal intelligent agent model, and automatically output actions.

2. An improved DS-PPO reinforcement learning method for intelligent game deduction according to claim 1, characterized in that: The Actor network includes: a parallel GRU network, a concatenation layer, a fully connected linear layer, and a mask layer; the two parallel GRU networks are used to capture the time correlation of the environment, respectively extract the features of the local current state s1 and the global historical state s2, and then perform a concatenation operation Concatenate, and generate fused state features through a series of fully connected linear layers to represent the local state and the global state; the mask layer is used to shield the impossible actions in the optional action set, and apply the strategy function to generate an action a t .

3. An improved DS-PPO reinforcement learning method for intelligent game deduction according to claim 2, characterized in that: The application strategy function calculates the probability distribution of the action π(a|s t ), generates an action a t ,include: f θ ′(s t ,a)=f θ (s t ,a)·m(a)+(1-m(a))·(-∞) π(a|s t )=softmax(f θ ′(s t ,a)) Among them, f θ (s t , a) is the action logits output by the Actor neural network; m(a) is the action mask, m(a) = 0 means that the action cannot occur, and m(a) = 1 means that the action may occur; through the mask operation, the impossible action is set to infinity to shield it.

4. An improved DS-PPO reinforcement learning method for intelligent game deduction according to claim 1, characterized in that: The Critic network includes: a parallel GRU network, a concatenation layer, a fully connected linear layer, a state value calculation unit, and an advantage function calculation unit; two parallel GRU networks are used to capture the time correlation of the environment, respectively provide the features of the local current state s1 and the global historical state s2, perform concatenation operations, and generate fused state features through a series of fully connected linear layers to represent the local state and the global state; the state value calculation unit calculates the given state s according to the value function t = the state value at time s; the advantage function calculation unit adopts the weighted temporal difference error GAE method to perform weighted summation on the temporal difference error.

5. An improved DS-PPO reinforcement learning method for intelligent game deduction according to claim 4, characterized in that: The value function is: Among them, π represents the strategy adopted by the agent, γ is the discount factor, and r t+k represents the reward at time step t+k.

6. An improved DS-PPO reinforcement learning method for intelligent game deduction according to claim 4, characterized in that: The advantage function δ t =r t +γV(s t+1 )-V(S t ) Among them, δ t is the temporal difference error with time step t, (γλ) u is a combination of the discount factor γ and the GAE parameter λ, representing the discount for time step u.

7. An improved DS-PPO reinforcement learning method for intelligent game deduction according to claim 1, characterized in that: It uses the strategy loss function L clip (θ), value loss function L value (φ) Update the parameters θ and φ of the Actor network and Critic network respectively; L value (φ)=E t [v φ (s t )-r t ] 2 Among them, θ and φ are the parameters of the two networks, Used to calculate the ratio between the current strategy and the old strategy, ∈ is the clipping coefficient; α and β represent the learning rate of the network, which are used to control the magnitude of each update. Represents the gradient of the two. The former maximizes the strategy loss function L by gradient ascent clip (θ), which minimizes the value loss function L by gradient descent value (φ), ← indicates assignment update.

Images