Air combat combat strategy training method based on PPO algorithm and Transform-XL framework

Abstract

The invention discloses an air combat combat strategy training method based on a PPO algorithm and a Transform-XL framework, and the method comprises the steps: enabling an intelligent agent to have more advantages in the processing of long-time sequence data through the segmentation memory and the modeling of a long-range dependency relationship in the Gated Transform-XL framework, and improving the capturing capability of the intelligent agent for the long-time dependency relationship; the intelligent agent can better adapt to the dynamically changing battlefield environment, and the applicability and robustness of the strategy are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of battlefield simulation, and in particular relates to an air combat strategy training method based on a PPO algorithm and a Transformer-XL framework. Background Art

[0002] Traditional air combat strategies mainly refer to air combat methods based on experience and tactical rules without the support of modern intelligent technology. Generally speaking, this method first extracts relevant features from the original air combat data, and then uses these features to quantitatively evaluate the detection and attack threat capabilities of the enemy, or to evaluate the threat level of the enemy to the party, so as to evaluate the overall air combat situation. Next, a series of operational steps are formulated based on air combat experience and theory. Traditional algorithms include differential game methods, expert systems, and influence diagram methods. However, traditional algorithms face the problem of low computational efficiency, which limits their ability to cope with complex air combat environments.

[0003] Reinforcement learning is increasingly used in various fields, especially in decision-making and control problems in complex dynamic environments. So far, reinforcement learning has been widely used in games, robotics, autonomous driving, recommendation systems, and resource management.

[0004] Since 2016, deep reinforcement learning (DRL) methods have achieved remarkable success in the field of intelligent decision-making. The AlphaGo series of algorithms defeated the human champion in the field of Go, attracting global attention. AlphaFold v2 has achieved outstanding results in the field of protein structure, and AlphaTensor has achieved a breakthrough in the field of matrix fast algorithms. The famous scholar Silver once said that the foundation of general artificial intelligence (AGI) based on the DRL method is already in place. The outstanding performance of DRL in high-dimensional decision-making tasks has brought new opportunities to solve the dimensionality limitation problem of traditional air combat decision-making methods, prompting a series of research. In 2019, Kurniawan adopted the actor-critic (AC) architecture for air combat decision-making, combining the advantages of value-based and policy-based methods. In the same year, Yang et al. used the deep deterministic policy gradient (DDPG) method to solve the "dimensionality explosion" problem in traditional air combat decision-making and achieved action output in continuous space. Piao et al. achieved tactical innovation by self-evolving air combat actions through reinforcement learning methods.

[0005] Among the existing solutions, a method for studying combat strategies based on value approximation networks is proposed. In complex air combat processes, drones are required to make continuous multi-step decisions. If deep reinforcement learning is used, the action value function Q(s,a) needs to be obtained. Since each dimension in the flight state of the drone is a continuous value and cannot be recorded in a table, a common solution is to use the idea of ​​function approximation. The algorithm has a single combat strategy and a low winning rate in more difficult tasks, and the strategy is not very applicable. Therefore, it is necessary to select a combat strategy training method that is more suitable for complex air combat environments, so that the intelligent agent can perform better in more complex and changeable air combat environments. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide an air combat strategy training method based on the PPO algorithm and the Transformer-XL framework.

[0007] An air combat strategy training method, comprising:

[0008] The state vector of the aircraft is first passed into the multi-layer perceptron as input, adjusted to the embedding dimension required by the Transformer encoder block, and then input into multiple stacked Transformer encoder blocks to obtain the context information encoded by the input state vector as the state representation result;

[0009] The state representation results are input into the value network and policy network under the PPO framework respectively. The value network calculates the advantage function Adv(s,a); the policy network outputs the probability distribution of the action, and the agent randomly selects the action according to the probability distribution; then, the policy network calculates the action ratio, that is, the ratio between the new strategy and the old strategy, to measure the change in the probability of selecting a certain action a under a certain state s;

[0010] A loss function is calculated based on the advantage function Adv(s,a) and the action ratio. The value network and the policy network use the same loss function to train and update their respective network parameters. The loss function is a weighted sum of the policy loss, the value loss and the entropy regularization term. The policy loss is expressed as the probability ratio of comparing the current policy and the old policy. The value loss is expressed as the difference between the current value estimate and the target value. The entropy regularization term is used to encourage the exploratory nature of the policy.

[0011] Preferably, the strategy loss is expressed as:

[0012]

[0013] Among them, π θ (a|s) is the probability of action a under the current strategy, is the probability under the old strategy, Adv(s,a) is the advantage function, clip() represents the clipping process, and ∈ is the set clipping range.

[0014] Preferably, the value loss is expressed as:

[0015]

[0016] Among them, V t is the state value function V(s t ), represents the true value of the cumulative return, ∈ is the clipping range; the first term Represents the current value function V(s t ) and actual returns The standard square error between is a clipped error, limiting V t In V t -∈ and V t +∈ between.

[0017] Preferably, the entropy regularization term is expressed as:

[0018]

[0019] Among them, Entropy(p(a|s)) is the entropy of the strategy, for a given state s t The strategy π(a|s t ), Entropy(p(a|s))=-∑ a π(a|s t )logπ(a|s t ), where π(a|s t ) indicates that the strategy π is in state s t The probability of choosing action a; It means to find the expectation of state s according to the state distribution p(s), that is, to take the weighted average of the entropy of all possible states.

[0020] Preferably, the loss function is expressed as:

[0021] loss=(L_p-α·L_v+β·entropy_bonus)

[0022] Among them, α is the weight coefficient of value loss, and β is the coefficient of entropy regularization term.

[0023] The present invention has the following beneficial effects:

[0024] 1. By modeling segmented memory and long-range dependencies in the Gated Transformer-XL framework, the intelligent agent has more advantages in processing long-term series data and enhances its ability to capture long-term dependencies.

[0025] 2. Intelligent agents can better adapt to the dynamically changing battlefield environment and improve the applicability and robustness of strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is the basic framework diagram for reinforcement learning;

[0027] Figure 2 is a schematic diagram of aerodynamic angle and axis;

[0028] Figure 3 It is the flow chart of PPO algorithm;

[0029] Figure 4 Transformer-XL framework with L-layer encoder stack;

[0030] Figure 5 This is the PPO-GTrXL framework structure diagram. DETAILED DESCRIPTION

[0031] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0032] 1. Basic components of reinforcement learning

[0033] The basic framework of reinforcement learning is the interaction between the agent and the environment. The agent perceives the state of the environment and selects actions based on a certain strategy to influence the environment. The environment receives the agent's actions and changes its state, while returning the state and reward to the agent. The agent adjusts and optimizes its strategy based on the rewards obtained to maximize the long-term cumulative rewards. In this process, the agent's decision depends on the currently observed state and its defined strategy, and the choice of action directly affects the interaction between the agent and the environment.

[0034] 1. Construction of the intelligent agent physical model

[0035] There are three important parts in the agent model:

[0036] State: The state is a description of the environment at a certain moment. It can include all or part of the information of the environment (i.e., part of the observable state). In practical applications, the state can be the position of a robot, the current layout of a Go game, etc.

[0037] In the present invention, we assume that the aircraft can observe the state information of the missile through radar, etc. The state components of the aircraft include the three-dimensional coordinates of the aircraft, the yaw and pitch angles of the aircraft, the three-dimensional coordinates of the missile, and the yaw and pitch angles of the missile.

[0038] Action: An action is an operation that an agent can choose in a certain state. The action determines how the agent moves from the current state to the next state. For example, a robot can choose to move forward, turn, etc.; a Go player can choose to place a piece at a certain position on the board.

[0039] In the present invention, the actions of the aircraft include steady flight, left turn, right turn, upward pull-up, and downward dive.

[0040] Reward: A reward is a feedback signal that an agent receives after performing an action in a certain state. Reward is a numerical value that measures the quality of an action. The goal of the reward is to guide the agent to take actions that maximize the cumulative reward. For example, a robot may receive a positive reward for reaching its destination, while encountering an obstacle may receive a negative reward.

[0041] Among them, reward design, action design, and aircraft physics modeling require detailed explanation.

[0042] 1) Reward design

[0043] In the present invention, rewards include: result rewards, process rewards and height rewards.

[0044] Result reward: Since the air combat environment is a drone-missile pursuit environment, it is set in one round. If the missile still fails to track the aircraft within 100 time steps, the aircraft is judged to have successfully escaped and can get the reward in the result reward function; if the missile catches up with the drone within 100 time steps, the drone is judged to have failed to escape and will be punished in the result reward function.

[0045] r = -5000, if succeed

[0046] r = -5000.else

[0047] Process reward: In the process of reinforcement learning training, if there is only result reward, then the reward is undoubtedly sparse, which makes it difficult to conduct effective training. Therefore, we add process reward and define the process reward as follows

[0048] r=-0.01*(5000-distance),ifdistance<5000

[0049] r=-0.005*(10000-distance),if 5000 <distance<10000

[0050] r=0.else

[0051] High reward: In the process of reinforcement learning training, if there is only result reward, then the reward is undoubtedly sparse, which makes it difficult to conduct effective training. Therefore, we add process reward and define the process reward as follows:

[0052] r=0,if0 <h<20000

[0053] r=-100,ifh<0or h>20000

[0054] 2) Action design

[0055] In order to cover the basic techniques commonly used by pilots in actual air combat and adapt to various combat situations and tactical requirements, the UAV maneuvering action is designed by using the preset waypoint guidance method. The UAV uses PD control to change its own state components, thereby reducing the coordinate difference with the waypoint. By flexibly setting the waypoint, complex maneuvers can be achieved. A total of five maneuvers are designed, namely steady flight, left turn, right turn, pull up and dive down. Selecting these basic actions can effectively simplify the complexity of the action space. Compared with the continuous action space, the discrete action space makes the agent more controllable and explainable when learning and making decisions. Each action has a clear meaning and effect, which helps the agent quickly learn the best action selection in different situations.

[0056] 3) Aircraft physical modeling

[0057] Figure 1 The relative wind direction of the aircraft and three right-handed coordinate systems (forward, right, and down) with their origin at the center of mass of the aircraft are shown, along with the aerodynamic angles α and β. The x-axis of the body axis system is parallel to the fuselage reference axis (the axis used in the figure), and the z-axis lies in the plane of symmetry of the (conventional) aircraft. Let α be the fuselage angle of attack, measured as the angle from the projection of the relative wind direction onto the xz plane of the body axis system to the fuselage reference axis, with positive values ​​for relative wind direction coming from below the aircraft. The sideslip angle is measured from this projection line to the relative wind direction vector, with positive values ​​for relative wind direction coming from the right side of the aircraft.

[0058] The aircraft changes its flight state mainly through four inputs, namely throttle, elevator, aileron, and rudder. Among them, the throttle is measured in pounds, and the other three control quantities are measured in degrees.

[0059] The throttle acts on the x-axis of the body axis, and positive throttle will increase the acceleration along the x-axis. Ailerons are usually located on the wingtips of the aircraft and are used to control the rotation of the aircraft around its longitudinal axis (pitch axis). When one aileron is tilted downward and the other is tilted upward, a rolling moment is generated, causing the aircraft to rotate around the longitudinal axis, thereby changing the aircraft's roll angle. The elevator is located at the tail of the aircraft and is used to control the rotation of the aircraft around its lateral axis (roll axis). When the elevator is tilted upward, a lift force is generated, causing the aircraft to rise. When the elevator is tilted downward, a descending force is generated, causing the aircraft to descend. By adjusting the position of the elevator, the aircraft's elevation or pitch angle can be changed. The rudder is usually located on the vertical tail of the aircraft and is used to control the rotation of the aircraft around its vertical axis (yaw axis). When the rudder is turned left or right, a yaw moment is generated, causing the aircraft to rotate to change its direction. The rudder is mainly used to maintain balance in horizontal flight and assist the aircraft in changing its flight direction. The maximum value and unit of each control surface are shown in Table 1.

[0060] Table 1 Maximum values ​​and units of each control surface

[0061]

[0062] For the longitudinal component and the lateral component, it is generally chosen to model them separately. The following formula shows the state space equation corresponding to the longitudinal component, with the input being the throttle offset δ t and elevator deflection δ e :

[0063]

[0064] The following is the state space equation of the lateral component, with the input being the throttle offset δ t , aileron offset δ a and rudder deflection δ r .

[0065]

[0066] Table 3-2 shows the specific names of the aircraft state components after removing the input offset.

[0067] Table 3-2 Aircraft status components

[0068]

[0069] After obtaining the state space equation of the aircraft, the derivative of the state component can be obtained. Then the fourth-order Runge-Kutta method is used to solve the differential equation to obtain the state component of the next time step, thereby realizing the state update.

[0070] 2. External environment construction

[0071] The external environment of this experiment is a three-dimensional air combat environment. A missile model is built to simulate an actual opponent chasing an aircraft, and to train the aircraft's ability to escape from the pursuit of missiles with different thrusts.

[0072] In the inertial coordinate system, the missile kinematic equation is:

[0073]

[0074] In the formula, (x, y, z) is the coordinate of the missile in the inertial coordinate system, (v, θ, φ) is the missile's speed, track pitch angle and track yaw angle, all of which are functions of the flight time t. According to the missile's flight, it is divided into active and passive segments. In the inertial system, the forces on the active segment of the missile mainly include: thrust T(t), gravity G=m(t)g, aerodynamic drag D(t) (in the non-inertial system, the surface visual acuity C also needs to be considered). Therefore, in the ballistic coordinate system, the missile's particle dynamics equation is:

[0075]

[0076] in, is the speed direction overload, n y ,n z is the lateral control overload of the missile in the yaw and pitch directions, calculated by the proportional guidance method; m(t) is the current mass of the missile, g is the gravitational acceleration constant, and g=9.81m / s 2 .

[0077] T(t) is the missile thrust, and its magnitude satisfies:

[0078]

[0079] In the formula, I sp is the specific impulse (assumed to be a constant), is the fuel mass combustion rate.

[0080] D(t) is the aerodynamic resistance, and its magnitude satisfies:

[0081]

[0082] Where v(t) is the velocity of the missile at time t, z(t) is the horizontal height of the missile at time t, S is the measuring area (defined as the cross-sectional area of ​​the missile orthogonal to the velocity), c D (v) is the drag coefficient, which is a function of the velocity v and can be approximately considered a constant. D (v)≡0.1. ρ(z) is the air density, which is a function of the height z and satisfies the approximate relationship Where ρ0 = 1.225 kg / m 3 ,k=9300m.

[0083] For the calculation area S of the missile, its size approximately satisfies:

[0084]

[0085] Where d is the missile diameter, L is the missile radius, Δθ, Δφ are the differences between the missile attitude and track in pitch angle and yaw angle, which can be approximately considered as the change of track pitch angle and yaw angle θ, φ in Δt time, that is,

[0086] The missile guidance adopts the proportional guidance law. Assuming that the guidance coefficient in the two mutually perpendicular control planes is K=3, the two lateral control overloads in the yaw and pitch directions are defined as:

[0087]

[0088] In the formula, β and ε are the sight deflection angle and sight inclination angle respectively. are the derivatives of the sight deflection angle and sight inclination angle with time, respectively.

[0089]

[0090] The sight vector is the distance vector Yes x =x t -x m ,r y =y t -y m ,r z =z t -z m .x t ,y t ,z t is the target machine position, and the modulus value is defined as The line of sight deflection angle and line of sight inclination angle and their derivatives with time are defined as:

[0091]

[0092] 2. PPO Algorithm and Transformer-XL Framework

[0093] 1. PPO algorithm

[0094] After completing the construction of the agent's physical model, it is necessary to embed a reinforcement learning algorithm to enable it to interact with the environment and complete the iteration and improvement of the air combat strategy. Proximal Policy Optimization (PPO) is an advanced policy optimization algorithm that maximizes the cumulative reward by adjusting the strategy when training the agent.

[0095] Although PPO ensures the stability and efficiency of training by limiting the amplitude of policy updates, its essence is still based on the "Actor-Critic" framework. It successfully balances exploration and exploitation by introducing a stable policy update method in the "Actor-Critic" framework, avoiding the instability that may occur during the policy update process. It combines the advantages of the Actor-Critic method and achieves efficient and stable training in complex reinforcement learning tasks by limiting the amplitude of policy updates.

[0096] The air combat decision-making training process of the PPO algorithm based on the "Actor-Critic" framework is as follows:

[0097] Initialize the F_16env environment class, agent, and experience replay pool buffer;

[0098] Initialize the strategy network parameters θ and value network parameters φ;

[0099]

[0100]

[0101] Draw reward and win rate curves, draw action distribution pie charts, and visualize single successful rounds

[0102] The PPO algorithm flow chart is as follows Figure 3 shown.

[0103] The input of the value network and the policy network are both states, and the outputs are value (scalar) and the probability distribution of action (vector). The value network uses the state of the reward when calculating the loss function.<r,s’> .

[0104] The agent interacts with the combat environment and converts the quadruple of each step in a round<s,a,r,s’> The experience is stored in the combat experience pool (memory pool). Among them, the reward r and the next state s i+1 It is passed as input to the value network (the reward r refers to the reward of all steps in this round, and the same applies to the state). The loss function of the value network is N represents the number of samples in a batch (i.e., the number of samples taken from the experience replay pool). Usually, a batch of samples is used to calculate the loss during training, so the loss of each sample needs to be averaged. i Indicates taking action a at time step i i The immediate reward obtained after the action. This is the actual feedback obtained from the environment, indicating the direct benefit brought by the current action. γ is the discount factor, which is used to balance the weights of current rewards and future rewards. γ∈[0,1], the larger the value, the greater the impact of future rewards on current decisions. max a Q ω -(s i+1 ,a) refers to the state s i+1 In the example, the maximum Q value selected based on the target network (maximizes the expected future return). ω - is usually a delayed update Q network to stabilize training. This term reflects the time from the next step s i+1 Starting from state Q, the maximum future reward of the optimal action. ω (s i ,a i ) is the current Q network (i.e. the Q network being updated) in state s i Next action a i It indicates that under the current policy, the state s i and action a i The expected cumulative return. ω and ω - represents the parameters of the neural network. ω is the parameter of the current Q network, and ω - are the parameters of the target Q network. The parameters of the target network ω - It is usually updated synchronously from ω at regularly spaced time steps.

[0105] In order to minimize the loss function, the value network performs gradient descent to update its own parameters ω, and its gradient is At the same time, the value network calculates the advantage function A using the generalized advantage estimation (GAE) method.

[0106] Let me briefly introduce the GAE approach. First, use δ t =r t +γV(s t+1 )-V(s t ) represents the temporal difference error, where V is a learned state value function. According to the idea of ​​multi-step temporal difference, we have:

[0107]

[0108] GAE then takes an exponentially weighted average of the advantage estimates for these different steps:

[0109]

[0110] Among them, λ∈[0,1] is an additional hyperparameter introduced in GAE. When λ=0, That is, the advantage of only looking at one step of difference; when λ=1, The complete average of the advantages is obtained by looking at the difference at each step.

[0111] Agent state s i The input is sent to the policy network, which is a neural network whose input dimension is the state dimension and output dimension is the number of actions. The output is the probability distribution of the action, and the agent randomly selects the action according to the probability. Then, the policy network calculates the action ratio, which is the new policy. and the old strategy The ratio between is used to measure the change in the probability of selecting an action a in a certain state s:

[0112]

[0113] in, is the probability that the current strategy (new strategy) chooses action a in state s, is the probability that the old policy chooses action a in state s.

[0114] After obtaining the advantage function A and the action ratio, the parameters of the policy network can be updated according to the objective function of the policy network. The objective function of the policy network is The purpose of parameter updating is to maximize the objective function. clip(x,l,r):=max(min(x,r),l), that is, limit x to [l,r]. In the above formula, ∈ is a hyperparameter, indicating the range of clipping.

[0115] if This means that the value of this action is higher than the average, and maximizing this formula will increase But it will not exceed 1+∈. On the contrary, if Maximizing this formula will reduce But it will not exceed 1-∈. This ensures that the gap between the new parameters and the old parameters is not too large.

[0116] 2. Transformer-XL Framework

[0117] The Transformer model is a model for processing serialized data. In recent years, it has achieved remarkable results in image processing, semantic understanding, and other aspects, and has shown excellent performance in DRL (Deep Reinforce Learning). As a powerful alternative to RNN (Recurrent Neural Network), thanks to the attention mechanism, the Transformer model can process the entire input sequence at the same time and determine the importance of each part. This mechanism not only supports efficient parallel processing, but also helps the model capture long-term dependencies and relationships in the data.

[0118] In sequence-to-sequence tasks, the Transformer model uses both encoders and decoders, but DRL tasks are usually considered sequence-to-one tasks, where the input is a sequence state vector and the output is a perceptual representation of the sequence, usually a numerical vector of fixed dimension after embedding. Therefore, generally only the encoder in the Transformer is selected to solve the DRL task. An encoder module consists of multi-head attention (MHA), positional encoding, fully connected layer (MLP), layer normalization, and residual connections.

[0119] The Transformer-XL (Extra Long) framework is based on the traditional Transformer architecture and adopts a segment-level regression mechanism, which enables the Transformer model to interact with inputs that exceed the sequence length, making it easier to capture long-term dependencies in the input context.

[0120] The Transformer architecture does not have an intrinsic temporal structure like RNN, and a way is needed to convey the position information of each element in the sequence. The introduction of position encoding can effectively solve the above problems. In sequence data, the order and position of each element are very important. For example, in natural language processing, the order of words in a sentence affects the meaning of the sentence. The Transformer model processes data in parallel, so the position information needs to be explicitly injected into the input so that the model can use this position information to better understand and process the sequence. There are two main types of position encoding: absolute position encoding and relative position encoding.

[0121] The fully connected layer, also known as the dense layer or affine layer, is one of the most basic and common layers in a neural network. Each input node is connected to each node in the output node. In a fully connected layer, the input vector x is linearly transformed by a weight matrix W and a bias vector b, and the output result is processed by a nonlinear activation function (such as ReLU, Sigmoid, etc.) to obtain the output vector y. Its mathematical expression is:

[0122] y=f(Wx+b)

[0123] Layer normalization is a technique used to speed up and stabilize the training of neural networks, especially recurrent neural networks (RNNs) and feedforward neural networks.

[0124] Residual connection, also known as skip connection, is a deep neural network structure introduced by ResNet (Residual Network). It is mainly used to solve the problems of gradient vanishing and gradient exploding in deep neural networks, thereby enabling the training of deep neural networks.

[0125] Residual links form shortcuts by directly skipping the input to the output of some subsequent layers. Specifically, assuming that the input of a layer is x, the output obtained through a conventional neural network layer (such as a convolutional layer, a fully connected layer, etc.) is Then the output after adding the residual link is

[0126]

[0127] in, Represents the result of transforming the input x. x is the input, which is directly added to the transformed result.

[0128] Figure 4 The complete Transformer-XL architecture is shown, which consists of multiple encoder blocks stacked together. Figure 4 (a) shown.

[0129] 3. Combination structure based on PPO and Transformer-XL framework

[0130] The basis of the combined structure based on PPO and Transformer-XL framework is still Actor-Critic. Figure 5 Schematic diagram of the PPO-GTrXL framework structure.

[0131] The state vector of the aircraft is first passed as input to the multilayer perceptron, whose purpose is to transform the input features into the embedding dimensions required by the Transformer in the model. It is a mapping from the input feature space to the feature space required by the Transformer model. The features output by the multilayer perceptron are then fed into the stacked encoder block of the Transformer, such as Figure 4 (b) The output of the last Transformer encoder block represents the contextual information encoded by the entire input state vector, which contains a deep understanding of the environment state. This output, as the final representation result, captures long-term dependencies and attention weights. This information can be used to make decisions about the current environment state.

[0132] This part of the output can be used as the input of the policy network to generate the action distribution taken by the agent in the current state.

[0133] On the other hand, the final output can also be passed to the value network through another branch to estimate the value of the current state.

[0134] The loss function of the PPO-Transformer framework consists of three parts: policy loss, value loss, and entropy regularization. Policy loss is used to optimize the policy so that the actions taken in a given state have a higher expected return. The policy loss function is calculated by comparing the probability ratio of the current policy and the old policy. This can be expressed by the following formula:

[0135]

[0136] Among them, π θ (a|s) is the probability of action a under the current strategy, is the probability under the old strategy, Adv(s,a) is the advantage function, and ∈ is the clipping range.

[0137] The value loss is optimized by calculating the difference between the current value estimate and the target value:

[0138]

[0139] Among them, V t is the state value function V(s t ), which means that from state s t Initially, the expected value of the cumulative return that can be obtained when executing a certain strategy. It represents the true value of the cumulative return obtained by sampling starting from time step t, and ∈ is the clipping range. Represents the current value function V(s t ) and actual returns If the difference between the estimated value and the actual value of the strategy in this state is large, the loss will increase. is a clipped error, ensuring that the update of the value function does not deviate too much. By clipping, V is limited t In V t -∈ and V t +∈. When calculating the loss, the objective function adopts a more conservative strategy and chooses the larger value of the square error and the clipping error. This is to ensure that the value function does not change too quickly even during the policy optimization process, ensuring the stability of the update.

[0140] The entropy regularization term is used to encourage the exploration of the strategy:

[0141]

[0142] Among them, Entropy(p(a|s)) is the entropy of the strategy, and β is the coefficient of the entropy regularization term. For a given state s t The strategy π(a|s t ), Entropy(p(a|s))=-∑ a π(a|s t )logπ(a|s t ). Among them, π(a|s t ) indicates that the strategy π is in state s t Entropy(p(a|s)) measures the probability of selecting action a in state s. t The randomness or uncertainty of the distribution of different action choices by the policy under the condition of . The higher the entropy of the policy, the more random the agent is in choosing between multiple actions, without a strong preference for a particular action. In other words, the policy will explore multiple possible actions in a given state. The lower the entropy of the policy, the more the agent has a clear preference for a certain action, that is, the policy becomes more certain, more inclined to choose a certain action, and less willing to try other actions.

[0143] The comprehensive loss combines policy loss, value loss and entropy regularization term.

[0144] loss=(L_p-α·L_v+β·entropy_bonus)

[0145] Among them, α is the weight coefficient of value loss, and β is the coefficient of entropy regularization term.

[0146] During the training process, the parameters of the policy network and the value network are updated by optimizing the total loss function. This approach ensures a balance between the optimization of the policy, the accuracy of the value function, and the exploratory nature of the policy.

[0147] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for training air combat strategy, characterized in that: include: The state vector of the aircraft is first passed into the multi-layer perceptron as input, adjusted to the embedding dimension required by the Transformer encoder block, and then input into multiple stacked Transformer encoder blocks to obtain the context information encoded by the input state vector as the state representation result; The state representation results are input into the value network and policy network under the PPO framework respectively. The value network calculates the advantage function Adv(s,a); the policy network outputs the probability distribution of the action, and the agent randomly selects the action according to the probability distribution; then, the policy network calculates the action ratio, that is, the ratio between the new strategy and the old strategy, to measure the change in the probability of selecting a certain action a under a certain state s; A loss function is calculated based on the advantage function Adv(s,a) and the action ratio. The value network and the policy network use the same loss function to train and update their respective network parameters. The loss function is a weighted sum of the policy loss, the value loss and the entropy regularization term. The policy loss is expressed as the probability ratio of comparing the current policy and the old policy. The value loss is expressed as the difference between the current value estimate and the target value. The entropy regularization term is used to encourage the exploratory nature of the policy.

2. The air combat strategy training method as claimed in claim 1, characterized in that: The strategy loss is expressed as: Among them, π θ (a|s) is the probability of action a under the current strategy, is the probability under the old strategy, Adv(s,a) is the advantage function, clip() represents the clipping process, and ∈ is the set clipping range.

3. The air combat strategy training method as claimed in claim 2, characterized in that: The loss in value is expressed as: Among them, V t is the state value function V(s t ), represents the true value of the cumulative return, ∈ is the clipping range; the first term Represents the current value function V(s t ) and actual returns The standard square error between is a clipped error, limiting V t In V t -∈ and V t +∈ between.

4. The air combat strategy training method as claimed in claim 3, characterized in that: The entropy regularization term is expressed as: Among them, Entropy(p(a|s)) is the entropy of the strategy, for a given state s t The strategy π(a|s t ), Entropy(p(a|s))=-∑ a π(a|s t )logπ(a|s t ), where π(a|s t ) indicates that the strategy π is in state s t The probability of choosing action a; It means to find the expectation of state s according to the state distribution p(s), that is, to take the weighted average of the entropy of all possible states.

5. The air combat strategy training method as claimed in claim 4, characterized in that: The loss function is expressed as: loss=(L_p-α·L_v+β·entropy_bonus) Among them, α is the weight coefficient of value loss, and β is the coefficient of entropy regularization term.

Images