An unmanned aerial vehicle air combat maneuver decision-making method based on proximal policy optimization and gated recurrent unit

Abstract

The application relates to the technical field of artificial intelligence, and provides a UAV air combat action decision-making method based on proximal policy optimization and a gated recurrent unit, which comprises the following steps: a three-dimensional UAV air combat simulation environment is constructed; an Actor-Critic neural network architecture based on a gated recurrent unit (GRU) is constructed; and the neural network is iteratively trained through a large amount of self-contradiction and learning. The scheme of the application enhances the time sequence feature sensing capability; through the introduction of the GRU network, the agent can better capture and understand the time sequence dependence relationship of the flight state in the air combat environment, for example, inferring the target intention or predicting the future situation from the continuous state change, so that a more forward-looking decision can be made. The UAV air combat environment has high dynamics and uncertainty. Through the online interaction and learning mechanism of reinforcement learning and the processing capability of the GRU for dynamic sequences, the method can enable the UAV to gradually adapt to the complex environment and learn an effective countermeasure.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, particularly the application of reinforcement learning in the autonomous control of unmanned aerial vehicles (UAVs). More specifically, this invention relates to a UAV air combat autonomous action decision-making method that combines the Proximal Policy Optimization (PPO) algorithm and the Gated Recurrent Unit (GRU), aiming to improve the autonomous decision-making capability and combat effectiveness of UAVs in complex and dynamic air combat environments. Background Technology

[0002] With the rapid development of UAV technology, its application in the military field is becoming increasingly widespread, and autonomous UAV air combat has become an important research direction. How to enable UAVs to autonomously make maneuvering decisions based on the battlefield situation in a rapidly changing air combat environment to gain an advantage, avoid threats, and ultimately achieve combat objectives is one of the core challenges in UAV intelligence research. Reinforcement learning (RL), as a machine learning method that learns optimal policies through interaction with the environment, has achieved significant success in fields such as game AI and robot control, and is considered an effective way to achieve autonomous UAV decision-making. Among them, the Proximal Policy Optimization (PPO) algorithm has become one of the mainstream algorithms in the field of deep reinforcement learning due to its good stability and data utilization efficiency. However, traditional PPO algorithms still face some challenges when applied to tasks such as UAV air combat, which have complex state spaces, continuous action spaces, and significant temporal dependencies: Temporal feature extraction: UAV air combat is a dynamic process; current decisions not only depend on the current instantaneous state but are also closely related to the historical state sequence. Standard feedforward neural networks (FNNs) struggle to effectively capture this temporal dependency. Strategy Stability and Exploration Efficiency: Although PPO improves training stability through mechanisms such as pruning the objective function, balancing exploration and utilization to efficiently learn robust maneuver strategies in high-dimensional and complex air combat environments remains a challenge. Environment Modeling and Reward Design: Accurate air combat environment modeling and reasonable reward function design are crucial for reinforcement learning performance, requiring careful consideration of various elements and tactical objectives in air combat. Recurrent Neural Networks (RNNs) and their variants, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs), can effectively process sequential data and capture dependencies in time series. Introducing RNN structures into deep reinforcement learning algorithms is expected to improve the decision-making performance of agents in dynamic environments. GRUs, as an effective variant of LSTMs, have fewer parameters and similar performance, offering greater efficiency advantages in practical applications. Therefore, researching how to combine the advantages of recurrent neural networks such as GRUs with the PPO algorithm and apply it to UAV air combat maneuver decision-making to improve their understanding of dynamic battlefield situations and the intelligence of maneuver strategies has significant theoretical and practical value. Summary of the Invention

[0003] To address the shortcomings of traditional reinforcement learning algorithms in UAV air combat maneuver decision-making, such as insufficient extraction of temporal features, and the need to improve decision-making efficiency and stability, this invention proposes a UAV air combat maneuver decision-making method based on Proximal Policy Optimization (PPO) and Gated Recurrent Units (GRUs). Its main objectives include: enhancing the UAV's ability to understand and utilize temporal state information in the air combat environment; improving the learning efficiency and policy robustness of the PPO algorithm in complex and dynamic air combat environments; and achieving efficient autonomous maneuver decision-making for UAVs in three-dimensional air combat scenarios, thereby improving air combat effectiveness.

[0004] This invention provides a method for UAV air combat action decision-making based on near-end strategy optimization and gated cyclic units, comprising the following steps:

[0005] Step 1: Construct a 3D UAV air combat simulation environment. Construct a 3D adversarial environment containing both friendly and enemy UAVs. This environment operates based on UAV dynamics models and preset air combat rules. It is responsible for receiving action commands from friendly UAVs to update the status of all UAVs on the battlefield and outputting real-time battlefield situation information as the status observations of friendly agents.

[0006] Step 2: Construct a GRU-based Actor-Critic neural network architecture. Next, design a neural network architecture comprising an Actor (policy) network and a Critic (value) network. The core feature of this architecture is that both the Actor and Critic networks incorporate at least one Gated Recurrent Unit (GRU) layer to process the temporal state observations of the input. The Actor network is responsible for outputting the probability distribution parameters that define the UAV's maneuvers, while the Critic network is responsible for evaluating the value of the current state.

[0007] Step 3: Iterative training based on the PPO algorithm. Finally, the neural network is trained through extensive interaction between our UAV and the simulation environment. In this step, the agent executes actions according to the policy generated by the Actor network and collects experience data from the environment. Once sufficient data has been collected, the Proximal Policy Optimization (PPO) algorithm is used to calculate the advantage function using the value estimate output by the Critic network, and combined with a unique pruning objective function, to stably and efficiently update the parameters of the Actor and Critic networks until the network model converges.

[0008] Furthermore, the state observation values ​​include at least the absolute coordinates (x, y, z) and flight parameters (velocity V, flight path angle γ, heading angle ψ) of our UAV, and the absolute coordinates (x, y, z) and flight parameters (velocity V, flight path angle γ, heading angle ψ) of the enemy UAV.

[0009] Furthermore, updating the parameters of the Actor network and the Critic network in step S3 specifically includes:

[0010] (a) During the interaction between the UAV and the environment, collect trajectory data including status, actions, rewards, next status and end-of-round markers, and store it in the experience replay buffer;

[0011] (b) When the amount of data in the experience replay buffer reaches a preset threshold, calculate the dominance function and the PPO pruning objective function;

[0012] (c) Based on the calculated advantage function and the PPO pruning objective function, update and optimize the parameters of the Actor network and the Critic network.

[0013] Furthermore, the GRU layers in the Actor network and Critic network are configured to receive state observations at a single time step as an input sequence with a sequence length of 1, and process them independently at each time step.

[0014] Furthermore, the state observations are normalized before being input into the Actor network and Critic network, mapping the original state values ​​to a preset numerical range.

[0015] Furthermore, the action distribution parameters output by the Actor network are the mean and standard deviation of a Gaussian distribution, and the actions are obtained by sampling this Gaussian distribution and cropping it to an effective control range.

[0016] Furthermore, the design of the reward function in the three-dimensional air combat simulation environment considers at least one or more of the following factors:

[0017] a. The relative situation between our UAV and the enemy UAV, including distance, target line-of-sight angle (ATA), antenna training angle (AA), and altitude difference;

[0018] b. Whether our drones have met the preset conditions for a superior attack;

[0019] c. Whether the enemy drone has met the preset conditions for a superior attack;

[0020] d. Did the drone collide or fly out of bounds?

[0021] e. Has the round timed out?

[0022] Furthermore, during the parameter update process of the PPO algorithm, the calculation of the advantage function adopts Generalized Advantage Estimation.

[0023] Furthermore, in step (c) of optimizing network parameters, the loss function of the Actor network includes a PPO pruning objective function term and an entropy reward term, while the loss function of the Critic network is the mean squared error of the state value estimation.

[0024] Furthermore, the three-dimensional air combat simulation environment module can simulate air combat scenarios that include at least our own UAVs and enemy UAVs, and perform state transitions and reward calculations according to preset air combat rules and UAV dynamics models.

[0025] The UAV air combat maneuver decision-making method based on PPO and GRU proposed in this invention has the following beneficial effects:

[0026] Enhanced Temporal Feature Perception: By introducing a GRU network, the agent can better capture and understand the temporal dependencies of flight states in an air combat environment. For example, it can infer target intentions or predict future situations from continuous state changes, thereby making more forward-looking decisions. Adaptation to Complex Dynamic Environments: The UAV air combat environment is highly dynamic and uncertain. The method of this invention, through the online interaction and learning mechanism of reinforcement learning, and the GRU's ability to process dynamic sequences, enables the UAV to gradually adapt to this complex environment and learn effective countermeasures. Attached Figure Description

[0027] Figure 1 Here is a flowchart of the proposed training algorithm;

[0028] Figure 2 It is the training reward of the standard PPO algorithm;

[0029] Figure 3 It is an improved PPO algorithm training reward. Detailed Implementation

[0030] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0031] refer to Figure 1 The UAV air combat maneuver decision-making method based on near-end strategy optimization and gated loop unit proposed in this invention includes the following core modules and processes in a specific embodiment, which strictly correspond to the steps defined in claim 1 of this invention.

[0032] Step S1: 3D UAV Air Combat Simulation Environment

[0033] This embodiment first constructs a detailed three-dimensional UAV air combat simulation environment. This environment, based on the UAV's flight mechanics model, can accurately simulate its state parameters, including position (x, y, z), velocity (V), flight path angle (γ), and heading angle (ψ). Simultaneously, the environment defines the UAV's controllable inputs, i.e., the action space, including tangential overload (Nx), normal overload (Nz), and roll angle (ψ). The environment defines clear air combat boundaries, flight altitude and speed limits, and control quantity constraints. This environment supports online combat between at least one "friendly" drone controlled by the method of this invention and an "enemy" drone controlled by a preset maneuver strategy (such as programmed evasion or attack mode). At each time step, the environment receives action commands from the friendly drone and calculates the state of all drones at the next moment and the corresponding reward value based on the dynamics model and air combat rules.

[0034] Step S2: GRU-based Actor-Critic Network Architecture

[0035] This embodiment adopts the Actor-Critic reinforcement learning framework, the core of which is that both the Actor (policy) network and the Critic (value) network integrate GRU layers to effectively process the temporal information of the air combat status.

[0036] First, the raw state observations obtained from the simulation environment (including multi-dimensional flight parameters of both our side and the enemy) are spliced ​​and normalized according to the predefined observation space boundary before being input into the network, and mapped to the standard interval [-1, 1] to facilitate network training.

[0037] Both the Actor network and the Critic network contain at least one GRU layer. These GRU layers are configured to process input data at a single time step (i.e., the input sequence length is 1). Through their internal update and reset gate mechanisms, the GRU units can effectively capture the temporal dependencies in the state sequence. Even in single-step input mode, the updates of their internal hidden states implicitly involve the memorization and processing of historical information.

[0038] At the network output, the final output layer of the Actor network produces a Gaussian distribution with mean and standard deviation. The mean vector is constrained to the range [-1, 1] by the Tanh activation function, corresponding to the normalized action space; the standard deviation vector is ensured to be positive by the Softplus activation function. The Critic network then outputs a scalar value, which is the value estimate V(s) of the input state.

[0039] Step S3: Iterative training process based on the PPO algorithm

[0040] After the 3D UAV air combat simulation environment and the GRU-based Actor-Critic network were constructed, iterative training began. For details, please refer to [link / reference needed]. Figure 1 And detailed as follows:

[0041] S31 Initialization and Interaction Data Collection: At the start of training, the total number of rounds is initialized. Before the start of each round, the simulation environment is reset to obtain the initial state. During a round, at each time step t, the agent samples the action at based on the policy π(at|st) output by the current Actor network and interacts with the environment. After the environment executes the action, it returns the reward rt and the next state st+1. These interaction data tuples (st, at, rt, st+1, V(st), logπ_old(at|st), done_flag) are stored in the experience replay buffer. Here, V(st) and logπ_old(at|st) are the current Critic network's value estimate of the state and the log probability of the old policy, respectively.

[0042] S32 Advantage Function and Target Return Calculation: Once a sufficient amount of interaction data (e.g., a batch size) has accumulated in the experience replay buffer, the advantage function is calculated using the value estimate V(s) provided by the Critic network. This advantage function quantifies how much better performing a specific action in a given state is compared to the average performance of following the current policy. A positive advantage value indicates that the action is better than the average and can bring higher long-term returns, while a negative value indicates it is worse than the average. The result of this advantage function calculation directly guides the update direction of the Actor network. This embodiment uses Generalized Advantage Estimation (GAE) to calculate the advantage value At at each step, a method that effectively balances bias and variance.

[0043]

[0044]

[0045] in, It is a discount factor. This is the smoothing parameter for GAE. The target return is... The calculated advantage function is standardized (subtracted from the mean and divided by the standard deviation) to stabilize the training.

[0046] S33 Network Parameter Update: Based on the calculated advantage function and objective reward, the PPO algorithm is used to update the network parameters. The core of PPO lies in optimizing a pruned alternative objective function to update the Actor network, limiting the step size of each update to avoid drastic policy changes.

[0047]

[0048] in, , is the probability ratio of the new strategy to the old strategy. These are pruning hyperparameters. The total loss function of an Actor network typically also includes an entropy regularization term. To encourage the exploration of strategies:

[0049]

[0050] Loss function of Critic network Defined as the mean squared error (MSE) in state value estimation:

[0051]

[0052] The overall loss function is a weighted sum of the Actor and Critic losses. Using the collected batches of trajectory data, the parameters of the Actor and Critic networks are iteratively updated multiple times through an optimizer such as Adam. In each iteration, the data is divided into mini-batches for stochastic gradient descent.

[0053] S34 Loop and Termination: After training a batch of data, clear the buffer and continue the interactive-collection-training loop until the current round ends (e.g., the maximum number of steps is reached or the termination condition is triggered). Then reset the environment to enter the next round, until all preset training rounds are completed.

[0054] Experimental results:

[0055] like Figure 2 and Figure 3 As shown in the reward curves, a comparison of the two reward curves clearly demonstrates that both models converge to similar high reward levels in the later stages of training, proving the effectiveness of the PPO algorithm in solving this type of task. Furthermore, the GRU-based model proposed in this invention learns much faster than the baseline model. The GRU model's average reward rapidly climbs and stabilizes at a high level around the 1200th round, while the baseline model only reaches a similar performance level around the 2500th round. This indicates that introducing a GRU network enables the agent to learn effective maneuver strategies more efficiently from time-series data of air combat, improving the convergence speed.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for UAV air combat action decision-making based on proximal strategy optimization and gated cyclic units, characterized in that, Includes the following steps: S1: Construct a three-dimensional UAV air combat simulation environment, the environment including at least one friendly UAV controlled by an intelligent agent and at least one enemy UAV controlled by a preset strategy or another intelligent agent; the simulation environment is used to update the environmental state according to the received action instructions and output battlefield situation information as the state observation value of the intelligent agent. S2: Construct an Actor-Critic neural network architecture based on a gated recurrent unit (GRU), including an Actor network and a Critic network; wherein, the Critic network receives the current state observation, processes it through at least one GRU layer and at least one fully connected layer, and outputs a value estimate of the current state; the Actor network receives the current state observation, processes it through at least one GRU layer and at least one fully connected layer, and outputs action distribution parameters, wherein the actions are the control commands of the UAV, including at least normalized tangential overload, normal overload, and roll angle; S3: Our UAV randomly or deterministically selects actions based on the action distribution parameters output by the Actor network, and inputs the actions into the three-dimensional air combat simulation environment for execution, collecting environmental feedback; calculates the advantage function based on the value estimate output by the Critic network; and trains the neural network using the near-end strategy optimization PPO algorithm based on environmental feedback and the advantage function, updates the parameters of the Actor and Critic networks, and iteratively trains the neural network through adversarial and learning processes until the network model converges. Specifically, updating the parameters of the Actor network and the Critic network includes: (a) During the interaction between the UAV and the environment, collect trajectory data including status, actions, rewards, next status and end-of-round markers, and store it in the experience replay buffer; (b) When the amount of data in the experience replay buffer reaches a preset threshold, the dominance function and the PPO pruning objective function are calculated; the calculation result of the dominance function will directly guide the update direction of the Actor network; the generalized dominance estimation (GAE) is used to calculate the dominance value at each step. The details are as follows: ； ； in, It is a discount factor. It is the smoothing parameter of GAE; the target return is... The calculated advantage function will be standardized to stabilize the training. (c) Based on the calculated advantage function and the PPO pruning objective function, update and optimize the parameters of the Actor network and the Critic network; the loss function of the Actor network includes a PPO pruning objective function term and an entropy reward term, and the loss function of the Critic network is the mean squared error of the state value estimation; wherein the loss function of the Actor network is expressed as follows: ; in, , is the probability ratio of the new strategy to the old strategy. These are pruning hyperparameters; the total loss function of an Actor network typically also includes an entropy regularization term. To encourage the exploration of strategies: ; Loss function of Critic network The mean square error (MSE) of the state value estimation is defined as follows: 。 2. The method for UAV air combat action decision-making based on near-end strategy optimization and gated loop unit as described in claim 1, characterized in that, The state observation values ​​include at least the absolute coordinates (x, y, z) and flight parameters of our UAV, and the absolute coordinates (x, y, z) and flight parameters of the enemy UAV; the flight parameters of both our UAV and the enemy UAV include speed, flight path angle and heading angle.

3. The method for UAV air combat action decision-making based on near-end strategy optimization and gated loop unit as described in claim 1, characterized in that, The GRU layers in the Actor and Critic networks are configured to receive state observations at a single time step as an input sequence of length 1, and process them independently at each time step.

4. The method for UAV air combat action decision-making based on near-end strategy optimization and gated loop unit as described in claim 1, characterized in that, The state observations are normalized before being input into the Actor and Critic networks, mapping the original state values ​​to a preset numerical range.

5. The method for UAV air combat action decision-making based on near-end strategy optimization and gated cyclic unit as described in claim 1, characterized in that, The action distribution parameters output by the Actor network are the mean and standard deviation of a Gaussian distribution. The actions are obtained by sampling this Gaussian distribution and cropping it to an effective control range.

6. The method for UAV air combat action decision-making based on near-end strategy optimization and gated loop unit as described in claim 1, characterized in that, The design of the reward function in a three-dimensional air combat simulation environment takes into account at least one or more of the following factors: a. The relative situation between our UAV and the enemy UAV, including distance, target line-of-sight angle (ATA), antenna training angle (AA), and altitude difference; b. Whether our drones have met the preset conditions for a superior attack; c. Whether the enemy drone has met the preset conditions for a superior attack; d. Did the drone collide or fly out of bounds? e. Has the round timed out? 7. The method for UAV air combat action decision-making based on near-end strategy optimization and gated loop unit as described in claim 1, characterized in that, During the parameter update process of the PPO algorithm, the calculation of the advantage function adopts the generalized advantage estimation.

8. The method for UAV air combat action decision-making based on near-end strategy optimization and gated loop unit as described in claim 1, characterized in that, The three-dimensional air combat simulation environment can simulate air combat scenarios that include at least our own UAVs and enemy UAVs, and perform state transitions and reward calculations according to preset air combat rules and UAV dynamics models.

Images