Improved unmanned aerial vehicle path planning method based on deep reinforcement learning

Abstract

The invention relates to the technical field of unmanned aerial vehicle control, in particular to an unmanned aerial vehicle path planning method based on deep reinforcement learning. The method comprises the following steps: constructing an unmanned aerial vehicle-missile confrontation environment model for providing reward calculation, termination judgment and a visual interface so as to provide training data and evaluate the path planning performance of a reinforcement learning strategy; an Actor-Critic deep neural network model based on strategy gradient is constructed, the unmanned aerial vehicle state information is taken as input, the maneuvering acceleration is taken as output, and adaptive learning of the unmanned aerial vehicle maneuvering strategy is realized; in the reinforcement learning process, mechanisms such as dominant function normalization, generalized dominant estimation and gradient cutting are introduced, and the training stability and convergence speed are improved; by adopting a layered reward mechanism and an improved PPO algorithm, the method effectively solves the problems of difficulty in training convergence, poor strategy stability and low energy utilization rate of a traditional deep reinforcement learning method in a complex air combat environment, realizes autonomous decision making and optimal avoidance path planning of the unmanned aerial vehicle under a high confrontation condition, and improves the unmanned aerial vehicle experience. And the avoidance success rate and the training efficiency are obviously improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and more specifically to an improved UAV path planning method based on deep reinforcement learning. Background Technology

[0002] With the widespread application of unmanned aerial vehicles (UAVs) in modern warfare, they play a crucial role in reconnaissance, strike, and jamming missions. However, in environments posing a high threat from enemy missiles, the survivability of UAVs faces severe challenges. Existing UAV path planning and avoidance methods mainly include model analysis, expert system methods, artificial potential field methods, and game theory methods. The first two rely on empirical rules and are difficult to handle complex dynamic scenarios; while the artificial potential field method is computationally efficient, it is prone to getting trapped in local minima and has a single avoidance direction; although game theory methods can characterize offensive and defensive relationships, they involve large computational loads and have poor real-time performance.

[0003] In recent years, deep reinforcement learning (DRL) has demonstrated excellent self-learning and decision-making capabilities in the field of continuous control. However, its application to UAV missile evasion missions still has shortcomings: first, rewards are sparse and training is unstable in high-dynamic environments; second, traditional algorithms such as PPO are prone to convergence oscillations during policy updates; and third, attitude constraints and energy utilization are not fully considered, resulting in uneven trajectories and insufficient policy robustness. Summary of the Invention

[0004] Based on this, to address the problems of training convergence difficulties, poor strategy stability, and low energy utilization in path planning and missile evasion missions of UAVs in high-dynamic air combat environments in existing technologies, this invention proposes an improved UAV path planning method based on deep reinforcement learning. The method includes: constructing a UAV-missile confrontation environment model, which includes a state space, action space, reward function, and termination conditions, providing interfaces for training interaction, reward calculation, termination determination, and visualization; designing a hierarchical reward mechanism in the environment model, which includes four parts: distance reward, attitude reward, energy consumption penalty, and termination reward, used to comprehensively evaluate the safety, energy utilization, and maneuverability of the UAV; and establishing an Actor-Cri based on policy gradient. The critic deep neural network model employs two subsystems: the Actor network takes the current state of the UAV as input and outputs a continuous action vector; the Critic network takes the state-action pair as input and outputs the corresponding value estimation result. In the near-end policy optimization algorithm, advantage function normalization, generalized advantage estimation, gradient pruning, and learning rate decay mechanisms are introduced to reduce gradient oscillations and improve model convergence stability by limiting the policy update magnitude and optimization direction. The Actor-Critic network is trained using batch advantage estimation and multi-round gradient updates through continuous interaction with the environment to sample the state-action-reward dataset until policy convergence. After training, the resulting optimal policy model is used for UAV maneuver control, enabling autonomous evasion and optimal path planning in complex air combat environments.

[0005] Furthermore, in the construction of the UAV-missile confrontation environment model, the three-dimensional air combat simulation environment is used to provide state input and reward feedback for the reinforcement learning algorithm. The environment includes dynamic models of the UAV and missile, relative motion constraints, and a physical simulation calculation module; the environment input includes information such as the relative position, relative velocity, attitude angle, angular rate, and flight time of the UAV and missile; the environment output includes the state vector s for the next moment. t+1 Instant rewards t With the termination marker done t Within each time step, the environment reacts according to the action 'a' output by the agent. t Calculate the new state of the drone: s t+1 =f(s) t ,a t This completes the "state-action-state" transition, providing a dynamic interactive basis for subsequent reward calculations and strategy updates.

[0006] Furthermore, in the tiered reward mechanism, a tiered reward function is designed to characterize the maneuvering behavior of the UAV in evasion missions.

[0007] The reward function is determined by the termination condition reward r. done Yaw angle bonus and pitch angle bonus It consists of three parts: the termination condition reward is based on the distance R between the drone and the missile. t And the flight time t determines the mission result: The yaw angle bonus measures the attitude deviation of the UAV and the missile in the horizontal direction: in, This represents the difference in yaw angle between the UAV and the missile at time t. When the missile is far away, the UAV is encouraged to maintain a yaw angle difference of approximately π / 6 with the missile; when the distance is close, the UAV is encouraged to maneuver in the opposite direction of the missile to disrupt its guidance.

[0008] The pitch angle bonus measures the vertical attitude deviation between the UAV and the missile: in, This represents the difference in pitch angle between the UAV and the missile at time t. When the missile is far away, the UAV is encouraged to maintain the same pitch direction as the missile. When the distance is close, the UAV is encouraged to quickly adjust its pitch angle to avoid the attack.

[0009] Furthermore, the constructed improved PPO deep reinforcement learning model includes an Actor-Critic dual-network architecture, where the Actor network generates the UAV's action policy, and the Critic network estimates the value function of the current state. The network input is a state vector s. t The Actor outputs a continuous action a t Critic outputs state value V(s) t To improve training stability and generalization ability, the algorithm introduces advantage function normalization, generalized advantage estimation (GAE), and gradient pruning mechanisms on top of PPO. A policy network is constructed, and the state vectors are... Mapped to the mean μ of a Gaussian distribution θ (s) and diagonal variance and with As a continuous action strategy, actions are sampled and log probabilities are returned; a value network is constructed to map states to state values, which are used to construct temporal difference objectives and advantage estimates; δ is calculated on the sampled samples. t =r t +γV φ (s t+1 )-V φ (s t ), Use advantage normalization To reduce variance; using the old strategy The logarithmic probability yields the ratio Constructing PPO-Clip targets Mean square error in And add policy entropy To enhance exploration; with L = -L clip (θ)+c v L V (φ)-c e H(π θ As the optimization objective, the Adam optimizer is used to update the action network parameters and value network parameters respectively, and norm clipping is performed on the joint gradient: ||g||2≤g max Each update employs mini-batch partitioning and K-round iterations; using an approximation of KL divergence. Monitoring strategy changes when KL > 1.5KL target The current update will be terminated early.

[0010] Furthermore, in the continuous interactive sampling state-action-reward dataset of the environment, the environment and the agent interact in real time to form training samples. At each time step t, the agent, based on the current state s... t Select action a using the Actor network. t The action is input into the environment; after the environment executes the action, it returns to a new state s. t+1 Rewards r t and the termination marker "done" t , forming a sample: (s t ,a t ,r t ,s t+1 ,done t The samples are sequentially written into the experience replay pool, which uses a circular storage structure. When the pool is full, it automatically overwrites old data to support subsequent random batch sampling and training. This step forms a dynamic closed loop of "state input - action output - reward feedback - sample storage".

[0011] Furthermore, in step (5), the agent gradually constructs a state-action-reward dataset during continuous interaction with the environment, and trains the policy network based on batch advantage estimation and multi-round gradient update. Specifically, in each training round, the agent follows the current policy network π... θ After several rounds of interaction with the environment, multiple state transition samples (s) are obtained through sampling. t ,a t ,r t ,s t+1 ,done t The samples are then stored in the experience replay pool. Once the sample size reaches a preset threshold, samples are randomly sampled from the experience replay pool in mini-batches, and the generalized dominance estimate is calculated. δ t =r t +γV φ(s t+1 )-V φ (s t Where γ is the discount factor, λ is the time-dependent decay coefficient, and V φ (s t This represents the value estimate of the state by the Critic network. After normalization using the advantage function, it is then... This is used to correct the policy gradient direction, thereby balancing the policy update magnitude and learning stability. Subsequently, the policy loss, value loss, and entropy regularization term are calculated for each batch: L total =-L clip (θ)+c v L V (φ)-c e H(π θ The Actor-Critic network parameters are updated using the Adam optimizer through multiple rounds of gradient updates. Each update limits the range of KL divergence variation to prevent excessive policy shift. During training, the algorithm continuously repeats the closed-loop operation of "interactive sampling—advantage estimation—batch update" until the policy converges. Training terminates when the average reward or success rate reaches a set threshold, ultimately yielding the converged optimal policy network parameters θ. * With value network parameters φ * .

[0012] Furthermore, the optimal strategy model is used in the maneuver control of the UAV. After the network training converges, the optimal strategy network parameters are solidified into the control module. During real-time operation, the UAV senses its own state and its relative position to the missile, inputs the state into the strategy network, and outputs corresponding acceleration and attitude angle control commands to achieve autonomous evasion and optimal path planning in a highly dynamic air combat environment.

[0013] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the reinforcement learning-based UAV path planning method described above.

[0014] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the reinforcement learning-based unmanned aerial vehicle path planning method described above.

[0015] This invention has the following beneficial effects: it constructs a physically constrained 3D UAV-missile confrontation simulation environment, providing a high-fidelity training platform for reinforcement learning; it designs a hierarchical reward mechanism, effectively alleviating the sparse reward problem and enabling the intelligent agent to continuously obtain meaningful feedback; the improved PPO algorithm introduces GAE and gradient pruning mechanisms, significantly improving policy stability and convergence speed; through experience replay and policy iteration, it realizes a closed-loop reinforcement learning structure, improving training efficiency and generalization performance; and it achieves autonomous avoidance and optimal path generation for UAVs in high-adversarial environments, significantly improving the avoidance success rate. Attached Figure Description

[0016] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of a UAV path planning method based on reinforcement learning is provided as an embodiment of the present invention;

[0018] Figure 2 This is a block diagram of a reinforcement learning-based unmanned aerial vehicle (UAV) path planning system provided in one embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0021] With the widespread application of unmanned combat equipment in modern warfare, drones have played a crucial role in reconnaissance, strike, and surveillance missions. However, in complex and dynamic air combat scenarios, the evasion capability of drones when facing high-speed missile threats directly affects the success or failure of the mission. Traditional evasion methods mainly include artificial potential field methods, expert system methods, and game theory methods. Artificial potential field methods rely on physical modeling and distance potential function design. Although they have a certain degree of engineering feasibility and real-time performance, they are difficult to balance robustness and flexibility in highly dynamic and multi-constrained three-dimensional air combat environments. Expert system methods, by constructing rule bases or decision trees, can achieve stable decision-making under preset scenarios, but they cannot adapt to the changing environment of unknown threats. Although game theory-based evasion strategies can characterize the adversarial relationship between drones and missiles, their solution process is complex and computationally expensive, which is not conducive to the execution of real-time evasion missions.

[0022] To address the aforementioned issues, deep reinforcement learning has been introduced into UAV avoidance tasks in recent years. Leveraging the strong fitting and generalization capabilities of deep neural networks, it can directly learn avoidance strategies through environmental interaction without prior modeling. Commonly used algorithms include the Deep Deterministic Policy Gradient (DDPG) algorithm and the Proximal Policy Optimization (PPO) algorithm. DDPG achieves UAV maneuver control through policy gradient updates in a continuous action space, exhibiting strong convergence capabilities. However, it suffers from policy oscillations and insufficient stability in highly adversarial environments. The PPO algorithm utilizes a truncated proportional function to suppress the magnitude of policy changes, balancing policy exploration and stable updates, effectively avoiding gradient explosion and policy degradation. However, in highly dynamic air combat scenarios, traditional PPO still has the following shortcomings: the advantage estimation is not normalized, leading to uneven sample weights at different stages and affecting convergence speed; the reward design is simplistic, failing to reflect the dynamic situational changes of the UAV in continuous time; and the policy update step size is fixed, easily getting trapped in local optima in highly sparse reward scenarios.

[0023] To address the aforementioned problems, this invention proposes an improved UAV path planning method based on deep reinforcement learning, introducing a proximal policy optimization algorithm to enhance stability and generalization. The algorithm improves upon the original PPO structure as follows:

[0024] ① Advantage function normalization mechanism: By standardizing the advantage values ​​at different time steps, the variance difference between samples is reduced, and the accuracy of gradient estimation and the stability of policy update are improved.

[0025] ② Reward function scaling and hierarchical design: The reward signal simultaneously considers multiple factors such as termination conditions, yaw angle and pitch angle changes, forming a hierarchical reward mechanism, so that the UAV can obtain dense feedback signals during continuous flight, thereby avoiding the training non-convergence problem caused by sparse rewards;

[0026] ③ Gradient clipping and reward entropy optimization: Gradient clipping and entropy term penalty are introduced into the objective function to prevent premature convergence to a local optimum by limiting the update step size and enhancing the exploration of the policy.

[0027] ④ Batch advantage estimation and multi-round update mechanism: By updating multiple small batches of samples instead of a single round of large batch training, the stability and convergence speed of the algorithm in complex air combat scenarios are improved.

[0028] Please see Figure 1 This invention illustrates an improved UAV path planning method based on deep reinforcement learning, provided by an embodiment of the present invention. The method includes: Step S101: Construction of a three-dimensional air combat environment. Currently, most studies employ an integrated modeling approach in UAV evasion modeling, unifying the description of the dynamics, guidance, and control processes of the UAV and missile, with control commands as input and pose changes as output. However, this approach suffers from high model coupling and complex parameter adjustments, making efficient state updates difficult during reinforcement learning training. To improve model scalability and simulation accuracy, this invention proposes a three-dimensional air combat environment modeling method based on hierarchical decoupling. This method models the UAV and missile separately, establishing independent kinematic equations and guidance laws, allowing models to run independently, be trained interactively, or be combined for simulation. Compared to traditional coupled modeling methods, this method offers significant advantages in parameter controllability, physical interpretability, and training efficiency. It effectively reduces error accumulation between models, improves state feedback accuracy, and provides a stable and reliable dynamic environment for subsequent deep reinforcement learning-based path planning algorithms. In the specific implementation process, the present invention first constructs a three-dimensional simulation environment for UAV-missile confrontation. The environment input includes the relative position, speed, attitude and distance information of the UAV and the missile, and the output includes state vector, reward signal and termination flag.

[0029] The structure of the three-dimensional air combat environment model proposed in this application is described below.

[0030] To improve the algorithm's versatility and reduce simulation computational complexity, this invention makes the following assumptions in environmental modeling: the parameters of both the UAV and the missile are known; the influence of Earth's curvature and surface deformation on the flight trajectory is ignored; aerodynamic disturbances are ignored, and only the relative motion relationship between the missile and the UAV is considered; the missile is in a cruise state during the launch phase, with a constant speed and direction that changes with guidance commands.

[0031] In the inertial coordinate system, let the magnitude of the missile's velocity at time t be... Yaw angle is pitch angle is Location is The velocity vector is Its three-axis velocity components can be expressed as:

[0032]

[0033] Velocity and position information are updated with time step Δt according to the following rules:

[0034]

[0035] in To accelerate the missile, and Let be the rates of change of yaw and pitch angles, respectively. Similarly, in missile proportional guidance, in the inertial coordinate system, let the magnitude of the UAV's velocity at time t be... Yaw angle is pitch angle is Location is The three-axis velocity components and their update rules are as follows:

[0036]

[0037] in For the acceleration of the drone, and The yaw and pitch angles are the rates of change, and their values ​​are determined by the actions output by the reinforcement learning policy.

[0038] The missile guidance system employs a proportional guidance law model. The principle is to adjust the missile's trajectory so that its acceleration is proportional to the line-of-sight angular velocity between the target and the missile, thereby achieving efficient interception. The line-of-sight distance and angular velocity are defined as follows:

[0039]

[0040] The missile's horizontal and vertical accelerations in the line-of-sight coordinate system are as follows:

[0041]

[0042] Where K is the guidance scaling factor. Through the rotation matrix... Transforming the missile acceleration from the line-of-sight coordinate system to the inertial coordinate system, we get:

[0043]

[0044] This yields the missile's three-dimensional acceleration vector.

[0045] In the aforementioned simulation environment, the state variables (position, velocity, attitude angle, etc.) of the UAV and missile serve as inputs to the reinforcement learning model. The environment updates its state and calculates reward signals based on the actions output by the algorithm (UAV maneuvering acceleration and attitude changes). The hierarchical reward mechanism proposed in this application is explained below.

[0046] Step S102: Design of a hierarchical reward mechanism; To achieve intelligent evasion decision-making for UAVs during missile attacks, this invention designs an evaluation mechanism based on hierarchical rewards during reinforcement learning training. The reward function dynamically assigns values ​​based on the relative state characteristics of the UAV and the missile at different stages, guiding the agent to gradually develop maneuvering behaviors consistent with air combat principles during training. Specifically, it includes three sub-items: termination condition reward, yaw angle reward, and pitch angle reward.

[0047] The termination condition reward is used to determine the success of the drone's mission evasion. When the relative distance between the drone and the missile is less than a safety threshold, it is considered a hit and a severe penalty is imposed; if the drone successfully evades the missile attack by the end of the mission, a positive reward is given. The reward is designed as follows:

[0048]

[0049] Among them, R t This represents the distance between the drone and the missile at the current moment, where t is the time step. This reward term, by clearly distinguishing between successful and failed states, ensures that the policy gradient converges rapidly towards successful evasion during training.

[0050] Yaw angle bonus is used to balance the horizontal attitude difference between the UAV and the missile. When the missile is far away, the UAV is encouraged to maintain an appropriate yaw angle difference with the missile to maintain an evasive posture; when the missile approaches, the UAV is encouraged to quickly adjust its course to disrupt the missile's guidance. The specific calculation method is as follows:

[0051]

[0052] in, This represents the yaw angle difference between the drone and the missile. When R t When the yaw angle is greater than 30, the system encourages the UAV to maintain a yaw angle difference of around π / 6 to maintain a long-range evasive posture; when R t When the value is ≤30, UAVs are encouraged to change course rapidly, thereby effectively interfering with the missile's guidance direction.

[0053] Pitch angle reward measures the vertical attitude difference between the UAV and the missile. Similar to yaw angle reward, it encourages maintaining a stable pitch angle when the missile is far away, and encourages rapid adjustment of the pitch angle to change the flight trajectory when the missile approaches. The specific calculation formula is as follows:

[0054]

[0055] in, This indicates the pitch angle difference between the UAV and the missile. When the distance is greater than 30m, the UAV is encouraged to maintain a pitch angle difference close to 0 with the missile; when the distance is less than 30m, the UAV is encouraged to quickly change its pitch angle to escape the missile's acquisition range.

[0056] To comprehensively measure the overall effectiveness of drone evasion behavior, this invention uses a weighted summation method to merge the three types of rewards, forming a total reward function:

[0057]

[0058] This tiered reward mechanism effectively solves the problem of sparse rewards, enabling agents to obtain real-time feedback consistent with changes in the air combat situation in a continuous time sequence, thereby gradually forming maneuvering evasion behaviors that conform to the air combat mechanism during the training process.

[0059] In one implementation, a UAV combat simulation platform is built based on the Python language. The platform includes a 3D air combat environment modeling module, an intelligent agent interaction module, and a visualization training interface module. This platform enables state interaction, action decision-making, and reward feedback between UAVs and missiles, providing high-precision simulation training data for deep reinforcement learning models.

[0060] In the simulation platform, the initial state parameters of the UAV and missile are first set, including position, velocity, attitude angle and proportional guidance law parameters. Then, multiple rounds of simulation are run in continuous space according to different state combinations. The state quantities (position, velocity, attitude angle) of the UAV and missile, the control actions (pitch angle, yaw angle, acceleration) output by the agent and the reward value are collected at each time step to form a preliminary UAV simulation dataset.

[0061] Step S103: Random Initialization and State Sampling Strategy; To ensure data balance and representativeness, this invention employs a data sampling strategy combining random initialization and seed control. In one embodiment, a random initialization method for three-dimensional air combat simulation is provided to generate the initial states of UAVs (targets) and missiles; the method supports random seed control to ensure experimental reproducibility, and the specific steps are as follows.

[0062] Step S201: Input an optional random seed. If the seed is not empty, the pseudo-random number generator is invoked to set the seed; if the seed is empty, the system will generate a random number in the interval [0, 10]. 6 A random integer is generated within the seed and sent back to ensure the reproducibility of subsequent experiments.

[0063] Step S202: Initial UAV position sampling. Independently and uniformly sample the target position vector within the 3D bounding box.

[0064]

[0065] in

[0066] Step S203: Missile initial position sampling (ring domain + geometric constraints). Uniformly sample direction d on a unit sphere and perform norm normalization ||d||=1; then sample the distance interval [R... min ,R max The missile position is obtained by uniformly sampling at a scale r within the range [MISSILE_AROUND_MIN, MISSILE_AROUND_MAX]. Sampling must meet the following constraints: Height constraint: Rear-end collision constraints: (Ensure the missile is at least 50m behind the target). If this condition is not met, backtrack and resample until both constraints are met simultaneously.

[0067] Step S204: Initialize the speed and attitude of the UAV and missile. Speed ​​and attitude are sampled independently using the following formula: Where ψ represents the yaw angle and θ represents the pitch angle. Let the unit direction of the missile pointing towards the target be . The missile velocity is then taken as a constant. The missile's yaw and pitch angles are calculated as follows:

[0068]

[0069] This ensures the missile's initial orientation is towards the target and handles degradation cases with minimal horizontal components (ε is a numerical stability term).

[0070] Step S206: State vector encapsulation and output. Output the target state vector. With missile state vector And the seed used to reproduce the experiment. The above method, while maintaining the original random range and physical constraints, ensures the controllability and reproducibility of the initial situation through joint constraints of "ring domain + tail-tracking + altitude"; the missile orientation is calculated according to the geometric relationship of "pointing at the target," avoiding invalid initial attitudes. This initialization scheme, combined with a hierarchical reward mechanism and PPO training process, can stably produce training trajectories with broad coverage, reproducibility, and sufficient adversarial characteristics.

[0071] Step S104: UAV missile evasion simulation and data acquisition; In the simulation platform, the process of using the platform to conduct UAV missile evasion simulation and acquire training data is as follows:

[0072] ① Run the simulation main program, set the training parameters, number of rounds, random seed and save path in the configuration file; initialize the three-dimensional spatial position, attitude angle and velocity of the UAV and missile to complete the construction of the adversarial environment;

[0073] ② The simulation loop is started through the environment interaction module. At each time step, the agent outputs an action vector (including the acceleration, yaw rate and pitch rate of the UAV) from the policy network according to the current state, and inputs the action into the simulation environment to update the state. The environment returns the new state quantity, immediate reward and termination flag, realizing the closed-loop interaction of state-action-reward.

[0074] ③ Record the kinematic state variables of the UAV and missile at a preset sampling frequency. The data includes position coordinates, linear velocity, attitude angle, angular velocity, acceleration, relative distance and angle information, etc. At the same time, save the reward sequence, strategy output and simulation termination conditions for each round.

[0075] ④ When a training round ends or a set number of steps are reached, the system automatically writes the state-action-reward sequence of that round into the data cache or storage file, forming a complete training sample. To ensure sample balance, a random initialization and seed control mechanism is adopted to ensure that the initial relative position, speed, and attitude of the UAV and missile are randomly distributed within a reasonable range;

[0076] ⑤ Repeat steps ①–④ to perform batch simulation sampling, accumulating multiple batches of UAV missile evasion interaction trajectories and reward data. During the data collection process, the platform updates the environmental state at a time step of 60Hz, generating an average of approximately 10 data points per simulation round. 4 10 data records, eventually forming a scale of over 10 6 The training dataset of samples can be used for offline pre-training and online policy updates of reinforcement learning models.

[0077] This data acquisition process can generate high-fidelity behavioral samples of UAVs under multiple initial conditions and multiple adversarial scenarios while ensuring the consistency of environmental physical constraints, providing stable data support for subsequent PPO strategy optimization and generalization verification.

[0078] Step S105: Training and Policy Generation of the Improved Near-End Policy Optimization Algorithm. The improved near-end policy optimization algorithm model is trained using the training dataset generated by the three-dimensional air combat environment model. During training, the Actor-Critic network structure is first initialized based on preset hyperparameters (including learning rate, discount factor, entropy coefficient, and shearing threshold, etc.). Subsequently, the state, action, and reward data of the UAV and missile are continuously sampled through environmental interaction, forming a state-action-reward sequence and storing it in the experience replay pool. The model calculates the advantage function based on the experience data and performs multiple rounds of gradient updates until the policy converges. The hyperparameters include: a learning rate of 3×10⁻⁶. -4 Discount factor γ = 0.99, generalized advantage estimation (GAE) coefficient λ = 0.95, policy shearing threshold ε = 0.2, entropy coefficient c e =0.01, value function loss weight c v =0.5. The network training batch size is set to 4096, and each batch is further divided into mini-batches of 256 for 10 rounds of iterative updates. Parameter optimization uses the Adam optimizer, with the upper limit of the gradient norm set to 0.5 to prevent gradient explosion. Subsequently, the model continuously samples the state, action, and reward data of the UAV and missile through continuous interaction with the simulation environment, forming state-action-reward triples and storing them in the experience replay pool. When the accumulated number of samples in the experience pool reaches the batch size, the algorithm starts the policy update process, calculates the advantage function for each step, and performs multiple rounds of gradient descent. To improve training stability, a linear learning rate decay strategy is adopted during training, and a KL divergence early termination mechanism is used to monitor policy changes. When the current KL divergence exceeds the target value of 1.5 × 10⁻², the update round ends early.

[0079] The basic training process of the proximal policy optimization algorithm can be found in relevant technical literature. Its core includes the following steps: based on the environmental interaction sampling state-action-reward sequence, calculate the corresponding expected return and advantage function, and then construct the policy loss function; subsequently, update the parameters of the policy network and value network using gradient descent, and continuously optimize the policy in multiple iterations to improve the agent's cumulative return. The calculation of expectation, advantage estimation, parameter updating, and iterative optimization processes of the above algorithm can all refer to existing technical solutions for proximal policy optimization algorithms, and will not be elaborated further here.

[0080] The focus of this invention is to address the problems of training instability, excessive policy update amplitude, sparse rewards, and insufficient generalization inherent in standard near-end policy optimization algorithms in highly dynamic 3D air combat scenarios, and to propose an improved mechanism. This mechanism achieves a balance between the stability and exploratory nature of policy updates by introducing advantage function normalization, generalized advantage estimation, gradient pruning, and joint optimization with entropy regularization terms. The improved near-end policy optimization algorithm proposed in this invention will be described in detail below.

[0081] In one implementation, to improve the convergence speed and stability of the near-end policy optimization algorithm in a three-dimensional air combat environment, this invention proposes an improved near-end policy optimization algorithm. This algorithm introduces several stability enhancement mechanisms on the basis of the original framework to suppress gradient oscillations, reduce training variance, and improve the generalization ability of the policy.

[0082] Before each parameter update, the improved proximal policy optimization algorithm first standardizes the batch advantage function to reduce variance and accelerate convergence. The calculation formula is as follows: in, Here, μ(A) represents the standardized odds value, σ(A) represents the mean of the batch odds, and σ(A) represents the standard deviation of the batch odds. This standardization process effectively suppresses batch variance fluctuations and improves training stability.

[0083] To further reduce the fluctuation range of cumulative rewards, this invention adopts a sliding normalized discount reward design, specifically calculated as follows: Where, r t For immediate rewards, γ is the discount factor. For the moving average reward, Var(R) t Let be the variance of the reward. This design can suppress outliers and improve the stability of the Critic network objective when the reward signal is sparse.

[0084] During the parameter update phase, to prevent gradient explosion or over-update, a gradient clipping mechanism is used to constrain the gradient norm. The update formula is as follows: g←clip(g,-g) max ,g max ), where g is the gradient vector, g max This is the gradient clipping threshold. By limiting the gradient magnitude, policy oscillations caused by excessively rapid parameter updates in the early stages of training can be effectively prevented.

[0085] Meanwhile, to ensure optimization stability during long-term training, this invention introduces a linear decay mechanism for the learning rate in each iteration. The learning rate update formula is as follows: Where, α tLet α be the current learning rate, α0 be the initial learning rate, t be the current iteration number, and T be the total number of training steps. This mechanism can achieve a dynamic balance from the exploration to the convergence phase, avoiding premature entry into local optima.

[0086] Furthermore, to enhance the model's generalization ability and robustness to the input state, this invention introduces a moving mean variance normalization process for the environmental state variables, the calculation formula of which is as follows: Among them, s t Let μ(s) be the current state vector, μ(s) be the moving mean of the states, and σ(s) be the moving standard deviation. This normalization operation can effectively reduce the impact of differences in input distribution under different adversarial scenarios on model stability.

[0087] In summary, the improved near-end policy optimization algorithm, based on the standard near-end policy optimization algorithm, achieves high stability and generalized generalization capability in policy updates through multiple mechanisms such as advantage function standardization, reward sliding normalization, gradient pruning, and dynamic learning rate decay. Multiple adversarial experiments have verified that this algorithm can achieve a smoother convergence process and superior UAV evasion strategy performance in complex air combat environments.

[0088] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a UAV path planning training method based on an improved proximal policy optimization algorithm.

[0089] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0090] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0091] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink, DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An improved UAV path planning method based on deep reinforcement learning, characterized in that, The method includes: A three-dimensional air combat simulation environment is constructed, and the initial state information of the UAV and missile in the simulation environment is obtained. This state information includes the relative position, velocity, attitude, and distance parameters of the UAV and missile. The three-dimensional air combat simulation environment is initialized, including resetting the initial position, velocity, and attitude of the UAV and missile to begin a new training round. An improved deep reinforcement learning model based on a near-end policy optimization algorithm is used to select actions based on the current state of the UAV and missile. These actions include the UAV's maneuvering acceleration and angular velocity commands, and the actions are output to the three-dimensional air combat simulation environment. The three-dimensional air combat simulation environment updates the state of the UAV and missile based on these actions, calculates the reward value of the UAV and missile at the current moment, and determines whether the current round has ended. Termination is generated when any of the following conditions are met: successful escape, being hit, or reaching the maximum time step. The current round is marked and ended; the state of the environment at adjacent time steps, the action, the corresponding reward value, and the state at the next time step are stored in the experience replay pool; when the amount of data stored in the experience replay pool reaches a preset threshold, a batch of data is randomly sampled from the experience replay pool, and the corresponding loss function is calculated. The loss function includes policy loss, value function loss, and entropy regularization term; gradient descent is performed based on the loss function to update the parameters of the policy network and the value network, and the state interaction between the UAV and the environment and the parameter update of the policy network and the value network are defined as the training interaction and parameter update process; the training interaction and parameter update process is repeated until the number of training rounds reaches a preset number; after training is completed, the optimized policy network is used to perform path planning control on the UAV to realize the autonomous decision-making and optimal avoidance path generation of the UAV in a high-confrontation environment.

2. The improved UAV path planning method based on deep reinforcement learning according to claim 1, characterized in that, The three-dimensional air combat simulation environment includes a nine-dimensional state vector as the observation output, defined as follows: Where Δx, Δy, and Δz are the relative position components between the UAV and the missile, v t For the speed of the drone, These represent the yaw and pitch angles of the UAV, respectively. m For the speed of the missile, These are the missile's yaw and pitch angles.

3. An improved UAV path planning method based on deep reinforcement learning according to claim 1, characterized in that, The method for initializing the three-dimensional air combat simulation environment includes: fixing the randomness of the environment by calling a random seed function to ensure the reproducibility of the training process; randomly generating the initial position coordinates of the UAV within a given spatial range, and setting the initial velocity V of the UAV. t ∈[8,10]m / s, initial yaw and pitch angles are uniformly sampled within a defined range; based on the UAV position p t Generate the initial position of the missile, p m =p t +R[cosα,sinα,sinβ],R∈[75,125]m where the missile's lag distance relative to the UAV in the x-direction is not less than 50m; set the missile's initial velocity V. m =20m / s, the missile yaw angle and missile pitch angle point to the UAV position; the relative position, relative velocity, attitude angle and distance information of the UAV and the missile are combined into a nine-dimensional initial state vector, and the simulation time step t is reset to 0, and the cumulative reward R is 0.

4. An improved UAV path planning method based on deep reinforcement learning according to claim 1, characterized in that, The improved deep reinforcement learning model based on the proximal policy optimization algorithm includes: constructing a policy network and processing the state vector... Mapped to the mean μ of a Gaussian distribution θ (s) and diagonal variance and with As a continuous action strategy, a policy network generates continuous action control commands for the UAV based on the current state information, and simultaneously calculates the probability evaluation index corresponding to the generated actions for subsequent policy updates; a value network is constructed to output the corresponding state value evaluation result based on the input state information, for subsequent reward feedback analysis and advantage information estimation; the collected training samples are stored and processed. δ is calculated. t =r t +γV φ (s t+1 )-V φ (s t ), Use strength normalization to reduce variance; use the old strategy The logarithmic probability yields the ratio Constructing PPO-Clip targets Mean square error in And add policy entropy To enhance exploration; with L = -L clip (θ)+c v L V (φ)-c e H(π θ As the optimization objective, the Adam optimizer is used to update the action network parameters and value network parameters respectively, and norm clipping is performed on the joint gradient: ||g||2≤g max Each update employs mini-batch partitioning and K-round iterations; using an approximation of KL divergence. Monitoring strategy changes when KL > 1.5KL target The current update will be terminated early.

5. An improved UAV path planning method based on deep reinforcement learning as described in claim 1, characterized in that, A tiered reward mechanism is introduced, comprising three parts: termination condition reward, yaw angle reward, and pitch angle reward. The termination condition reward is used to determine the outcome of the engagement between the UAV and the missile, and is calculated based on the distance between the UAV and the missile and the current time, defined as follows: Among them, R t This represents the distance between the UAV and the missile at time t, where t is the current simulation time step. A strong negative reward is given when the UAV is hit by the missile or approaches a danger threshold to punish failure; a positive reward is given when the UAV successfully avoids the missile or is not hit by the missile at the end of the mission. The yaw angle reward is used to measure the attitude deviation between the UAV and the missile in the vertical direction, and is defined as follows: in, This represents the yaw angle difference between the UAV and the missile at time t. When the missile is far away, the UAV is encouraged to maintain a yaw angle difference of approximately π / 6 with the missile; when the distance is close, the UAV is encouraged to maneuver in the opposite direction to the missile to disrupt guidance. The pitch angle reward is used to measure the attitude deviation between the UAV and the missile in the vertical direction, and is defined as follows: in, This represents the difference in pitch angle between the UAV and the missile at time t. When the missile is far away, the UAV is encouraged to maintain the same pitch direction as the missile. When the distance is close, the UAV is encouraged to quickly adjust its pitch angle to avoid the attack.

6. The improved UAV path planning method based on deep reinforcement learning as described in claim 1, characterized in that, The experience replay pool is a matrix-based circular cache structure used to store interaction samples of UAVs and missiles in a three-dimensional air combat simulation environment in chronological order. The experience replay pool includes: setting a maximum capacity and configuring a sample counter to determine the write index to achieve circular overlay storage; using sample fields to store state vectors, action vectors, policy log probabilities, immediate rewards, next moment state, round end flags, and "win / loss / time limit" flags to support the calculation of temporal difference and generalized advantage estimation; and jointly constructing participation targets and advantages to achieve differentiated processing of termination and time limit truncation samples in value targets, thereby improving the stability of policy evaluation and updates.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the reinforcement learning-based unmanned aerial vehicle path planning method as described in any one of claims 1 to 6.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the reinforcement learning-based unmanned aerial vehicle path planning method as described in any one of claims 1 to 6.

Images