Unmanned aerial vehicle flight path planning method and system based on integral compensation

Abstract

The invention discloses an unmanned aerial vehicle flight path planning method and system based on integral compensation, and relates to the technical field of unmanned aerial vehicle flight path planning, an environment simulator for simulating the flight environment of an unmanned aerial vehicle is constructed, and threat and target positions are randomly generated when the environment simulator is reset each time; training an intelligent agent in the environment simulator by adopting reinforcement learning, modeling a route planning problem into a partial observable Markov decision process or a complete observable Markov decision process, and obtaining a baseline strategy of route planning; correcting the state error of the intelligent agent in real time based on an integral compensation mechanism, and smoothing the action instruction output by the intelligent agent by adopting an action filter; a strategy is optimized through an integral compensation reinforcement learning planning module, and the planning module uses a population optimization method to select a strategy with the maximum accumulated reward as an execution strategy and interacts with an environment simulator to generate a final track, so that high-success-rate, smooth and drivable track planning is realized.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) trajectory planning technology, and in particular to a UAV trajectory planning method and system based on integral compensation. Background Technology

[0002] Unmanned Aerial Vehicle (UAV) trajectory planning is the most crucial part of mission planning and a key technology for UAV collaborative operations and future networked warfare. Effective planning allows UAVs to effectively avoid threats, improving survivability and combat efficiency. Using intelligent optimization algorithms to solve the trajectory planning problem is currently the most widely used method, with Ant Colony Optimization (ACO) based on a grid model being one of the most important. Compared to the VORONOI graph method, this method eliminates the need for navigation nodes and VORONOI graph construction, automatically searching for the minimum-cost trajectory in free space. It exhibits strong adaptability and is applicable to various threat types.

[0003] Conversely, Reinforcement Learning (RL) offers advantages such as better real-time performance, excellent generalization, and a more universal design flow, enabling it to achieve superior performance in path planning problems in fields such as robotics and drones. Specifically, using probabilistic roadmaps (PRM) to segment multiple local target points on a large map, and then using a reinforcement learning agent trained with deep deterministic policy gradient (RL) to guide the robot towards these local target points, can solve long-distance path planning problems for robots in complex environments. Using the entire map image as the observation state of the reinforcement learning agent, the deep Q-network algorithm performs excellently in environments with both static and dynamic obstacles. All of the above methods are based on model-free reinforcement learning and use an "offline training and drone usage" model, but they do not discuss countermeasures for failures of the reinforcement learning agent during drone usage. Although gradient optimization methods used in the offline training phase can be used to continue training the reinforcement learning agent during drone usage as a countermeasure, this is too computationally expensive and reduces real-time performance.

[0004] Dynamic integral compensation (IC) mechanisms contain internal state variables and are relatively complex to design. However, the projection theorem can be used to transform the non-convex stability criterion into an LMI (Linear Matrix Inequality) stability criterion, which has weaker feasibility constraints and better dynamic response of the system. For the above integral compensation method, when the system is open-loop stable, it can achieve global asymptotic stability; when the system is open-loop unstable, it can only achieve local asymptotic stability.

[0005] Therefore, traditional methods such as Ant Colony Optimization (ACO) and genetic algorithms suffer from problems such as high computational cost, poor real-time performance, and susceptibility to local optima. Furthermore, existing methods have low success rates in dynamic threat environments and lack effective constraints on trajectory overload. There is an urgent need for an integral compensation reinforcement learning algorithm planning method that can simultaneously consider static and dynamic anti-integral compensation mechanisms, combining integral compensation and reinforcement learning algorithms to meet the trajectory planning requirements of unmanned aerial vehicles (UAVs). Summary of the Invention

[0006] In view of this, to address the problems existing in current unmanned aerial vehicle (UAV) trajectory planning methods, the present invention aims to provide an UAV trajectory planning method and system based on integral compensation reinforcement learning (RL-IC). By integrating the integral compensation mechanism with the RL framework, it eliminates static error, enhances action correlation, and introduces an environment simulator and action filter to achieve high success rate and smooth, flyable trajectory planning. The system supports both single-aircraft and formation missions and possesses strong generalization capabilities.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a method for unmanned aerial vehicle trajectory planning based on integral compensation, comprising the following steps:

[0009] An environment simulator is constructed to simulate the flight environment of an unmanned aerial vehicle. The environment simulator randomly generates threat and target locations each time it is reset.

[0010] The agent is trained in the environment simulator using reinforcement learning, and the trajectory planning problem is modeled as a partially observable Markov decision process or a fully observable Markov decision process to obtain the baseline strategy for trajectory planning.

[0011] The state error of the agent is corrected in real time based on the integral compensation mechanism, and the action command output by the agent is smoothed by the action filter.

[0012] The strategy is optimized through an integral compensation reinforcement learning planning module. The planning module uses a population optimization method to select the strategy with the largest cumulative reward as the execution strategy and interacts with the environment simulator to generate the final trajectory.

[0013] As a further aspect of the present invention, the environment simulator is used to simulate static threats, dynamic threats, target points, and aircraft dynamics models in the flight environment of unmanned aerial vehicles.

[0014] As a further aspect of the present invention, in the integral compensation mechanism, the attenuation factor λ is 0.8 and the compensation coefficient β is 0.3; the state error after integral compensation replaces the original state error without increasing the dimension of the state space, and is used to eliminate the steady position error, so that the position tracking error is reduced to zero.

[0015] As a further aspect of the present invention, the integral compensation state error in the integral compensation mechanism is defined as:

[0016] ;

[0017] In the formula, for The error of the original state at time t. for The state error after compensation at time step, For compensation coefficient, As the attenuation factor, Weights used to adjust for past errors; The contribution of the old error decays exponentially over time; From time step 1 to The weighted cumulative state error represents the sum of all past errors, with the weight of past errors decreasing as the time distance from the present increases.

[0018] As a further aspect of the present invention, the motion filter includes a momentum filter and an interpolation filter; the momentum filter is used to calculate the local average value of the motion difference through an exponentially weighted average, and the calculation formula of the momentum filter is:

[0019] ;

[0020] ;

[0021] In the formula, To simulate steps The exponentially weighted average of the action differences; These are the weighting coefficients used to control the weight of historical values; This is a weighted average of the previous simulation steps; For intelligent agents at the decision-making moment The original action output; The action actually passed to the simulator in the previous simulation step; For the clipping function, limit the difference to... Within the range; The smoothed motion; The motion update rate is used to control the magnitude of changes in smooth motion.

[0022] The interpolation filter is used to generate action transition functions through Hermite interpolation. The calculation formula for the interpolation filter is as follows:

[0023] ;

[0024] In the formula, In order to make decisions Smoothed motion; This is a smoothing action from the previous decision-making moment; For the agent at the current decision moment The original action output; These are the weighting coefficients used to balance the old and new actions.

[0025] As a further aspect of the present invention, when selecting the strategy with the largest cumulative reward as the execution strategy, the cumulative reward is a continuous heuristic reward function, and the reward function is calculated as follows:

[0026] ;

[0027] In the formula, This is the total reward value based on the Euclidean distance between the current coordinates and the target point; Basic rewards; This is a weighting factor used to adjust the strength of heuristic rewards; The distance is Euclidean. and These are weight parameters; The reward is a fixed distance threshold; when an obstacle is encountered, the reward is negative infinity; when the destination is reached, the reward is 100; and in other situations, the reward is 0.

[0028] As a further aspect of the present invention, the optimization process of the integral compensation reinforcement learning planning module includes:

[0029] Set the population size P=100, the number of elite individuals E=10, the planning length H=250 steps, and the number of iterations K=10;

[0030] The cumulative reward J(θ) is calculated by predicting future states using a sub-environment simulator.

[0031] Update the strategy parameters using a soft update method;

[0032] If the integral compensation optimization fails, revert to the baseline strategy;

[0033] The planning module supports multi-UAV formation planning and achieves collaborative trajectory generation by introducing formation status and rewards.

[0034] As a further aspect of the present invention, the network structure of the intelligent agent is a policy-evaluation architecture, wherein the policy network contains two hidden layers, each with 128 neurons and ReLU activation function, and the output layer uses the tank activation function to limit the range of control signals; the evaluation network input includes state and action, the hidden layer structure is the same as the policy network, and the output layer uses a linear activation function to estimate the Q value; the network parameters are updated through gradient pruning and priority experience replay.

[0035] Secondly, the present invention also provides an unmanned aerial vehicle trajectory planning system based on integral compensation, comprising:

[0036] The environment simulator module is used to build an environment simulator that simulates the flight environment of unmanned aerial vehicles. The environment simulator randomly generates threat and target locations each time it is reset.

[0037] The reinforcement learning training module is used to train the agent in the environment simulator using reinforcement learning, model the trajectory planning problem as a partially observable Markov decision process or a fully observable Markov decision process, and obtain the baseline policy for trajectory planning.

[0038] The compensation and filtering module is used to correct the state error of the agent in real time based on the integral compensation mechanism, and to smooth the action commands output by the agent using the action filter.

[0039] The integral compensation planning module is used to optimize the strategy through the integral compensation reinforcement learning planning module. The planning module uses a population optimization method to select the strategy with the maximum cumulative reward as the execution strategy and interacts with the environment simulator to generate the final trajectory.

[0040] As a further aspect of the present invention, the environment simulator module further includes a sub-simulator unit for predicting future states; the environment simulator module sets a random initial position each time it is reset to ensure that the planning strategy does not depend on a fixed environment, and ensures the smoothness of the pose points through the action filtering module.

[0041] As a further aspect of the present invention, the unmanned aerial vehicle trajectory planning system also includes a reward function module, which designs a continuous heuristic reward function, including a basic reward, a distance reward, a heading reward, a target point reward, and a required overload reward.

[0042] Compared with existing technologies, the unmanned aerial vehicle trajectory planning method and system based on integral compensation provided in this invention have the following beneficial effects:

[0043] 1. This invention sets up an action filter to smooth the actions output by the agent through exponential weighted averaging; it uses a momentum filter to calculate the local average of the action differences and generates smooth actions with a small update rate; the interpolation filter achieves action transition through Hermite interpolation, ensuring the continuity of actions at adjacent decision moments, eliminating action abrupt changes, ensuring curvature continuity, improving track smoothness, and solving the problem that traditional RL algorithms are prone to abrupt changes in track curvature due to the lack of temporal correlation of actions, which affects flight stability.

[0044] 2. This invention uses an integral compensation mechanism to accumulate and weight-compensate past errors to the current state, eliminating the steady-state error problem in position control that exists in conventional RL algorithms. The compensated state error replaces the original error, avoiding the expansion of the state space dimension, while ensuring that the policy network continuously outputs corrective actions until the steady-state error is zero.

[0045] 3. This invention also models trajectory planning as a POMDP / MDP problem, randomly generating threats and targets through an environment simulator to train an agent independent of a fixed environment. The RL-IC online planning phase uses a population optimization method to select the strategy with the highest cumulative reward as the execution strategy. Continuous guidance is provided through continuous heuristic rewards, avoiding the blindness of traditional discrete rewards. This invention can be extended to multi-UAV formation planning. In a formation environment, each UAV is independently trained through a role network and shares formation state and rewards, supporting both standalone and formation applications. This invention reduces the sensitivity to reward function settings through an integral compensation mechanism.

[0046] In summary, the present invention provides an unmanned aerial vehicle (UAV) trajectory planning method and system based on integral compensation. Through integral compensation to eliminate steady-state error, motion filtering to ensure smoothness, and RL-IC optimization strategy, it outperforms traditional methods in terms of control accuracy, success rate, and real-time performance. Experiments demonstrate that the system can adapt to complex dynamic environments, supports both single-unit and formation missions, and exhibits strong generalization ability and robustness.

[0047] These or other aspects of the invention will become more apparent from the following description of embodiments. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. In the drawings:

[0049] Figure 1 This is a flowchart of an unmanned aerial vehicle trajectory planning method based on integral compensation according to the present invention.

[0050] Figure 2 This is a flowchart illustrating the effective trajectory generation in an unmanned aerial vehicle trajectory planning method based on integral compensation according to the present invention.

[0051] Figure 3 This is a schematic diagram of the unmanned aerial vehicle (UAV) trajectory planning structure in an integral compensation-based UAV trajectory planning method of the present invention.

[0052] Figure 4 This is a schematic diagram of the training curve for an unmanned aerial vehicle trajectory planning method based on integral compensation according to the present invention.

[0053] Figure 5 This diagram illustrates some successful cases of an unmanned aerial vehicle trajectory planning method based on integral compensation according to the present invention.

[0054] Figure 6 This diagram illustrates the penetration success rate in an unmanned aerial vehicle trajectory planning method based on integral compensation according to the present invention.

[0055] Figure 7 This is a schematic diagram of the unmanned aerial vehicle (UAV) trajectory planning signal in an UAV trajectory planning method based on integral compensation according to the present invention.

[0056] Figure 8 This is a schematic diagram of the position and attitude response curve of an unmanned aerial vehicle (UAV) under the action of trajectory planning in an unmanned aerial vehicle trajectory planning method based on integral compensation according to the present invention.

[0057] Figure 9 This is a schematic diagram of the unmanned aerial vehicle (UAV) trajectory planning signal in an UAV trajectory planning method based on integral compensation according to the present invention.

[0058] Figure 10 This is a three-dimensional visualization of the UAV formation learning process in a 3D environment in an unmanned aerial vehicle trajectory planning method based on integral compensation according to the present invention.

[0059] Figure 11 This diagram illustrates the convergence process of the traditional Q-learning algorithm and the improved Q-learning algorithm in a simple environment in the trajectory planning method for unmanned aerial vehicles based on integral compensation according to the present invention. Detailed Implementation

[0060] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0061] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0062] like Figure 1 and Figure 2 As shown, one embodiment of the present invention provides a method for unmanned aerial vehicle trajectory planning based on integral compensation, including the following steps:

[0063] Step S10: Construct an environment simulator that simulates the flight environment of an unmanned aerial vehicle. The environment simulator randomly generates threat and target locations each time it is reset.

[0064] Step S20: Train the agent in the environment simulator using reinforcement learning, model the trajectory planning problem as a partially observable Markov decision process or a fully observable Markov decision process, and obtain the baseline strategy for trajectory planning.

[0065] Step S30: Correct the state error of the agent in real time based on the integral compensation mechanism, and smooth the action commands output by the agent using an action filter.

[0066] Step S40: Optimize the strategy through the integral compensation reinforcement learning planning module. The planning module uses a population optimization method to select the strategy with the largest cumulative reward as the execution strategy and interacts with the environment simulator to generate the final trajectory.

[0067] In this embodiment, the environment simulator is used to simulate static threats, dynamic threats, target points, and aircraft dynamics models in the flight environment of unmanned aerial vehicles.

[0068] The unmanned aerial vehicle (UAV) trajectory planning method of this invention consists of two stages: 1) an offline training stage, in which an RL agent independent of a fixed environment is trained as the baseline policy for trajectory planning; 2) an online planning stage, in which RL-IC uses an environment simulator to predict future states for planning, and then selects the policy superior to the baseline policy as the execution policy; otherwise, the baseline policy is used. In the first stage, an environment simulator is constructed to simulate a real flight environment containing the aircraft, threats, and targets. Each time the environment simulator is reset, a random initial position is assigned to the aircraft, threats, and targets, ensuring that the RL agent's planning policy is independent of a fixed environment. The ultimate goal of this stage is to train an RL agent to successfully control the virtual aircraft to perform a penetration mission in the virtual environment. In the second stage, RL-IC plans the optimal policy as the execution policy, which generates an effective trajectory through interaction with the environment simulator. Figure 1The process is described as follows: First, an IC policy is planned using IC, which maximizes the cumulative reward over the next H time steps starting from a given initial state. Then, the policy with the largest cumulative reward between the baseline policy and the IC policy is selected as the optimal policy. During the next planning round, this optimal policy is used as the execution policy to interact with the environment simulator. RL-IC is planned every few time steps. The initial state used for planning is Tp time steps ahead of the actual state at the start of this planning round. The initial state and its subsequent states are predicted by the sub-simulator. Planning and execution operations are performed asynchronously, which ensures the real-time performance of RL-IC. At each sampling moment of the real aircraft system, the environment simulator sends the pose point of the virtual aircraft at that time as the expected pose point of the real aircraft. The temporal smoothness of the expected pose point is ensured by the motion filter. The environment simulator will periodically update its dynamics according to the real environment.

[0069] The integral compensation state error in the integral compensation mechanism is defined as follows:

[0070] ;

[0071] In the formula, for The error of the original state at time t. for The state error after compensation at time step, For compensation coefficient, As the attenuation factor, Weights used to adjust for past errors; The contribution of the old error decays exponentially over time; From time step 1 to The weighted cumulative state error represents the sum of all past errors, with the weight of past errors decreasing as the time distance from the present increases.

[0072] Because the integral compensator increases the system's inertia and lengthens the unmanned aerial vehicle's trajectory planning settling time, and because the defined integral compensation state error includes both the current error and all accumulated errors from the past, the cumulative error becomes increasingly significant with time and the number of control rounds. This leads the control algorithm to overemphasize the accumulated error of old data and reduce its focus on the current state error, thereby degrading the algorithm's performance. In this embodiment, a weighting factor is introduced... This is used to adjust the proportion of past time-to-time errors in the cumulative error, so that the role of errors from previous time-to-time gradually decreases, ensuring that the algorithm pays attention to recent state errors, and achieving a balance between focusing on current performance and past cumulative errors.

[0073] In this embodiment, the unmanned aerial vehicle trajectory planning structure based on the RL-IC algorithm is as follows: Figure 2As shown, the overall framework of UAV trajectory planning adopts a policy-evaluation architecture, which must meet control requirements and ensure the training speed of UAV trajectory planning. Therefore, it is necessary to select a suitable neural network to construct the control policy and evaluation function.

[0074] In the integral compensation mechanism, the attenuation factor λ is set to 0.8, and the compensation coefficient β is set to 0.3. The state error after integral compensation replaces the original state error without increasing the dimension of the state space, and is used to eliminate steady-state error, reducing the position tracking error to zero. The policy network consists of a fully connected neural network with two hidden layers, each containing 128 neurons. The activation function of the hidden layers is the ReLU function. This policy network receives the compensated UAV state error and outputs four control signals. To ensure that the output control signals meet the requirements of the UAV, the output layer of this network uses the tank activation function. The specific structure of the evaluation network is as follows... Figure 3 As shown, the network consists of two fully connected neural networks with two hidden layers. Each hidden layer contains 128 neurons and uses the ReLU activation function. The network's inputs are the current state of the UAV and four control signals. The UAV state is directly input, while the four control signals are input in the first hidden layer. The network's output layer uses a linear activation function to output an approximation of the Q-function, thus obtaining the policy gradient, which is used to update the policy network.

[0075] In this embodiment, the motion filter includes a momentum filter and an interpolation filter; the momentum filter is used to calculate the local average of the motion difference using an exponentially weighted average, and the calculation formula for the momentum filter is:

[0076] ;

[0077] ;

[0078] In the formula, To simulate steps The exponentially weighted average of the action differences; These are the weighting coefficients used to control the weight of historical values; This is a weighted average of the previous simulation steps; For intelligent agents at the decision-making moment The original action output; The action actually passed to the simulator in the previous simulation step; For the clipping function, limit the difference to... Within the range; The smoothed motion; The motion update rate is used to control the magnitude of changes in smooth motion.

[0079] The interpolation filter is used to generate action transition functions through Hermite interpolation, and the calculation formula for the interpolation filter is as follows:

[0080] ;

[0081] In the formula, In order to make decisions Smoothed motion; This is a smoothing action from the previous decision-making moment; For the agent at the current decision moment The original action output; The weighting coefficients are used to balance the old and new actions. The interpolation filter calculates the local average of the actions given by the agent using an exponentially weighted average, and uses this local average as the smoothed action. Then, the smooth transition function of the actions at adjacent decision times is obtained through Hermite interpolation, and the actual action passed to the simulator is calculated in this way. This transition function is obtained by constraint interpolation, which can effectively ensure the smoothness of the action.

[0082] This invention sets up an action filter to smooth the actions output by the agent through exponential weighted averaging; it uses a momentum filter to calculate the local average of the action differences and generates smooth actions with a small update rate; the interpolation filter achieves action transition through Hermite interpolation, ensuring the continuity of actions between adjacent decision moments, eliminating abrupt action changes, ensuring curvature continuity, and improving track smoothness. This solves the problem that traditional RL algorithms are prone to abrupt changes in track curvature due to the lack of temporal correlation of actions, which affects flight stability.

[0083] When selecting the strategy with the largest cumulative reward as the execution strategy, the cumulative reward is a continuous heuristic reward function, and the reward function is calculated as follows:

[0084] ;

[0085] In the formula, This is the total reward value based on the Euclidean distance between the current coordinates and the target point; Basic rewards; This is a weighting factor used to adjust the strength of heuristic rewards; The distance is Euclidean. and These are weight parameters; A fixed distance threshold is used; the reward is negative infinity when encountering an obstacle, 100 when reaching the destination, and 0 in other situations. This heuristic reward function provides continuous, real-time rewards to the robot during the learning process, allowing it to quickly determine the most valuable action in its current action set, ensuring the robot consistently moves towards the destination and improving convergence efficiency.

[0086] In this embodiment, RL is combined with IC, and the IC is used to optimize the policy of the RL agent in the practical application stage. The optimization process of the integral compensation reinforcement learning planning module includes:

[0087] Set the population size P=100, the number of elite individuals E=10, the planning length H=250 steps, and the number of iterations K=10;

[0088] The cumulative reward J(θ) is calculated by predicting future states using a sub-environment simulator.

[0089] Update the strategy parameters using a soft update method;

[0090] If the integral compensation optimization fails, revert to the baseline strategy;

[0091] The planning module supports multi-UAV formation planning and achieves collaborative trajectory generation by introducing formation status and rewards.

[0092] This invention uses an integral compensation mechanism to accumulate and weight-compensate past errors to the current state, eliminating the steady-state error problem in position control that exists in conventional RL algorithms. The compensated state error replaces the original error, avoiding the expansion of the state space dimension, while ensuring that the policy network continuously outputs corrective actions until the steady-state error is zero.

[0093] In this embodiment, the network structure of the agent is a policy-evaluation architecture. The policy network contains two hidden layers, each with 128 neurons and ReLU activation function. The output layer uses the tank activation function to limit the range of control signals. The evaluation network inputs include state and action. The hidden layer structure is the same as the policy network. The output layer uses a linear activation function to estimate the Q value. The network parameters are updated through gradient pruning and priority experience replay.

[0094] This embodiment uses the Proximal Policy Optimization (PPO) algorithm to train the RL agent, and the relevant hyperparameters are reported in Table 1.

[0095] Table 1 Hyperparameters during the training phase

[0096] hyperparameters value Number of vectorized environments 32 Gradient clipping threshold 5 The weight vector k of the reward [1，5，1.5，1，1，1] hyperparameters [1，5，1.5，1，0.1，1]

[0097] In addition, this embodiment introduces a soft actor-critic (SAC) algorithm for comparison. The selection of hyperparameters, based on reference values, utilizes priority experience replay and fine-tunes the reward weight vector. The trajectory planned by the RL agent in this embodiment satisfies the constraints shown. Furthermore, the reward function design minimizes the required overload and trajectory length (estimated flight time) as much as possible. Fifteen threats with a radius of 50 km are set in the training environment, and their locations are randomly generated each time the environment is reset.

[0098] exist Figure 4 The training results on three sets of random number seeds are presented. The training results show that the modeling method of this embodiment can achieve the same performance on different RL algorithms; PPO has more stable performance during training; momentum filter has better performance than interpolation filter; the MDP (Markov Decision Process) method has higher sample efficiency than POMDP (Partially Observable Markov Decision Process), which this embodiment believes is due to the network structure used in MDP and the introduction of global information.

[0099] Since PPO exhibits more stable performance during training, this embodiment will primarily use agents trained with PPO for experimental verification. Figure 5 The figure shows some successful cases and the curves of required overload on the flight path. As can be seen from the figure, the motion filter designed in this embodiment effectively ensures the smoothness of required overload.

[0100] In this embodiment, the penetration success rate is used as the evaluation metric. All trained RL agents were evaluated in the training environment using a no-exploration-noise strategy across 500 scenarios, while RL-IC was evaluated across 100 scenarios. The hyperparameters used by RL-IC are shown in Table 2, and their respective penetration success rates are as follows: Figure 6 As shown in the figure, the penetration success rate of the RL agents trained in MDP is significantly higher than that in POMDP. Furthermore, the combination of RL-IC greatly improves the performance of each RL agent, achieving a near 100% success rate. In summary, the RL agents trained in MDP provide a reliable baseline policy that can be used not only for planning but also as a backup policy for contingencies. RL-IC compensates for the shortcomings of RL agents, further improving the penetration success rate.

[0101] Table 2 Hyperparameters of RL-IC

[0102] hyperparameters value Population size 100 Elite Individuals 10 Planning length 250 Number of iterations 10 Soft update rate 0.25 initial mean [0…0] initial standard deviation [0.5…0.5]

[0103] The UAV trajectory planning based on the conventional RL algorithm has obvious errors in position control, with position errors of 0.5m, 0.7m, and 0.6m in the x, y, and z directions, respectively. In contrast, the two UAV trajectory planning methods based on the RL-IC algorithm can stably control the UAV trajectory to the desired position with zero position error, indicating that the introduction of integral compensation can effectively eliminate steady-state error. Figure 7 The results show that the control signals output by the two UAV trajectory planning methods based on the RL-IC algorithm are significantly smoother than those based on the conventional RL algorithm, indicating that the introduction of integral compensation is beneficial to enhancing the stability of UAV trajectory planning.

[0104] This setting is used to verify the algorithm's performance under a coarse reward function. The control effect and control signals for the UAV trajectory planning are as follows: Figure 8 and Figure 9 As shown. Figure 8 The solid line represents the poor trajectory planning and control performance of the UAV based on the conventional RL algorithm, with no convergence in the position and attitude response curves. This indicates that, under the condition of a coarse reward function, the trajectory planning of the UAV based on the conventional RL algorithm has not learned a stable control strategy. The trajectory planning of the UAV based on the RL-IC (r=1) algorithm, represented by the dashed line in the figure, shows better control performance than the conventional RL UAV trajectory planning. Attitude control is oscillating, indicating that the addition of accumulated error enhances the algorithm's stability. However, the convergence speed of the RL-IC (r=1) UAV trajectory planning is too slow, failing to control the position to the desired position within the simulated 50 seconds. This problem arises because the proportion of accumulated error is too large, causing the UAV trajectory planning to overemphasize accumulated error, resulting in a reduced control rate and decreased control performance. Figure 6 The midpoint line representing the RLIC (r=0.8) UAV trajectory planning can stably control the UAV to the desired position in a short time. This UAV trajectory planning uses parameters to attenuate past errors, ensuring a balance between current errors and accumulated past errors. This allows the UAV trajectory planning to still learn a stable control strategy for controlling the UAV's attitude and position, while reducing adjustment time. Furthermore, compared to... Figure 4 Compared to the response curves in the previous examples, the control performance of UAV trajectory planning based on the conventional RL algorithm deteriorates significantly when the reward function worsens, and it even fails to achieve stable convergence. However, the two UAV trajectory planning methods based on the RL-IC algorithm are less affected. In particular, the response curves of the RL-IC UAV trajectory planning remain basically unchanged under the two reward function settings, indicating that the introduction of integral compensation can reduce the requirements of the learning algorithm on the reward function setting and improve the performance of the learning algorithm.

[0105] In the flight formation environment, four role networks were used (similar to four single UAV environments), and the formation state and formation reward were additionally considered. The RL-IC algorithm was adopted, which performed best in the single UAV environment. The formation's goal was to move from the starting point to the target point while maintaining a certain distance between the UAVs. A total of 10 networks were trained, including the A2C network for each UAV, as well as the target network and prediction network of RND. Figure 10 A 3D visualization of the UAV formation learning process (where black circles represent the air defense network, black solid lines represent the movement paths of UAVs, and blue solid lines represent the movement paths of missiles). Figure 10 (a) For the random maneuver 1 of the UAV formation during the learning process (external rewards reduce the possibility of encountering missiles, so it will either move in a curved path or evade them), Figure 10 (b) For the random maneuvering of the UAV formation during the learning process 2 (when external rewards decrease sharply, it may evade missiles if encountered), Figure 10 (c) For the UAV formation's random maneuver during the learning process 3 (external rewards decrease sharply, immediately leave the air defense network), Figure 10 (d) The UAV formation has been trained to find method 1 to fly to the destination (one aircraft flies straight towards the target, while the other three evade the missile). Figure 10 (e) Method 2 for the UAV formation to find its destination after training (all UAVs evade missiles). Figure 10 (f) The UAV formation has been trained to find a way to reach its destination (one aircraft flies in a curve towards the target, while the other three evade the missile in a straight line). Figure 10 (g)- Figure 10 (i) The optimal control strategies 1 (RL-IC), 2 (RL-IC), and 3 (RL-IC) for reaching the destination were learned for the UAV formation. Figure 10 It can be seen that, similar to the behavior in a single UAV environment, each UAV in a UAV formation initially exhibits random flight, and then gradually begins to move toward the target point.

[0106] The convergence process of traditional Q-learning algorithms (Q-learning algorithms) and improved Q-learning algorithms in simple environments is as follows: Figure 11 As shown. From Figure 11It can be observed that, firstly, the traditional Q-learning algorithm exhibits significant fluctuations in its learning curve during the initial iterations of finding the shortest path, indicating a low level of utilization of new knowledge and requiring multiple learning iterations to converge. Secondly, the lack of a suitable reward function leads to somewhat arbitrary action selection, prolonging the exploration time. The improved RL algorithm, having acquired prior knowledge, converges quickly and finds the shortest path in the early learning stages. Thirdly, due to the continuous reward function, the robot can instantly assess the quality of action states and select the optimal action, resulting in smaller path length fluctuations and higher convergence stability during the initial learning iterations before convergence.

[0107] This invention also models trajectory planning as a POMDP / MDP problem, randomly generating threats and targets through an environment simulator to train an agent independent of a fixed environment. In the RL-IC online planning phase, a population optimization method is used to select the policy with the maximum cumulative reward as the execution policy. Continuous guidance is provided through continuous heuristic rewards, avoiding the blindness of traditional discrete rewards. This invention can be extended to multi-UAV formation planning. In a formation environment, each UAV is independently trained through a role network and shares formation state and rewards, supporting both standalone and formation applications. This invention reduces the sensitivity to reward function settings through an integral compensation mechanism.

[0108] Another embodiment of the present invention provides an unmanned aerial vehicle trajectory planning system based on integral compensation, comprising:

[0109] The environment simulator module is used to build an environment simulator that simulates the flight environment of unmanned aerial vehicles. The environment simulator randomly generates threat and target locations each time it is reset.

[0110] The reinforcement learning training module is used to train the agent in the environment simulator using reinforcement learning, model the trajectory planning problem as a partially observable Markov decision process or a fully observable Markov decision process, and obtain the baseline policy for trajectory planning.

[0111] The compensation and filtering module is used to correct the state error of the agent in real time based on the integral compensation mechanism, and to smooth the action commands output by the agent using the action filter.

[0112] The integral compensation planning module is used to optimize the strategy through the integral compensation reinforcement learning planning module. The planning module uses a population optimization method to select the strategy with the maximum cumulative reward as the execution strategy and interacts with the environment simulator to generate the final trajectory.

[0113] The environment simulator module further includes a sub-simulator unit for predicting future states; the environment simulator module sets a random initial position each time it is reset to ensure that the planning strategy does not depend on a fixed environment, and the pose smoothness is guaranteed by the action filtering module.

[0114] In this embodiment, the unmanned aerial vehicle trajectory planning system also includes a reward function module, which designs a continuous heuristic reward function, including a basic reward, a distance reward, a heading reward, a target point reward, and a required overload reward.

[0115] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for unmanned aerial vehicle trajectory planning based on integral compensation, characterized in that, Includes the following steps: An environment simulator is constructed to simulate the flight environment of an unmanned aerial vehicle. The environment simulator randomly generates threat and target locations each time it is reset. The agent is trained in the environment simulator using reinforcement learning, and the trajectory planning problem is modeled as a partially observable Markov decision process or a fully observable Markov decision process to obtain the baseline strategy for trajectory planning. The state error of the agent is corrected in real time based on the integral compensation mechanism, and the action command output by the agent is smoothed by the action filter. The strategy is optimized through an integral compensation reinforcement learning planning module. The planning module uses a population optimization method to select the strategy with the largest cumulative reward as the execution strategy and interacts with the environment simulator to generate the final trajectory.

2. The unmanned aerial vehicle trajectory planning method based on integral compensation as described in claim 1, characterized in that, The environment simulator is used to simulate static threats, dynamic threats, target points, and aircraft dynamics models in the flight environment of unmanned aerial vehicles.

3. The unmanned aerial vehicle trajectory planning method based on integral compensation as described in claim 2, characterized in that, In the integral compensation mechanism, the attenuation factor λ is 0.8 and the compensation coefficient β is 0.

3. The state error after integral compensation replaces the original state error without increasing the dimension of the state space. This is used to eliminate steady position error and reduce the position tracking error to zero.

4. The unmanned aerial vehicle trajectory planning method based on integral compensation as described in claim 3, characterized in that, The integral compensation state error in the integral compensation mechanism is defined as follows: ； In the formula, for The error of the original state at time t. for The state error after compensation at time step. For compensation coefficient, As the attenuation factor, Weights used to adjust for past errors; The contribution of the old error decays exponentially over time; From time step 1 to The weighted cumulative state error represents the sum of all past errors, with the weight of past errors decreasing as the time distance from the present increases.

5. The unmanned aerial vehicle trajectory planning method based on integral compensation as described in claim 1, characterized in that, The motion filter includes a momentum filter and an interpolation filter; the momentum filter is used to calculate the local average of the motion differences using an exponentially weighted average, and the calculation formula for the momentum filter is as follows: ； ； In the formula, To simulate steps The exponentially weighted average of the action differences; These are the weighting coefficients used to control the weights of historical values; This is a weighted average of the previous simulation steps; For intelligent agents at the decision-making moment The original action output; The action actually passed to the simulator in the previous simulation step; For the clipping function, limit the difference to... Within the range; The smoothed motion; The motion update rate is used to control the magnitude of changes in smooth motion. The interpolation filter is used to generate action transition functions through Hermite interpolation, and the calculation formula for the interpolation filter is as follows: ； In the formula, In order to make decisions Smoothed motion; This is a smoothing action from the previous decision-making moment; For the agent at the current decision moment The original action output; These are the weighting coefficients used to balance the old and new actions.

6. The unmanned aerial vehicle trajectory planning method based on integral compensation as described in claim 1, characterized in that, When selecting the strategy with the largest cumulative reward as the execution strategy, the cumulative reward is a continuous heuristic reward function, which is calculated as follows: ； In the formula, This is the total reward value based on the Euclidean distance between the current coordinates and the target point; Basic rewards; This is a weighting factor used to adjust the intensity of heuristic rewards; The distance is Euclidean. and These are weight parameters; The reward is a fixed distance threshold; when an obstacle is encountered, the reward is negative infinity; when the destination is reached, the reward is 100; and in other situations, the reward is 0.

7. The unmanned aerial vehicle trajectory planning method based on integral compensation as described in claim 6, characterized in that, The optimization process of the integral compensation reinforcement learning planning module includes: Set the population size P=100, the number of elite individuals E=10, the planning length H=250 steps, and the number of iterations K=10; The cumulative reward J(θ) is calculated by predicting future states using a sub-environment simulator. Update the strategy parameters using a soft update method; If the integral compensation optimization fails, revert to the baseline strategy; The planning module supports multi-UAV formation planning and achieves collaborative trajectory generation by introducing formation status and rewards.

8. The unmanned aerial vehicle trajectory planning method based on integral compensation as described in claim 7, characterized in that, The network structure of the agent is a policy-evaluation architecture. The policy network contains two hidden layers, each with 128 neurons and ReLU activation function. The output layer uses the tank activation function to limit the range of control signals. The evaluation network inputs include state and action. The hidden layer structure is the same as the policy network. The output layer uses a linear activation function to estimate the Q value. The network parameters are updated through gradient pruning and priority experience replay.

9. A trajectory planning system for unmanned aerial vehicles based on integral compensation, characterized in that, The step of performing the unmanned aerial vehicle trajectory planning method based on integral compensation as described in any one of claims 1-8, wherein the unmanned aerial vehicle trajectory planning system comprises: The environment simulator module is used to build an environment simulator that simulates the flight environment of unmanned aerial vehicles. The environment simulator randomly generates threat and target locations each time it is reset. The reinforcement learning training module is used to train the agent in the environment simulator using reinforcement learning, model the trajectory planning problem as a partially observable Markov decision process or a fully observable Markov decision process, and obtain the baseline policy for trajectory planning. The compensation and filtering module is used to correct the state error of the agent in real time based on the integral compensation mechanism, and to smooth the action commands output by the agent using the action filter. The integral compensation planning module is used to optimize the strategy through the integral compensation reinforcement learning planning module. The planning module uses a population optimization method to select the strategy with the maximum cumulative reward as the execution strategy and interacts with the environment simulator to generate the final trajectory.

10. The unmanned aerial vehicle trajectory planning system based on integral compensation as described in claim 9, characterized in that, The environment simulator module also includes a sub-simulator unit for predicting future states; the environment simulator module sets a random initial position each time it is reset to ensure that the planning strategy does not depend on a fixed environment, and ensures the smoothness of the pose points through the action filtering module.

Images