An aircraft real-time trajectory generation method based on enhanced PPO algorithm

Abstract

The application discloses a kind of aircraft real-time trajectory generation methods based on enhanced PPO algorithm, belong to aircraft trajectory planning technical field, method includes: S1, construct reinforcement learning simulation environment based on aircraft full state dynamics model;S2, model training is carried out by enhanced PPO algorithm, its training process includes: to sampling trajectory data is filtered and enhanced processing;According to target pose, trajectory is grouped and in-group advantage function normalization is carried out;And according to task success rate self-adaptive adjustment learning rate;S3, real-time trajectory is generated using the policy network trained;The application solves the problem that the maneuvering performance is not fully played, the trajectory is not real due to the simplification model of traditional trajectory generation method, and the problem that standard reinforcement learning algorithm is unstable in complex task training, and under the premise of using full state dynamics model, the efficiency, stability and authenticity of trajectory generation are significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of aircraft trajectory planning technology, and more specifically, relates to a method for generating real-time aircraft trajectories based on an enhanced PPO algorithm. Background Technology

[0002] With the development of autonomous control technology for aircraft, in many complex mission scenarios, such as emergency obstacle avoidance, rapid path switching in aerial mapping, or performing precision flight demonstrations, aircraft need to complete large-scale maneuvers in a very short time and accurately reach a preset target pose (i.e., position and attitude). To ensure mission success and flight safety, it is crucial for aircraft to achieve target pose control quickly and accurately at the end of maneuvering flight.

[0003] Traditional trajectory generation methods typically employ simplified aircraft dynamics models, which fails to fully utilize the aircraft's maneuverability, resulting in trajectories that are insufficient in terms of realism and feasibility. If a full-state dynamics model is used, traditional optimization algorithms face the problem of excessively high dimensionality of optimization variables, leading to a sharp decline in computational efficiency and making it difficult to meet real-time requirements. Therefore, how to efficiently generate real-time trajectories that meet the needs of complex maneuvering missions while ensuring trajectory realism and feasibility is a pressing technical challenge in the field. Summary of the Invention

[0004] To address the above shortcomings, this invention proposes a real-time trajectory generation method for aircraft based on an enhanced PPO algorithm, aiming to fully utilize the aircraft's comprehensive maneuverability, improve the realism and feasibility of trajectory generation, and increase the efficiency of trajectory generation for rapid aircraft positioning. The method includes the following steps:

[0005] S1. Based on the full-state dynamics model of a fixed-wing aircraft, construct a simulation environment for reinforcement learning training:

[0006] S1-1. Design of the state space:

[0007] Based on the full-state dynamics model of the fixed-wing aircraft, 13 state variables at the current moment are required for numerical simulation. These represent the aircraft's airspeed, angle of attack, sideslip angle, roll angle, pitch angle, yaw angle, roll rate, pitch rate, yaw rate, x-axis position, y-axis position, z-axis position in the ground coordinate system, and power, respectively. To determine the distance to the target, the aircraft needs to determine... ,in , To represent the x-direction position of the target point, and to ensure the uniqueness of the state transition during the Markov decision process, we include the current time t of the spacecraft and the position of the target pose. , representing the target point's position in the x-direction, y-direction, and z-direction, as well as the target's pitch and yaw angles, respectively. These are used in numerical simulations of aircraft. It only served as an initial value and did not appear on the right-hand side of the dynamic equations, therefore this state was deleted, hence the state space... It contains 22 state variables, denoted as ;

[0008] S1-2, Design of the motion space:

[0009] The operational space of the aircraft is , 'a' represents the action. , respectively representing the aircraft's throttle control amount and the deflection angles of the elevator, aileron, and rudder;

[0010] S1-3, Design of the reward function:

[0011] The rapid positioning mission of the aircraft aims to generate the fastest trajectory from the current position to the designated position. In order to ensure the theoretical optimality of the trajectory, the reward is only given according to the state at the end of the trajectory. No subjective guidance reward is given in the middle process. The process includes three types of rewards, including non-arrival process reward, over-boundary reward and arrival reward.

[0012] The "not-reached process reward" refers to the reward given to the aircraft during the rapid positioning process based on the terminal state of its trajectory, guiding the aircraft to reach the target pose more quickly. The reward consists of two parts, one of which is to guide the aircraft to approach... Another part of the reward is to guide the aircraft to reach its destination faster. ;

[0013] That is, the non-reaching process reward is given when the trajectory is unsuccessful and does not exceed the limit. Its value is designed as a function related to the distance of the closest point on the trajectory (the point in the trajectory closest to the target) from the target and the time taken, which aims to guide the policy network to generate trajectories that are closer to the target and take less time.

[0014] The over-limit reward is designed to satisfy the performance constraints of fixed-wing aircraft. When the constraints are violated, the numerical simulation process of the environment is terminated. If there is a constraint violation in a trajectory, there will be two types of rewards:

[0015] Firstly, a negative reward is given at the constraint violation point (the point that exceeds the limit); secondly, if the time interval between the nearest point and the point that exceeds the limit is... If the value exceeds a set constant C, a positive reward is given to the nearest point.

[0016] When the condition is met When the aircraft reaches the designated target pose, the simulation process of the environment is terminated, which is the reward. At this time, the design of the reward function mainly considers two aspects: one is to guide the aircraft to get closer to the target pose, and the other is to shorten the time for the aircraft to reach the target pose.

[0017] S2. Based on the simulation environment constructed in step S1, sampling is performed in the training simulation environment. The EHPPO algorithm is used to learn from the sampled data. The EHPPO algorithm is based on the PPO algorithm, and a discount factor is set. , Furthermore, a trajectory reset mechanism, a grouping normalization advantage function mechanism, and an adaptive learning rate mechanism were introduced. After establishing the policy network and the value network, the policy network and the value network were trained by the enhanced PPO algorithm to obtain the optimized value network and policy network.

[0018] The training process of the enhanced PPO algorithm includes:

[0019] S2-1. An experience pool is set up to store trajectories sampled in the simulation environment. The experience pool has a fixed size and can only store a limited number of trajectories. In this embodiment, the experience pool size is set to 200. Each trajectory in the experience pool contains multiple trajectory points. Each trajectory point mainly contains information such as its state, the action to be performed, and the reward obtained. It should be noted that after the enhanced PPO algorithm is trained based on the sampled data in the experience pool, the experience pool data will be resampled and the experience pool will be updated.

[0020] S2-2. Introduce a trajectory reset mechanism to filter and enhance trajectory data that terminates prematurely due to invalidity or success in a single sampling iteration according to preset rules, so as to improve the utilization efficiency of empirical data.

[0021] S2-3. Introduce a grouping normalization advantage function mechanism to group the trajectories according to the difficulty of the target pose in the sampled trajectory, and perform independent normalization processing on the advantage function in each group to balance the learning weight of tasks with different difficulty.

[0022] S2-4. An adaptive learning rate mechanism is introduced to dynamically adjust the learning rate used to update the policy network and the value network based on the ratio of the trajectories that successfully reach the target pose in the sampled data to the total trajectories, i.e. the task success rate.

[0023] S3. Based on the initial state of the aircraft, control actions are generated using the optimized policy network trained in step S2, and the real-time trajectory of the aircraft is obtained by iteratively integrating the control actions.

[0024] Furthermore, in steps S1-2, the values ​​of each state variable are continuous, and the state space of this task is a continuous state space.

[0025] The state variables of the aircraft are normalized according to the following formula, and the range of values ​​in the state space is (0,1):

[0026] (1)

[0027] Where 's' refers to Each state in the process, Represents the minimum value of the state. This represents the maximum value of the state.

[0028] Furthermore, in steps S1-2, the values ​​of the aircraft throttle control quantity and the deflection angles of the elevator, ailerons, and rudder are all continuous values, and the action space is a continuous action space. In the reinforcement learning algorithm, the policy network's output for the continuous action space is... Random values ​​under a distribution, where, , Therefore, the output values ​​of the policy network mostly fall within the range of [-3, 3], as... As the value gradually decreases, the output of the policy network can be considered to be in the range of [-1, 1]. To improve the convergence efficiency of the policy network, the output of the policy network to the environment undergoes the transformation shown in the following formula:

[0029] (2)

[0030] in, The action value input into the environment, its maximum value is The minimum value is 'a' represents the output action value of the policy network, and the clip function ensures that the transformed action value is within its constraints.

[0031] Furthermore, among the three types of rewards in steps 1-3, the unreached process reward and the over-boundary reward can coexist, while the other rewards cannot coexist.

[0032] Furthermore, the reward function in steps 1-3 is designed as follows:

[0033] When the condition is met hour,

[0034] (3)

[0035] in, This indicates points within a certain range around the specified target pose. Additionally...

[0036] (4)

[0037] in, , Indicates the specified target pose. This represents the pose at each moment in the trajectory. The initial pose is represented by d, and the shortest distance to the specified target pose is represented by d along the entire trajectory.

[0038] When the condition is met hour,

[0039] (5)

[0040] in, The reward value is calculated according to equation (3). This indicates points within a certain range around the specified target pose, and The same applies to the other four variables. It is a constant value. It is the shortest time to reach the point with the shortest distance to the specified target pose, based on the estimated value. The longest simulation time allowed for the trajectory is set to twice the shortest estimated time to reach the specified target pose;

[0041] also, (6)

[0042] Where t is the time step corresponding to the nearest point, and in addition:

[0043] (7)

[0044] in, The coefficient is constant.

[0045] The nearest point reward function is the same as that in equations (3) and (5), and the calculation method of the supercritical point negative reward function is as follows:

[0046] (8)

[0047] in, The pose of the superboundary point;

[0048] Reaching a reward means that when conditions are met... When the aircraft reaches the designated target pose, the simulation process terminates. At this point, the reward function design mainly considers two aspects: firstly, guiding the aircraft closer to the target pose; and secondly, shortening the time it takes for the aircraft to reach the target pose. The specific reward function design is shown in the following formula:

[0049] (9)

[0050] in, Slightly larger than the maximum reward calculated according to formula (3), where R represents the time reward. The calculation method for R is the same as that for formula (5), except that R is not included in the calculation. ,also, This represents the minimum estimated time from the initial pose to the target pose. Since the time it takes for the aircraft to reach the target pose varies depending on the target pose, if the minimum estimated time is the same, the reward is higher for trajectories closer to the target pose. Therefore, it is necessary to align the time rewards for different target poses. The trajectory length for different target poses is estimated using the Dubins curve. At this point, the minimum turning radius constraint of the aircraft is not considered. The shortest arrival time is obtained for a fixed target pose by learning. Assuming that the aircraft is in uniformly accelerated motion, the aircraft acceleration is calculated. Based on the estimated trajectory length, estimated acceleration, and initial velocity of the aircraft, the minimum estimated time for different target poses is obtained. The calculation formula is as follows:

[0051] (10)

[0052] in The initial velocity is... To estimate acceleration, The trajectory length is estimated based on the Dubins curve.

[0053] As a coefficient characterizing the relationship between the terminal pose of the aircraft trajectory and the target pose, its calculation method is as follows:

[0054] (11)

[0055] in, The constant coefficient, , As can be seen from equation (11), the smaller the distance to the target pose, The larger the value, the more important it is to ensure the stability of the algorithm. The derivative first increases and then decreases as the target pose approaches, where,

[0056] (12)

[0057] Furthermore, the sampling trajectories that need to be reset in the trajectory reset mechanism are of two types:

[0058] Firstly, within a single thread, the time it takes for the spacecraft to reach the target pose, violate constraints, or reach... The time is close When the resampled trajectory is shorter, the calculated terminal reward is inaccurate. Therefore, the thread's trajectory needs to be reset, and the reward for the shorter resampled trajectory should be set to 0.

[0059] Secondly, assume the trajectory of thread i is as follows:

[0060] , ,in This indicates the trajectory status, such as not reaching the target pose or exceeding the boundary. Indicates the state at the next moment. Indicates the action currently being taken. This represents the reward obtained by taking the current action in the current state. Express the expected return at time point. This represents the probability of taking the current action in the current state. Represents the value of the state at the next moment, when If a trajectory exists that leads to the target pose, record the trajectory. Clear , trajectory Repeat filling the entire .

[0061] Furthermore, in steps S2-3, the sampled trajectories are grouped according to the difficulty of reaching the target pose, as shown in the following formula:

[0062] (13)

[0063] The division of intervals is not necessarily uniform; it is mainly based on the difficulty of reaching the target pose. Intervals for different target states are arranged and combined, and for different combinations of target poses, [further steps are taken]. Grouping normalization, the final The calculation method is shown in the following formula:

[0064] (14)

[0065] in, Indicates the same combination of objectives The mean, Indicates the same combination The variance.

[0066] Furthermore, the determination of the adaptive learning rate mechanism includes the following steps:

[0067] S2-4-1. Calculate the policy network loss function, the formula of which is shown below:

[0068] (15)

[0069] Where I represents the total number of threads, T max This represents the maximum step size for each thread. Indicates the policy network parameters, This is the truncation value. The policy entropy is calculated using KL divergence, where c is a constant coefficient. , This represents the result calculated by the current policy network. Take in state The probability value, This represents the probability value calculated by the policy network based on the sampled data;

[0070] S2-4-2. Calculate the loss function of the value network. The calculation formula is shown below:

[0071] (16)

[0072] in, Indicates the cutoff value. Represents the parameters of the value network. Let represent the expected reward at step t for the i-th trajectory;

[0073] S2-4-3, The adaptive learning rate is shown below:

[0074] (17)

[0075] in, The initial learning rate, , , , , , , , It is a constant. This represents the success rate of the aircraft reaching the target pose, that is, the ratio of the trajectory reaching the target pose to the total trajectory in the sampled data. As... As the learning rate of the algorithm increases, the learning rate gradually decreases, and the rate of decrease follows the pattern in formula (17).

[0076] Compared with the prior art, the present invention has the following advantages:

[0077] 1. By designing the state space, action space, and reward function for the full-state dynamic model of the fixed-wing aircraft in step S1, the problems of insufficient utilization of the aircraft's maneuverability and unrealistic or infeasible generated trajectories caused by the simplified model used in traditional trajectory generation methods can be effectively solved.

[0078] 2. In step S2, the real-time trajectory for rapid aircraft positioning was generated based on the EHPPO algorithm, which greatly improved the operational efficiency compared to traditional trajectory generation methods.

[0079] 3. By introducing a trajectory reset mechanism in step S2, the data utilization rate of reinforcement learning is improved. By using a group-normalized advantage function mechanism, the problem of preference learning caused by the different difficulty levels of different tasks in traditional near-end policy optimization methods is solved. An adaptive learning rate is designed based on the success rate of the aircraft reaching the designated target position, which solves the oscillating learning problem caused by suboptimal trajectories and greatly improves the stability of the algorithm. Attached Figure Description

[0080] Figure 1 This is a flowchart illustrating the workflow of the present invention.

[0081] Figure 2 This is a schematic diagram of the over-boundary reward setting in the embodiment.

[0082] Figure 3 This is a schematic diagram illustrating the reason for setting the grouping normalization advantage function mechanism in the embodiment. Detailed Implementation

[0083] To facilitate understanding of the present invention, the apparatus of the present invention will now be described more fully with reference to the accompanying drawings. Embodiments of the apparatus are shown in the drawings. However, the apparatus can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to make the disclosure of the present invention more thorough and complete.

[0084] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "configuration" should be interpreted broadly. For example, they can refer to a fixed connection or configuration, a detachable connection or configuration, or an integral connection or configuration. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0085] Example

[0086] like Figure 1 As shown, this embodiment provides the full-state dynamic equations for fixed-wing aircraft, fully leveraging its maneuverability, ensuring the realism and feasibility of trajectory generation, while improving the efficiency of the trajectory generation algorithm to achieve real-time trajectory generation, including the following steps:

[0087] Step S1: Establish a reinforcement learning training simulation environment for the rapid positioning task of the aircraft.

[0088] The specific steps are as follows:

[0089] S1-1, Design State Space:

[0090] Based on the full-state dynamics model of the fixed-wing aircraft, 13 state variables at the current moment are required for numerical simulation. These represent the aircraft's airspeed, angle of attack, sideslip angle, roll angle, pitch angle, yaw angle, roll rate, pitch rate, yaw rate, x-axis position, y-axis position, z-axis position in the ground coordinate system, and power, respectively. To determine the distance to the target, the aircraft needs to determine... ,in , This indicates the x-direction position of the target point; similarly, other variables can be determined.

[0091] To ensure the uniqueness of the reward function design and the uniqueness of state transitions during the Markov decision process, the current time t of the aircraft and the position of the target pose are included. , representing the target point's position in the x-direction, y-direction, and z-direction, as well as the target's pitch and yaw angles, respectively. These are used in numerical simulations of the aircraft. It only served as an initial value and did not appear on the right side of the dynamic equation, so this state was deleted.

[0092] The state space of the aircraft's rapid positioning mission at this time It contains 22 state variables, namely:

[0093] Each state variable takes a continuous value, and the state space of this task is a continuous state space.

[0094] For neural network training, input normalization is performed to solve the problem of inconsistent input units, which can accelerate the convergence efficiency of the neural network and thus improve the convergence efficiency of the enhanced near-end policy optimization method. Therefore, the state variables of the aircraft are normalized, as shown in equation (1). At this time, the value range of the state space of the task is (0,1):

[0095] (1)

[0096] Among them, the minimum value of the state Set to:

[0097] ;

[0098] Maximum value of state Set to:

[0099] .

[0100] S1-2, Designing the Action Space:

[0101] The operational space of the aircraft is , 'a' represents the action. , representing the aircraft's throttle control input and the deflection angles of the elevator, ailerons, and rudder, respectively. Their values ​​are all continuous. The mission aircraft's action space is a continuous action space. In the reinforcement learning algorithm, the policy network's output for the continuous action space is... Random values ​​selected under a distribution. In this embodiment, , Therefore, the output values ​​of the policy network mostly fall within the range of [-3, 3], as... As the value gradually decreases, the output of the policy network can be considered to be within the range of [-1, 1]. To improve the convergence efficiency of the policy network, the policy network output is transformed as shown in equation (2) when it is sent to the environment:

[0102] (2)

[0103] Among them, the maximum value Set as minimum value Set as .

[0104] S1-3, Design the reward function:

[0105] The rapid positioning task for aircraft aims to generate the fastest possible trajectory from the current position to a designated position. To ensure the theoretical optimality of the trajectory, rewards are assigned only based on the state at the end of the trajectory; no subjective guidance rewards are assigned during the intermediate process. This process includes three types of rewards: non-arrival process reward, out-of-bounds reward, and arrival reward. It should be noted that the non-arrival process reward and out-of-bounds reward may coexist, while the other rewards will not coexist, as detailed below:

[0106] 1) Rewards for not reaching the destination:

[0107] The "not-reached process reward" refers to the reward given to the aircraft during the rapid positioning process based on the terminal state of its trajectory, guiding the aircraft to reach the target pose more quickly. The reward consists of two parts, one of which is to guide the aircraft to approach... Another part of the reward is to guide the aircraft to reach its destination faster. The reward function is designed as shown in equation (3).

[0108] When the condition is met hour;

[0109] (3)

[0110] in, This indicates a range of points near the specified target pose, set to... The other four variables are set similarly. In addition,

[0111] (4)

[0112] in, , Indicates the specified target pose. This represents the pose at each moment in the trajectory. The initial pose is represented by d, and the shortest distance to the specified target pose is represented by d along the entire trajectory.

[0113] When the condition is met hour,

[0114] (5)

[0115] in, The reward value is calculated according to equation (3). This indicates a range of points near the target pose, set to (200ft, 200ft, 200ft, 0.15rad, 0.15rad). It is a constant value, set to 60 time steps. It is the shortest time to reach the point with the shortest distance to the specified target pose, based on the estimated value, and this value is relatively conservative. The longest simulation time allowed for the trajectory is set to twice the shortest estimated time to reach the specified target pose.

[0116] also, (6)

[0117] Where t is the time step corresponding to the nearest point, and in addition:

[0118] (7)

[0119] in, This is a constant coefficient, set to 1000.

[0120] 2) Transcendental Rewards:

[0121] To meet the performance constraints of fixed-wing aircraft, a super-boundary reward is implemented. When a constraint is violated, the numerical simulation of the environment is terminated. If a constraint violation occurs within a trajectory, two types of rewards will apply:

[0122] One approach is to provide a negative reward at the point of constraint violation (the point of exceeding the limit);

[0123] Secondly, if the time interval between the nearest point and the superboundary node is... When the time step exceeds the set constant C=20, a positive reward is given to the nearest point. The nearest point reward function is the same as that in equations (3) and (5).

[0124] The calculation method of the negative reward function at the supercritical point is shown in equation (8):

[0125] (8)

[0126] in, The pose of the superboundary point, and the schematic diagram of the superboundary reward are as follows. Figure 2 As shown.

[0127] 3) Rewards for reaching the destination:

[0128] When the condition is met When the aircraft reaches the designated target pose, the simulation process of the environment is terminated, which is the arrival reward. At this time, the design of the reward function mainly considers two aspects: one is to guide the aircraft to get closer to the target pose, and the other is to shorten the time for the aircraft to reach the target pose. The specific reward function design is shown in equation (9):

[0129] (9)

[0130] in, It is slightly larger than the maximum reward calculated according to equation (3). R represents the time reward, and the calculation method of R is the same as that of equation (5). The difference is that R is not included in the calculation. ,also, This represents the minimum estimated time from the initial pose to the target pose. The time it takes for the aircraft to reach the target pose differs depending on the target pose. If the minimum estimated time is the same, the reward is higher for trajectories closer to the target pose. Therefore, it is necessary to align the time rewards for different target poses. The trajectory length for different target poses is estimated using the Dubins curve, without considering the minimum turning radius constraint of the aircraft. The shortest arrival time is obtained by learning for a fixed target pose. Assuming the aircraft is undergoing uniform acceleration, the aircraft acceleration is calculated. Based on the estimated trajectory length, estimated acceleration, and initial velocity of the aircraft, the minimum estimated time for different target poses is obtained, and its calculation formula is shown in equation (10).

[0131] (10)

[0132] in The initial velocity is... To estimate acceleration, it is set to 16.25 ft / s². The trajectory length is estimated based on the Dubins curve;

[0133] As a coefficient characterizing the relationship between the terminal pose of the aircraft trajectory and the target pose, its calculation method is shown in Equation (11):

[0134] (11)

[0135] in, As a constant coefficient, set to 1000. , As can be seen from the above formula, the smaller the distance to the target pose, the better. The larger the value, the more important it is to ensure the stability of the algorithm. The derivative first increases and then decreases as the target pose approaches, where:

[0136] (12)

[0137] S2. Sampling is performed in the training simulation environment. The EHPPO algorithm is used to learn from the sampled data. The EHPPO algorithm is based on the PPO algorithm, and a discount factor is set. , Furthermore, a trajectory reset mechanism, a grouped normalized advantage function mechanism, and an adaptive learning rate mechanism were introduced. After establishing the policy network and the value network, the policy network and the value network were trained using the enhanced PPO algorithm to obtain the optimized policy network.

[0138] The training process of the enhanced PPO algorithm includes:

[0139] S2-1. An experience pool is set up to store trajectories sampled in the simulation environment. The experience pool has a fixed size and can only store a limited number of trajectories; in this embodiment, the experience pool size is set to 200. Each trajectory in the experience pool contains multiple trajectory points. Each trajectory point mainly contains information such as its state, the action to be performed, and the reward obtained. It should be noted that after the enhanced PPO algorithm is trained based on the sampled data in the experience pool, the experience pool data will be resampled and updated.

[0140] S2-2. Introduce a trajectory reset mechanism to filter and enhance trajectory data that terminates prematurely due to invalidity or success in a single sampling iteration according to preset rules, so as to improve the utilization efficiency of empirical data.

[0141] Specifically, during sampling in the training simulation environment, a multi-threaded sampling method is adopted to increase sample sampling efficiency. The algorithm trains by sampling samples through multiple threads in each round. Since each thread samples at a different target pose, the time required for sampling varies, resulting in different maximum sampling times for different target poses. In more than Simulation sampling will restart at this time. Each thread is set to a maximum step size of [value missing]. This value is a constant; during sampling, the aircraft may violate constraints, reach the target pose, or exceed the limits. At this time, the simulation will restart and reset the following two sampling trajectories:

[0142] 1) In a single thread, the time it takes for the spacecraft to reach the target pose, violate constraints, or reach... The time is close When the resampled trajectory is shorter, the calculated terminal reward is inaccurate. Therefore, the thread's trajectory needs to be reset, and the reward for the shorter resampled trajectory is set to 0.

[0143] 2) Assume the trajectory of thread i is:

[0144] , ,in This indicates the trajectory status, such as not reaching the target pose or exceeding the boundary. Indicates the state at the next moment. Indicates the action currently being taken. This represents the reward obtained by taking the current action in the current state. Express the expected return at time point. This represents the probability of taking the current action in the current state. Represents the value of the state at the next moment, when If a trajectory exists that leads to the target pose, record the trajectory. Clear , trajectory Repeat filling the entire .

[0145] It should be noted that, in practice, the two processing methods 1) and 2) mentioned above can be used individually or in combination, together forming the trajectory reset mechanism.

[0146] S2-3. A grouped normalized advantage function mechanism is introduced to group trajectories according to the difficulty of the target pose in the sampled trajectories, and to perform independent normalization of the advantage function within each group, so as to balance the learning weights of tasks with different difficulties, such as... Figure 3 As shown;

[0147] S2-3-1. For the rapid positioning task of the aircraft, in order to ensure theoretical optimality, only the terminal reward is set. Since the importance of all points in the trajectory is the same, a discount factor is set. , The dominance function is calculated based on the sampled data, and its calculation method is shown in equation (18):

[0148] (18)

[0149] Where T is the time step of the trajectory, t is the current step, and i is the thread number. This represents the reward value after the current step performs the action. This indicates the value of the current step.

[0150] S2-3-2. Group the sampling trajectories according to the difficulty of reaching the target pose, as shown in the following formula (19):

[0151] (19)

[0152] S2-3-3, Arrange and combine the intervals of different target states, and for different target pose combinations, perform... Grouping normalization, the final The calculation method is shown in the following formula (14):

[0153] (14)

[0154] in, Indicates the same combination of objectives The mean, Indicates the same combination The variance.

[0155] S2-4. An adaptive learning rate mechanism is introduced to dynamically adjust the learning rate used to update the policy network and the value network based on the ratio of the trajectories that successfully reach the target pose in the sampled data to the total trajectories, i.e. the task success rate.

[0156] Determining the adaptive learning rate mechanism includes the following steps:

[0157] S2-4-1. Calculate the policy network loss function, the formula of which is shown below:

[0158] (15)

[0159] Where I=1000 represents the total number of threads, T max This represents the maximum step size for each thread. Indicates the policy network parameters, This is the truncation value. The policy entropy, obtained through KL divergence calculation, is c=0.01. , This represents the result calculated by the current policy network. Take in state The probability value, This represents the probability value calculated by the policy network based on the sampled data;

[0160] S2-4-2. Calculate the loss function of the value network. The calculation formula is shown below:

[0161] (16)

[0162] in, Indicates the cutoff value. Represents the parameters of the value network. Let represent the expected reward at step t for the i-th trajectory;

[0163] S2-4-3, The adaptive learning rate is shown below:

[0164] (17)

[0165] in, The initial learning rate, , , , , , . This represents the success rate of the drone reaching the target pose, i.e., the ratio of the trajectory reaching the target pose to the total trajectory in the sampled data. As... As the learning rate increases, the learning rate of the algorithm gradually decreases, and the rate of decrease follows the pattern in formula (18).

[0166] S3. Using the learning rate in step S2, optimize the parameters of the policy network and the value network with the Adam optimizer. Finally, generate control actions based on the obtained policy network model. Obtain the next state by integrating the initial state and the control actions. Repeat this process to obtain the final generated trajectory.

[0167] In summary, the real-time trajectory generation method for rapid aircraft positioning based on reinforcement learning described in this embodiment considers the full-state dynamics model of the fixed-wing aircraft when establishing the training simulation environment, thus solving the problems of insufficient maneuverability and unrealistic or infeasible generated trajectories caused by the simplification of the fixed-wing aircraft model in existing methods. By introducing a trajectory reset mechanism, a grouping normalization advantage function mechanism, and an adaptive learning rate mechanism to form the EHPPO algorithm from the PPO algorithm, it solves the problems of low data utilization, biased learning, and unstable algorithm learning that are prone to occur when reinforcement learning is used to deal with rapid aircraft positioning tasks. Based on the EHPPO algorithm, learning is performed on the sampled data in the training simulation environment, which solves the problem of low solution efficiency of trajectory generation for the full-state dynamics model of fixed-wing aircraft in existing methods.

[0168] It should be noted that the structure described in this invention can be implemented in many different forms and is not limited to the embodiments described. Any equivalent transformations made by those skilled in the art based on the description and drawings of this invention, or direct or indirect applications in other related technical fields, such as the loading and unloading of other items, are included within the protection scope of this invention.

Claims

1. A method for generating real-time trajectory of an aircraft based on an enhanced PPO algorithm, characterized in that, Includes the following steps: S1. Based on the full-state dynamics model of a fixed-wing aircraft, a simulation environment for reinforcement learning training is constructed. The simulation environment includes a state space, an action space, and a reward function. The reward function includes a non-arrival process reward, an over-boundary reward, and an arrival reward. S2. Based on the simulation environment constructed in step S1, the policy network and value network are trained using the enhanced PPO algorithm to obtain the optimized value network and policy network. The training process of the enhanced PPO algorithm includes: S2-1. Set up the experience pool for storing sampled trajectories in the simulation environment; S2-2. Trajectory reset processing, which involves filtering and enhancing trajectory data that terminates prematurely due to invalidity or success in a single sampling iteration, includes: When a trajectory in a sampling thread becomes too short due to premature termination, the reward for that trajectory is set to zero. When a trajectory successfully reaches the target pose exists in the sampling thread, the entire thread is populated with the data from that successful trajectory. S2-3. Group the trajectories according to the difficulty of the target pose in the sampled trajectory, and perform independent normalization processing on the dominance function within each group; S2-4. Based on an initial learning rate, and according to the ratio of the trajectories that successfully reach the target pose in the sampled data to the total trajectories, adjust the learning rate according to the exponential decay function. The learning rate decreases as the task success rate increases; S3. Based on the initial state of the aircraft, control actions are generated using the optimized policy network trained in step S2, and the real-time trajectory of the aircraft is obtained by iteratively integrating the control actions.

2. The method for generating a real-time trajectory of an aircraft based on the enhanced PPO algorithm as described in claim 1, characterized in that: The state space in step S1 is a 22-dimensional continuous vector, which includes the current state of the aircraft, the difference between the state of the aircraft and the target, and the position of the target pose.

3. The method for generating a real-time trajectory of an aircraft based on the enhanced PPO algorithm as described in claim 1, characterized in that: The action space in step S1 is a 4-dimensional continuous vector, including the aircraft throttle control amount, elevator deflection angle, aileron deflection angle, and rudder deflection angle.

4. The method for generating a real-time trajectory of an aircraft based on the enhanced PPO algorithm as described in claim 1, characterized in that: The reward function in step S1 is a sparse terminal reward function, which is assigned only to the terminal state of the trajectory.

5. The method for generating a real-time trajectory of an aircraft based on the enhanced PPO algorithm as described in claim 1, characterized in that: The over-limit reward is assigned when the aircraft violates performance constraints, including a negative reward related to the degree of violation and a reward related to the closest distance between the trajectory and the target; The non-arrival process reward refers to the reward given to the aircraft during the rapid positioning process based on the terminal state of the trajectory, which guides the aircraft to reach the target pose more quickly. The reward consists of two parts: one part is to guide the aircraft to approach the target pose, and the other part is to guide the aircraft to reach the target pose more quickly. The arrival reward is awarded after the aircraft reaches the designated target pose range, and the environmental simulation process is terminated at the same time.

6. The method for generating a real-time trajectory of an aircraft based on the enhanced PPO algorithm as described in claim 1, characterized in that: The grouping in step S2-3 is specifically as follows: Based on the values ​​of the target pose in different dimensions, the sampling trajectory is divided into multiple preset groups.

7. The method for generating a real-time trajectory of an aircraft based on the enhanced PPO algorithm as described in claim 1, characterized in that: The enhanced PPO algorithm is based on the PPO algorithm, and a discount factor is set during training. Generalized dominance estimation parameters .

Images