Supersonic aircraft dive section trajectory planning method based on deep reinforcement learning

Abstract

The invention discloses a supersonic aircraft dive section trajectory planning method based on deep reinforcement learning, and belongs to the field of aircraft control, and the method comprises the steps: building an aircraft trajectory planning simulation environment according to the dynamic characteristics of an air-breathing supersonic aircraft, including the building of a dynamic model and the building of a control problem; establishing a deep reinforcement learning task model, namely establishing a Markov decision model of the aircraft dive section, including establishing a state space, an action space and designing a reward function; and solving the learning task model by using a deep reinforcement learning algorithm, and performing online trajectory planning of the aircraft by using the trained agent to obtain a required trajectory. Compared with the prior art, through the design of the reward function and the application of the deep reinforcement learning algorithm, the real-time performance and the accuracy of aircraft diving section trajectory planning are improved, and a wider application value is provided for the field of aircraft online trajectory planning and design.

Description

Technical Field

[0001] The present invention belongs to the field of supersonic aircraft control, and in particular relates to a trajectory planning method for an aircraft dive phase based on deep reinforcement learning. Background Art

[0002] An air-breathing supersonic aircraft is an aircraft that uses air-breathing engine technology to achieve supersonic flight. The aircraft sprays oxygen from the atmosphere into the engine combustion chamber through the front air intake, and the fuel and combustion combine to generate thrust, pushing the aircraft into the air. Unlike traditional turbojet engines, air-breathing engines can maintain stable combustion and thrust during supersonic flight, and can perform tasks such as high-speed military reconnaissance, long-range strikes, and air transport.

[0003] Air-breathing supersonic vehicles have strong tactical advantages and penetration capabilities due to their high-speed flight and maneuverability. The dive phase trajectory plays a vital role as a key stage for the vehicle to strike the target. The trajectory optimization design of the dive phase can improve the attack accuracy, optimize the attack effect, increase the success rate of the attack and the raid performance.

[0004] At present, the trajectory planning methods for the dive phase of supersonic aircraft are mainly direct and indirect methods. Among them, the indirect method is based on the Pontryagin maximum principle. The process of converting the optimal control problem into a Hamilton boundary value problem is complicated, and the convergence domain when solving is very small, so the initial value accuracy is required to be high, which makes the indirect method time-consuming when applied and difficult to meet the time requirements of trajectory planning, but its solution accuracy is better than the direct method; the direct method mainly includes the shooting method, pseudo-spectral method, differential inclusion method, etc. Although the planning time is faster than the indirect method, when the aircraft is subject to strong interference and deviates from the pre-planned trajectory, the direct method cannot achieve the real-time planning of rapid trajectory change. In actual flight missions, factors such as environmental changes, model uncertainty, and mechanical property uncertainty can cause the actual situation of the aircraft to deviate from the designed trajectory or even make it invalid. This shows that real-time is a very important factor in the optimization design of aircraft trajectories. However, the current traditional trajectory planning methods have poor ability to cope with external real-time interference and it is difficult to ensure the real-time planning.

[0005] Therefore, how to plan a fast-changing, high-precision trajectory while satisfying the dynamic characteristics of the aircraft to meet the actual flight mission requirements is an important technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the present invention is to provide a supersonic aircraft dive segment trajectory planning method based on deep reinforcement learning. Based on the deep reinforcement learning algorithm, a deep reinforcement learning task model is established and solved to achieve the task of quickly changing the planned aircraft trajectory online to reach the specified target location due to the influence of external factors.

[0007] The technical solution to achieve the purpose of the present invention is:

[0008] A supersonic aircraft dive trajectory planning method based on deep reinforcement learning, the method comprising: constructing an aircraft trajectory planning simulation environment, including establishing a dynamic model and establishing boundary constraints;

[0009] Establish a deep reinforcement learning task model, that is, establish a Markov decision model for the dive phase of the aircraft, including constructing the state space, action space and designing the reward function, where the reward function design is based on the target landing area;

[0010] The deep reinforcement learning algorithm based on the actor-critic framework is used to solve the problem. The full state quantity is used as input, and the deep deterministic policy gradient algorithm is used for calculation to train the agent model.

[0011] The trained proxy model is called to perform online trajectory planning of the aircraft to obtain the required dive trajectory of the aircraft.

[0012] Compared with the prior art, the present invention has the following significant advantages:

[0013] 1. The present invention proposes a supersonic aircraft dive segment trajectory planning method based on deep reinforcement learning. By improving deep reinforcement learning, it can be applied to the trajectory planning problem of the aircraft and can quickly and effectively perform online trajectory planning for the aircraft.

[0014] 2. The present invention can effectively realize the actual flight mission of supersonic aircraft by constructing a dynamic model and interactive environment of supersonic aircraft, and is not limited to specific aircraft, which is conducive to the study of different types of aircraft and has more universality;

[0015] 3. The present invention innovatively designs relevant reward functions, which can effectively plan the trajectory of an aircraft that meets the predetermined goal. That is, it can effectively apply the deep reinforcement learning method to the field of aircraft trajectory planning, complete the designated flight mission, and lay the foundation for modern aircraft guidance and control technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 The figure is a flow chart of the overall algorithm of an embodiment of the present invention.

[0018] Figure 2 This is a graph showing the accumulation of reward returns over iteration cycles in the present invention.

[0019] Figure 3 It is the altitude-range trajectory diagram of the present invention.

[0020] Figure 4 It is the range-time curve diagram of the present invention.

[0021] Figure 5 It is the speed-range curve diagram of the present invention.

[0022] Figure 2 The mid-fold line is described as follows:

[0023] The light blue line represents the periodic reward, the dark blue line represents the average reward, and the yellow line is an estimate of the long-term reward discount. DETAILED DESCRIPTION

[0024] The present invention proposes a supersonic vehicle dive trajectory planning method based on deep reinforcement learning, and the specific steps are as follows:

[0025] S1: Establish an aircraft trajectory planning simulation environment based on the aircraft dynamics characteristics.

[0026] S101: Kinetic model establishment;

[0027] Assuming that the earth is a uniform sphere, ignoring the earth's rotation, ignoring the dynamic aerodynamic force and sideslip of the aircraft, the dynamic model of the aircraft is established as follows:

[0028]

[0029] Where t is the flight time, x is the range of the aircraft, and h is the flight altitude of the aircraft; θ and are the inclination and pitch angles of the aircraft, and the angle of attack v and q are the velocity vector and pitch angular velocity vector of the aircraft respectively; m is the mass of the aircraft; g is the gravitational acceleration; I yy is the moment of inertia, L is the lift and D is the drag, M yy is the pitching moment.

[0030] Among them, the lift L and the drag D are:

[0031]

[0032] In the formula, C L ,C D are lift coefficient and drag coefficient, respectively, both of which are related to the missile’s flight attack angle α and Mach number Ma; S is the missile reference area; ρ = 1.703ρ SL exp(-h / 6.33) is the air density, ρ SL =1.225kg / m 3 is the atmospheric parameter value at sea level;

[0033] Pitch moment M yy for:

[0034]

[0035] In the formula, is the mean aerodynamic chord length; C m is the pitch moment coefficient, which is related to α, Ma and the control quantity, i.e. the rudder deflection angle δ z related;

[0036] S102: Definition of boundary constraints:

[0037] The boundary constraints of the trajectory planning problem for the dive phase of the aircraft are:

[0038] The constraints satisfied by the aircraft at the initial moment are:

[0039]

[0040] In the formula, t0 is the initial time, x0 is the state value of the range at the initial time, h0 is the state value of the height at the initial time, v0 is the state value of the velocity at the initial time, θ0 is the state value of the inclination at the initial time, and m0 is the state value of the mass at the initial time. is the state value of the pitch angle at the initial moment, q0 is the state value of the pitch angular velocity at the initial moment.

[0041] The constraints satisfied by the target area (terminal moment) of the aircraft are:

[0042]

[0043] In the formula, t f is the terminal moment, h f is the state value at the end of height, θ f is the state value at the end of the inclination, v fmin is the minimum boundary of the state value at the end of the speed, m fmin is the minimum boundary of the state value at the end of the quality, It is the state value at the end of the pitch angle.

[0044] S2: Establish a deep reinforcement learning task model.

[0045] A Markov decision model for the dive phase of an aircraft is established, which includes the establishment of state space, action space, and reward function.

[0046] S201: construct state space;

[0047] In view of the strong interference and fast time-varying characteristics of the aircraft dive phase, the full state variables are selected as the state space, that is, the state space model is defined as:

[0048]

[0049] Where s is the state vector space.

[0050] S202: constructing action space;

[0051] According to the control quantity setting of the aircraft dynamics model, the rudder deflection angle δ z Select as the action space, that is, the action space model is defined as:

[0052] a=δ z (19)

[0053] Where a is the action vector space. And the action δ z At the minimum boundary δ zmin and the maximum boundary δ zmax Between, that is, δ zmin ≤δ z ≤δ zmax .

[0054] S203: Design reward function:

[0055] The purpose of trajectory planning for the dive phase of the aircraft is to make the aircraft reach the target area. The target area is the set state quantity terminal constraint, that is, Approaching When the aircraft approaches the target area by a certain distance, it means that the training body is close to the target area, that is, the mission approach condition is:

[0056]

[0057] In the formula, is the distance between the current state of the aircraft and the target state, e(t) is the current distance between the aircraft and the target point, t max is the maximum flight time, l1 is the activity radius of the planned target direction but not reaching the target area correctly. When the aircraft approaches the target area at a close distance, it means that the training body has reached the target area, that is, the mission is completed.

[0058] The conditions are:

[0059]

[0060] Among them, l2 is the radius of the target area, that is, the terminal strike accuracy error.

[0061] In order to avoid the program being stuck due to the long training time of reinforcement learning, it is necessary to set the maximum time of each round. Once the set maximum time is exceeded, the task is forced to stop. That is, the task timeout termination condition is:

[0062] t>t max (twenty two)

[0063] When the target direction is planned correctly but the target area is not reached correctly, a smaller reward value r1 is given, which is smaller than the reward for entering the target area; when entering the target area, a reward r2 is given, which is a larger reward compared to r1; when the target exceeds the time limit or deviates from the target area, no reward is given. The reward design is:

[0064]

[0065] In the formula, l1>l2, and the training ends when the aircraft reaches the target area.

[0066] Then the reward function r(Reward) is:

[0067] r(t)=r1+r2 (24)

[0068] S3: Solve the deep reinforcement learning task model.

[0069] Based on the S2 task model, the deep reinforcement learning algorithm based on Deep Deterministic Policy Gradient (DDPG) is used to solve it. DDPG uses the Actor-Critic algorithm as its basic framework, which consists of a Critic network Q(s,a|θ Q ), Actor network μ(s|θ μ ), Critic target network Q' and Actor target network μ'. Among them, s is the state, a is the action, θ Q is the parameter definition of the Critic network, θ μ is the definition of Actor network parameters. In the solution process, the Critic network is used to evaluate the action strategy, the Actor network is used to train the action, and then the experience replay pool R is used to store the experience samples. The specific steps are as follows:

[0070] S301: Initialize the Critic network Q(s,a|θ Q ) and the Actor network μ(s|θ μ );

[0071] In the formula, θ Q is the Critic network parameter, θ μ Actor network parameters.

[0072] S302: Initialize the Critic target network Q' and the Actor target network μ' using the network initialized in the previous step;

[0073] S303: Initialize the experience replay pool R and the noise distribution N;

[0074] S304: Initialize the state space to obtain the current state s at time t t ;

[0075] S305: According to the current state s obtained in the previous step t , combined with the Actor network μ(s|θ μ ) and noise N t Design output action a t , and pass it to the environment to perform the action, which is specifically:

[0076] a t =μ(s t |θ μ )+N t (25)

[0077] Where N t is the noise at time t, a t is the action at time t, s t is the state at time t.

[0078] S306: After executing a in the previous step t After that, the state s t Transition to the next state s in the environment t+1 (state s at time t+1), and obtain the reward r according to the designed reward function t (reward r at time t), and the relevant data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool R;

[0079] S307: When the experience replay pool R is full, n samples (s i ,a i ,r i ,si+1 ), as the training critic network Q(s,a|θ Q ) and the Actor network μ(s|θ μ ) sample library;

[0080] S308: Using the s obtained in S306 t 、s t+1 and r i Calculate the Critic target network Q' and the Actor target network μ', that is

[0081] y i =r i +γQ'(s i+1 ,μ'(s i+1 |θ μ' )|θ Q' ) (26)

[0082] Where i is the order of n samples randomly drawn from the playback pool R, y i is the target value of Q in the current state i, s i+1 is the state value of the next state i+1, γ is the discount factor, θ μ ' is the Actor target network parameter, θ Q ' is the Critic target network parameter.

[0083] S309: Update the Critic network Q(s,a|θ) by minimizing the loss function Critic Loss Q ),Right now

[0084]

[0085] S310: Update the Actor network μ(s|θ) through the Actor’s sampling strategy gradient μ ),Right now

[0086]

[0087] S311: Update the Critic target network Q' and the Actor target network μ', that is

[0088] θ Q′ ←τθ Q +(1-τ)θ Q′ (29)

[0089] θ μ′ ←τθ μ +(1-τ)θ μ′

[0090] Where τ is the update rate (τ∈(0,1));

[0091] S312: Repeat steps S304 to S311 for a specified iteration period E times to complete the environment interaction and parameter update of reinforcement learning, output the trained network, rewards and strategies, and save the trained network agent model.

[0092] S4: Call the trained proxy model to perform online trajectory planning of the aircraft and obtain the required trajectory output of the aircraft's dive phase state quantity.

[0093] Example

[0094] All simulations in this case were performed in MATLAB 2023a on a 12th Gen Intel(R) Core(TM) i7-12700H 2.30GHz PC. The aircraft was trained to reach the mission target, and the relevant simulation parameters were set as follows:

[0095] The initial height of the aircraft dive phase is 20km, the initial range is 0km, the initial speed is 1100m / s, and the initial inclination angle is 0°; the terminal expected target height is 0km, and the terminal expected target range is 90km; the iteration cycle is 18,000 times.

[0096] After interaction and training with the environment, we get the policy agent, and then evaluate the agent. The reward result is as follows: Figure 2 As shown in the figure, it can be seen that the reward return tends to converge when the iteration cycle reaches 10,000 steps; the trajectory planning of the aircraft is performed using the trained network agent model, and the Figure 3 The altitude-range trajectory, Figure 4 The range-time curve and Figure 5 The simulation case verifies that the supersonic vehicle dive phase trajectory planning method based on deep reinforcement learning proposed in this invention can complete the desired flight mission and reach the specified target position.

Claims

1. A supersonic vehicle dive trajectory planning method based on deep reinforcement learning, characterized in that: include: Constructing a simulation environment for aircraft trajectory planning, including the establishment of dynamic models and boundary constraints; Establish a deep reinforcement learning task model, that is, establish a Markov decision model for the dive phase of the aircraft, including constructing the state space, action space and designing the reward function, where the reward function design is based on the target landing area; The deep reinforcement learning algorithm based on the actor-critic framework is used to solve the problem. The full state quantity is used as input, and the deep deterministic policy gradient algorithm is used for calculation to train the agent model. The trained proxy model is called to perform online trajectory planning of the aircraft to obtain the required dive trajectory of the aircraft.

2. The method for planning a trajectory of an aircraft dive phase according to claim 1, characterized in that: In the aircraft trajectory planning simulation environment, the dynamic model of the aircraft is: Where t is the flight time, x is the range of the aircraft, and h is the flight altitude of the aircraft; θ and are the inclination and pitch angle of the aircraft, v and q are the velocity vector and pitch angle velocity vector of the aircraft, respectively; m is the mass of the aircraft; g is the gravitational acceleration; I yy is the moment of inertia, L is the lift, D is the drag, M yy is the pitching moment.

3. The method for planning a trajectory of an aircraft dive phase according to claim 2, characterized in that: In the construction of the aircraft trajectory planning simulation environment, the boundary constraints are: The boundary constraints of the dive segment trajectory planning problem are: The constraints satisfied by the aircraft at the initial moment are: In the formula, t0 is the initial time, x0 is the state value of the range at the initial time, h0 is the state value of the height at the initial time, v0 is the state value of the velocity at the initial time, θ0 is the state value of the inclination at the initial time, and m0 is the state value of the mass at the initial time. is the state value of the pitch angle at the initial moment, q0 is the state value of the pitch angular velocity at the initial moment; The constraints satisfied by the target area (terminal moment) of the aircraft are: Where, t f is the terminal moment, h f is the state value at the end of height, θ f is the state value at the end of the inclination, v fmin is the minimum boundary of the state value at the end of the speed, m fmin is the minimum boundary of the state value at the end of the quality, It is the state value at the end of the pitch angle.

4. The method for planning a trajectory of an aircraft dive phase according to claim 2, characterized in that: In establishing the deep reinforcement learning task model, the state space model constructed is: Where s is the state vector space.

5. The method for planning a trajectory of an aircraft dive phase according to claim 1, characterized in that In establishing the deep reinforcement learning task model, the action space model constructed is: a=δ z Where a is the action vector space, δ z is the rudder deflection angle.

6. The method for planning a trajectory of an aircraft dive phase according to claim 1, characterized in that: In establishing the deep reinforcement learning task model, the reward function constructed is: The reward function r(Reward) is: r(t)＝r1+r2 Among them, r1 is the reward when the aircraft correctly plans to the target direction but fails to reach the target area correctly, and r2 is the reward when it correctly enters the target area.

7. The method for planning a trajectory of an aircraft dive phase according to claim 6, characterized in that: Reward design is defined as: Where e(t) is the current distance between the aircraft and the target point, t is the flight time, and t max is the maximum flight time, l1 is the activity radius of the planned target direction but not reaching the target area correctly, l2 is the radius of the target area, l1>l2, and the training ends when the aircraft reaches the target area.

8. The method for planning a trajectory of an aircraft dive phase according to claim 2, characterized in that: The calculation formulas for lift L and drag D are: In the formula, C L ,C D are lift coefficient and drag coefficient, respectively, both of which are related to the missile’s flight attack angle α and Mach number Ma; S is the missile reference area; ρ = 1.703ρ SL exp(-h / 6.33) is the air density, ρ SL =1.225kg / m 3 is the atmospheric parameter value at sea level; Pitch moment M yy for: In the formula, is the mean aerodynamic chord length; C m is the pitch moment coefficient, which is related to α, Ma and the control quantity, i.e. the rudder deflection angle δ z related.

9. The method for planning a trajectory of an aircraft dive phase according to claim 1, characterized in that: The deep reinforcement learning algorithm based on the actor-critic framework is used to solve the problem. The full state quantity is used as input, and the deep deterministic policy gradient algorithm is used for calculation to train the agent model. The specific steps are as follows: S301: Initialize the Critic network Q(s,a|θ Q ) and the Actor network μ(s|θ μ ); In the formula, θ Q is the Critic network parameter, θ μ Actor network parameters; S302: Initialize the Critic target network Q' and the Actor target network μ' using the network initialized in the previous step; S303: Initialize the experience replay pool R and the noise distribution N; S304: Initialize the state space to obtain the current state s at time t t ; S305: According to the state s obtained in the previous step t , combined with the Actor network μ(s|θ μ ) and noise N t Design output action a t , and pass it to the environment to perform the action, which is specifically: a t =μ(s t |θ μ )+N t Where N t is the noise at time t, a t is the action at time t, s t is the state at time t; S306: After executing a in the previous step t After that, the state s t Transition to the next state s in the environment t+1 , and obtain the reward r according to the designed reward function t , and the relevant data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool R; S307: When the experience replay pool R is full, n samples (s i ,a i ,r i ,s i+1 ), as the training critic network Q(s,a|θ Q ) and the Actor network μ(s|θ μ ) sample library; S308: Using the s obtained in S306 t 、s t+1 and r i Calculate the Critic target network Q' and the Actor target network μ', that is y i =r i +γQ'(s i+1 ,m'(s i+1 |θ μ' )|θ Q' ) Where i is the order of n samples randomly drawn from the playback pool R, y i is the target value of Q in the current state i, s i+1 is the state value of the next state i+1, γ is the discount factor, θ μ ' is the Actor target network parameter, θ Q ' is the Critic target network parameter; S309: Update the Critic network Q(s,a|θ) by minimizing the loss function Critic Loss Q ),Right now S310: Update the Actor network μ(s|θ) through the Actor’s sampling strategy gradient μ ),Right now S311: Update the Critic target network Q' and the Actor target network μ', that is i Q′ ←tth Q +(1-τ)θ Q′ i μ' ←tth μ +(1-τ)θ μ' Where τ is the update rate (τ∈(0,1)); S312: Repeat steps S304 to S311 for a specified iteration period E times to complete the environment interaction and parameter update of reinforcement learning, output the trained network, rewards and strategies, and save the trained network agent model.

Images