Aircraft intelligent guidance method based on reinforcement learning

Abstract

According to the aircraft intelligent guidance method based on reinforcement learning, under the condition that constraint conditions such as a kinetic equation, a heat flow rate, overload and dynamic pressure are met, effective rewards are set to guide an aircraft to explore in the environment, a near-end strategy optimization algorithm in reinforcement learning is used for training an aircraft heeling angle guidance model, and the aircraft heeling angle guidance model is optimized. A heeling angle instruction is generated on line based on real-time state information, the constraint of a traditional guidance method in the aspect of a transverse corridor is subverted, and completely different flight paths are explored. Simulation results show that the aircraft intelligent guidance method based on reinforcement learning can give full play to the wide-area flight advantage of the aircraft, the flight profile is further expanded, meanwhile, the method has high robustness to state disturbance, and a good control effect is achieved under various conditions.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft guidance and control, and relates to an aircraft intelligent guidance method based on reinforcement learning, which is used to solve the problem that traditional guidance methods rely on pre-launch design of flight corridors, innovatively form new quality flight profiles, and enable the aircraft to make online decisions on roll angle instructions based on real-time status information, thereby supporting the leapfrog development of my country's aircraft towards intelligence. Background Art

[0002] Aircraft have high flight speed, wide flight airspace, long range, and strong maneuverability, playing an important role in military and civilian fields. During the reentry of an aircraft, the external environment such as atmospheric density and pressure changes dramatically, and the dynamics and aerodynamic characteristics of the aircraft are highly complex and uncertain. During the flight, the process and control constraints need to be met, which puts higher requirements on the reentry guidance technology of the aircraft.

[0003] At present, the reentry guidance methods of aircraft mainly include the following two types: guidance method based on nominal trajectory and prediction and correction guidance. The guidance method based on nominal trajectory can be guided by offline design of drag acceleration-energy, altitude-speed and other profiles, and real-time tracking of the nominal trajectory during flight. This method is simple in principle and has strong engineering feasibility, but it has obvious deficiencies in adaptability and robustness to complex environments. The prediction and correction guidance method integrates the dynamic equation based on the current state of the aircraft, calculates and predicts the terminal state of the aircraft, and corrects the guidance instructions based on the deviation between the terminal state and the target point, thereby achieving precise guidance of the aircraft. Compared with the guidance method based on nominal trajectory tracking, the prediction and correction guidance has stronger autonomy and adaptability to complex environments.

[0004] However, traditional reentry guidance and predictive correction guidance based on nominal trajectory mostly require the design of flight corridor parameters based on artificial experience. The guidance method based on nominal trajectory needs to convert the constraints such as heat flow and dynamic pressure during the reentry process of the aircraft into the upper and lower boundaries of the flight corridor, and then carefully design the nominal flight trajectory that meets the terminal constraints within the boundary range; the predictive correction guidance method needs to determine the sign of the aircraft's roll angle by designing a suitable heading angle error corridor. When the aircraft's heading angle error reaches the corridor boundary, the roll angle sign flips. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, to provide an aircraft intelligent guidance method based on reinforcement learning, to solve the problem that the traditional guidance method relies on the pre-launch flight corridor design, and to enable the aircraft to make online decisions on the roll angle command based on real-time status information.

[0006] The technical solution of the present invention is: an aircraft intelligent guidance method based on reinforcement learning, the method comprising the following steps:

[0007] S1. Define and obtain the observation space of the aircraft, wherein the observation space includes the aircraft state quantity, the aircraft state differential quantity, the difference between the aircraft state and the state of the mid-term handover point, the Euclidean distance between the longitude and latitude of the aircraft and the longitude and latitude of the mid-term handover point, and the difference between the aircraft energy and the terminal energy;

[0008] S2. Define the action space of the reinforcement learning strategy model as the roll angle reference of the aircraft, use the proximal strategy to optimize the reinforcement learning model, and generate the roll angle reference of the aircraft online based on the real-time state space;

[0009] S3, mapping the aircraft's roll angle reference output by the reinforcement learning strategy model to obtain the aircraft's theoretical roll angle;

[0010] S4. The theoretical roll angle of the aircraft is truncated to obtain the final roll angle instruction actually used to control the aircraft.

[0011] S5: Set the roll angle command Substitute the dynamics model to update the aircraft state and re-execute steps S1 to S5.

[0012] Preferably, the calculation method for truncating the theoretical roll angle is:

[0013]

[0014] Among them, σ max is the constraint on the roll angle amplitude, and η is the safety factor.

[0015] Preferably, the constraint σ of the roll angle amplitude max for:

[0016]

[0017]

[0018] in, is the heel angle amplitude threshold under heat flux constraint; is the heel angle amplitude threshold under overload constraint; is the roll angle amplitude threshold under dynamic pressure constraint; m is the mass of the aircraft; g is the acceleration of gravity; V is the speed of the aircraft; r is the distance from the center of the earth; S ref is the characteristic area of ​​the aircraft; k Q is the heat flow rate model coefficient; is the maximum allowable heat flux at the stagnation point; C L and C D are lift coefficient and drag coefficient respectively; n max is the maximum permissible overload; g0 is the gravitational acceleration at sea level; q max is the maximum dynamic pressure.

[0019] Preferably, the proximal policy optimization reinforcement learning model includes an Actor model and a Critic model, the Actor model outputs the action of the aircraft according to the observation space of the aircraft, the Critic model calculates the reward according to the observation space of the aircraft, and optimizes the Actor model with the maximization of reward expectation as the objective function.

[0020] Preferably, the objective function calculation formula is as follows:

[0021]

[0022] In the formula, R(s t ,a t ) is the reward function, indicating that the aircraft is in state s t Execute action a t The expected reward of the immediate reward obtained, including the lateral reward R h (s t ,a t ), vertical reward R v (s t ,a t ) and the final reward R f (s t ,a t ), κ t The discount factor for the current calculation period ranges from 0 to 1.

[0023] Preferably, the horizontal reward R h (s t ,a t ) is the sum of the latitude and longitude bonus and the direction bonus.

[0024] Preferably, the direction reward r dir By calculating the actual velocity direction ψ and the line of sight angle ψ of the aircraft →tgt The difference between the two guides the aircraft to fly to the middle and last handover points along the line of sight angle. The calculation formula is as follows:

[0025]

[0026] Where, L togo is the remaining range of the aircraft; It is a hyperparameter used to regulate weight changes in direction rewards; The weight used to adjust the impact of the directional bonus on the aircraft; represents the heading angle ψ t With sight angle difference;

[0027]

[0028] in, is the expected longitude value of the mid-to-final handover point, θ is the actual longitude value of the mid-to-final handover point; φ is the actual longitude value of the mid-to-final handover point, is the expected latitude of the mid- and end-shift handover points, t represents the current calculation cycle, and t-1 represents the previous calculation cycle;

[0029] The latitude and longitude bonus serves to guide the aircraft to fly close to the mid-term and final handover points, and the calculation formula is as follows:

[0030]

[0031] Among them, L togo is the remaining range of the aircraft, L * is the set distance threshold; It is a hyperparameter used to regulate the weight change in longitude and latitude rewards; is the weight used to adjust the influence of longitude and latitude bonus on the aircraft; θ is the actual longitude of the mid-term handover point; φ is the actual longitude of the mid-term handover point; Respectively represent the Euclidean distance between the aircraft longitude and latitude and the mid-end handover point longitude and latitude in the current calculation cycle and the previous calculation cycle.

[0032] Preferably, the vertical reward R v (s t ,a t ) is the sum of energy bonus, speed bonus and height bonus.

[0033] Preferably, the energy reward r E The calculation formula is as follows:

[0034]

[0035]

[0036] In the formula, Indicates that the mean is E μ , with a standard deviation of δ E From the Gaussian distribution of The probability of V is the bias used to control the speed error, b h is the offset used to adjust the height error, V is the speed of the aircraft, R 0 is the radius of the Earth, GM represents the gravitational constant;

[0037] The speed reward r V The calculation formula is as follows:

[0038]

[0039] In the formula, Indicates that the mean value is V μ , with a standard deviation of δ V Get V+b from the Gaussian distribution V The probability of V is the bias used to adjust the speed error;

[0040] The height reward r h The calculation formula is as follows:

[0041]

[0042] In the formula, Indicates that the mean is h μ , with a standard deviation of δ h Get h+b from the Gaussian distribution h The probability of h is the offset used to adjust the height error; This is the hyperparameter used to regulate the weight change in this reward, and h is the flight altitude of the aircraft.

[0043] Preferably, the final reward r tgt When the aircraft arrives at the mid-term and final handover point, the calculation formula is as follows:

[0044]

[0045] In the formula, ω r ,ω θ ,ω φ ,ω V are the influence coefficients used to balance the impact of aircraft distance, longitude, latitude, and speed amplitude, ε r ,ε θ ,ε φ ,ε V are the hyperparameters used to calculate the relative error of the aircraft distance, longitude, latitude, and speed. The reward at the end of the rest of the time is 0.

[0046] The advantages of the present invention compared with the prior art are:

[0047] (1) The aircraft intelligent guidance method based on reinforcement learning proposed in the present invention guides the aircraft to explore and make mistakes in the environment by setting effective rewards, and uses the PPO algorithm in reinforcement learning to train the aircraft roll angle guidance model, so as to give full play to the wide-range flight advantage of the aircraft, break through the limitations of traditional methods, and innovatively form a new quality flight profile, so as to support the leapfrog development of my country's aircraft towards intelligence.

[0048] (2) The aircraft intelligent guidance method based on reinforcement learning proposed in the present invention can realize the online decision-making of the aircraft's roll angle command based on real-time status information, effectively improving the autonomy of the aircraft guidance method and its adaptability to complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The comparison results between the PPO strategy and the prediction correction guidance method (the curve of altitude changing with speed);

[0050] Figure 2 The comparison results between the PPO strategy and the prediction correction guidance method (longitude and latitude change curves);

[0051] Figure 3 The comparison results between the PPO strategy and the prediction correction guidance method (roll angle vs. speed curve);

[0052] Figure 4 This is the comparison result between the PPO strategy and the prediction correction guidance method (the curve of heading angle changing with time). DETAILED DESCRIPTION

[0053] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0054] At present, preliminary exploratory research has been carried out on aircraft guidance and control technology based on artificial intelligence. It has the characteristics of small computational complexity and high precision. It can realize the rapid mapping of the real-time status of the aircraft to the guidance instructions, and has broad application prospects.

[0055] The present invention designs an intelligent aircraft guidance method based on reinforcement learning to solve the problem that traditional guidance methods rely on pre-launch design of flight corridors. Under the condition of satisfying constraints such as dynamic equations, heat flux, overload, dynamic pressure, etc., the aircraft is guided to explore in the environment by setting effective rewards, and the reinforcement learning model is optimized by using proximal strategies. The roll angle command is generated online based on real-time state information, which subverts the constraints of traditional guidance methods in terms of lateral corridors, explores completely different flight trajectories, and enables the aircraft to make online decisions on roll angle commands based on real-time state information.

[0056] The present invention is introduced from two aspects: aircraft modeling and aircraft intelligent guidance method based on reinforcement learning:

[0057] (1) Aircraft Modeling

[0058] (a) Aircraft dynamics modeling:

[0059]

[0060] Where r is the distance from the center of the earth; θ and φ represent the longitude and latitude of the aircraft respectively; V is the speed of the aircraft; γ and ψ are the track angle and heading angle of the aircraft respectively; σ is the tilt angle of the aircraft; m is the mass of the aircraft; g is the acceleration of gravity; L and D are the lift and drag of the aircraft; t is the time:

[0061]

[0062] Where ρ = ρ 0 e -βh is the atmospheric density at the aircraft height h, ρ 0 is the atmospheric density at sea level; β = 1 / H MCP , H MCP =7.11km is the base altitude; S ref is the characteristic area of ​​the aircraft; C L and C D are lift coefficient and drag coefficient respectively, which are related to the aircraft attack angle α and flight speed V. In a specific embodiment of the present invention, the US general aviation aircraft CAV-H is taken as the research object, and the expressions of lift coefficient and drag coefficient are:

[0063]

[0064] In the formula, C L0 , C L1 , C L2 and C L3 is the fitting coefficient of lift expression; C D0 , C D1 , C D2 and C D3 is the fitting coefficient of the resistance expression.

[0065] (b) Process constraint modeling: When a glider is flying at high speed, the high-speed airflow will generate a large amount of heat energy in the aircraft stationary area. Considering the safety of the aircraft, the heat flow rate constraint should be met:

[0066]

[0067] In the formula, represents the heat flow rate at the stagnation point; k Q is the heat flow rate model coefficient; is the maximum allowable heat flux rate at the stagnation point.

[0068] Overload refers to the multiple of the combined force of aerodynamic force and thrust acting on the aircraft relative to the aircraft's gravity. Due to the limited structural bearing capacity of the aircraft, the aircraft overload cannot exceed the maximum overload limit for safety reasons. Since the aircraft in the present invention has no thrust effect during the gliding phase, its overload constraint is

[0069]

[0070] Where n is the aircraft overload; n max is the maximum permissible overload; g 0 is the gravitational acceleration at sea level.

[0071] Also from the perspective of the structural bearing capacity of the aircraft, the aircraft should also meet the dynamic pressure constraint. During high-speed flight, the air flows in the opposite direction of the aircraft body at high speed. At this time, the air itself has kinetic energy. The pressure presented by this kinetic energy is called dynamic pressure:

[0072]

[0073] Where q is the dynamic pressure of the aircraft; q max is the maximum dynamic pressure.

[0074] The heat flux, overload and dynamic pressure constraints all involve aircraft safety issues and are hard constraints. The quasi-balanced gliding condition is a soft constraint to ensure the stability of the aircraft's altitude and track angle and avoid large changes. The mathematical form of this constraint is as follows:

[0075]

[0076] Substituting into the expression of the track angle of the dynamic model, the motion equation describing the quasi-balanced gliding can be obtained:

[0077]

[0078] It can be seen from the above formula that when the aircraft satisfies the above formula, the combined force of its gravity and aerodynamic lift provides the required centripetal force. At this time, the change in flight altitude is very small, and the rate of change of the track angle γ is Close to zero.

[0079] (c) Terminal constraint modeling: The terminal constraints of the aircraft include the altitude, speed, latitude and longitude constraints when arriving at the mid-terminal handover point:

[0080]

[0081] In the formula, h f 、V f ,θ f ,φ f They are the actual values ​​of altitude, speed, longitude and latitude when the aircraft arrives at the mid-term and final handover points; is the expected value of the altitude, speed, longitude and latitude of the mid-term and final handover points; ε h , ε V , ε θ , ε φ They are the maximum allowable errors of terminal altitude, speed, longitude and latitude respectively.

[0082] (d) Modeling of bank angle amplitude constraint: The aircraft needs to meet the above-mentioned hard constraints such as heat flux, overload and dynamic pressure during flight. However, repeated calculation of the aircraft's heat flux, overload and dynamic pressure will bring a large amount of calculation. Therefore, the hard constraints of heat flux, overload and dynamic pressure can be converted into constraints on bank angle amplitude, that is:

[0083]

[0084] The constraint form of the roll angle amplitude is:

[0085]

[0086] in, is the heel angle amplitude threshold under heat flux constraint; is the heel angle amplitude threshold under overload constraint; is the roll angle amplitude threshold under dynamic pressure constraint; m is the mass of the aircraft; g is the acceleration of gravity; V is the speed of the aircraft; r is the distance from the center of the earth; S ref is the characteristic area of ​​the aircraft; k Q is the heat flow rate model coefficient; is the maximum allowable heat flux at the stagnation point; C L and C D are lift coefficient and drag coefficient respectively; n max is the maximum permissible overload; g 0 is the gravitational acceleration at sea level; q max is the maximum dynamic pressure.

[0087] e) Control quantity change rate constraint modeling: Due to the function of the internal control mechanism of the aircraft, the change of the control quantity requires a certain change time and change speed, and cannot change to the specified value instantly. Since the attack angle α adopts the standard attack angle profile, the control quantity change rate constraint is mainly for the change rate of the roll angle σ, that is:

[0088]

[0089] In the formula, Represents the upper bound of the rate of change of the roll angle.

[0090] (2) Aircraft intelligent guidance method based on reinforcement learning

[0091] The reinforcement learning algorithm consists of five basic elements: agent, environment, state, action, and reward. The agent takes action, obtains reward value through interaction with the environment and enters the next state, and learns strategies repeatedly in this way. Eventually, the agent tends to take actions with large cumulative reward value. The following defines the observation space and action space, and designs the reward function from multiple perspectives, and finally designs the corresponding strategy model structure.

[0092] (a) Observation space: The observation space of the strategy model in the present invention is defined as follows:

[0093] o(t)=[r,θ,φ,V,γ,ψ,dr,dθ,dφ,dV,Δr,Δθ,Δφ,ΔV,l 2 (θ,φ),ΔE]∈R 16

[0094] It mainly includes the following parts: Aircraft state: [r,θ,φ,V,γ,ψ]∈R 6 , respectively, are the aircraft distance, longitude, latitude, speed, track angle, and heading angle, and the aircraft state differential: [dr, dθ, dφ, dV]∈R 4 , respectively, are the differentials of the aircraft distance, longitude, latitude, and speed, and the difference between the aircraft state and the state of the mid-term handover point: [Δr, Δθ, Δφ, ΔV]∈R 4 , respectively, are the difference in aircraft distance, longitude, latitude, and speed, and the Euclidean distance between the aircraft longitude and latitude and the mid-term handover point longitude and latitude: The difference between vehicle energy and terminal energy: The energy of the aircraft represents the energy of the last handover point in the aircraft; GM represents the gravitational constant, which is 3.986×10 14 N·m 2 / kg, and They are the expected longitude and latitude of the middle and last handover points respectively.

[0095] (b) Action space: The strategy model needs to output the vehicle’s roll angle reference (including its amplitude and sign) as a control variable, and must satisfy the constraint on the roll angle reference amplitude during the vehicle’s reentry.

[0096] (c) Intelligent guidance method for aircraft

[0097] In a specific implementation, the present invention calculates the roll angle according to the following process:

[0098] S1. Define and obtain the observation space of the aircraft, wherein the observation space includes the aircraft state quantity, the aircraft state differential quantity, the difference between the aircraft state and the state of the mid-term handover point, the Euclidean distance between the longitude and latitude of the aircraft and the longitude and latitude of the mid-term handover point, and the difference between the aircraft energy and the terminal energy;

[0099] S2. Define the action space of the reinforcement learning strategy model as the roll angle reference of the aircraft, use the proximal strategy to optimize the reinforcement learning model, and generate the roll angle reference of the aircraft online based on the real-time state space;

[0100] The output of the proximal policy optimization reinforcement learning model obeys the mean μ and standard deviation of the Gaussian distribution, that is, the output δ.

[0101] During the training process, a real number is randomly sampled from the Gaussian distribution as the output, the aircraft's roll angle reference z∈R. During the test, the mean of the Gaussian distribution is directly taken as the output, the aircraft's roll angle reference z=μ∈R;

[0102] S3, mapping the aircraft's roll angle reference value z output by the reinforcement learning strategy model to obtain the aircraft's theoretical roll angle;

[0103] The mapping relationship between the aircraft's roll angle reference value z and the aircraft's theoretical roll angle σ is:

[0104] σ=σ lim ×tanhz

[0105] Where tanh represents the hyperbolic tangent function, σ lim is the set heel angle boundary. In a specific embodiment of the present invention, σ lim The value is 80°.

[0106] S4. The theoretical roll angle of the aircraft is truncated to obtain the final roll angle instruction actually used to control the aircraft.

[0107] The calculation method for truncating the theoretical roll angle is:

[0108]

[0109] Among them, σ max is the heel angle amplitude threshold under the joint constraints of heat flux, overload and dynamic pressure, η is the safety factor, and the general recommended value is 0.8. max is the constraint on the roll angle amplitude.

[0110] S5: Set the roll angle command Substitute the dynamics model to update the aircraft state and re-execute steps S1 to S5.

[0111] (c) Proximal Strategy Optimization Reinforcement Learning Model

[0112] The PPO algorithm adopted by the proximal policy optimization reinforcement learning model of the present invention belongs to the Actor-Critic algorithm. The proximal policy optimization reinforcement learning model includes an Actor model and a Critic model. The Actor model outputs the action of the aircraft according to the observation space of the aircraft, and the Critic model calculates the reward according to the observation space of the aircraft, and optimizes the Actor model with the maximization of reward expectation as the objective function.

[0113] In the present invention, both the Actor model and the Critic model use a three-layer fully connected network, in which the length of the hidden variables in the middle layer is 64. The final output of the Actor is obtained through the roll angle calculation process in the action space to obtain the final control amount; the Critic output V(s)∈R is the value function of the current aircraft state.

[0114] The value function calculation formula is as follows:

[0115]

[0116] In the formula, R(s t ,a t ) is the reward function, indicating that the aircraft is in state s t Execute action a t The expected reward of the immediate reward obtained, including the lateral reward R h (s t ,a t ), vertical reward R v (s t ,a t ) and the final reward R f (s t ,a t ), κ t The discount factor for the current calculation period ranges from 0 to 1.

[0117] (d) Reward function design: According to the task requirements, the present invention designs multiple reward functions for PPO algorithm training from three aspects, namely, horizontal reward, vertical reward and end-stage reward.

[0118] Horizontal reward: It guides the aircraft to the mid-term handover point. In order to improve the exploration of the aircraft in the process of interaction with the environment, an exponentially changing reward weight is designed. When the aircraft is far away from the target point, the lateral reward is 0, allowing the aircraft to be flexible and maneuverable. When the aircraft approaches the mid-term handover point at a certain distance, the weight coefficient increases, and the guidance effect on the aircraft is enhanced, thus ensuring that the aircraft accurately reaches the last handover point. The lateral reward function will be designed from the two perspectives of aircraft direction and longitude and latitude, that is, the lateral reward R h (s t ,a t ) is the sum of the latitude and longitude bonus and the direction bonus.

[0119] Direction Reward dir It is calculated by calculating the actual velocity direction angle ψ and the line of sight angle ψ of the aircraft →tgt The difference between the two guides the aircraft to fly to the middle and last handover points along the line of sight:

[0120]

[0121] Where, L togo is the remaining range of the aircraft; It is a hyperparameter used to regulate weight changes in direction rewards; The weight used to adjust the impact of the direction reward on the aircraft; the weight increases as the remaining range decreases; represents the heading angle ψ t With sight angle difference.

[0122]

[0123] in, is the expected longitude value of the mid-to-final handover point, θ is the actual longitude value of the mid-to-final handover point; φ is the actual longitude value of the mid-to-final handover point, is the expected latitude of the mid- and end-shift handover points, t represents the current calculation cycle, and t-1 represents the previous calculation cycle.

[0124] The latitude and longitude rewards are the same as the direction rewards. dist It also guides the aircraft to fly closer to the mid-term handover point. In addition, when the aircraft is within a relatively close range to the mid-term handover point but starts to move away from the mid-term handover point, it will be given a larger penalty through weight changes. This negative reward will quickly offset the positive reward obtained in the process of approaching the mid-term handover point. The calculation formula is as follows:

[0125]

[0126] In the formula, L togo is the remaining range of the aircraft; L * is the set distance threshold, which is generally 2000km; It is a hyperparameter used to regulate the weight change in longitude and latitude rewards; is used to adjust the weight of the longitude and latitude bonus on the aircraft, and the weight increases as the remaining range decreases; θ is the actual longitude value of the mid-term handover point; φ is the actual longitude value of the mid-term handover point; Respectively represent the Euclidean distance between the aircraft longitude and latitude and the mid-end handover point longitude and latitude in the current calculation cycle and the previous calculation cycle.

[0127] Vertical reward: It plays a role in constraining the energy (speed and altitude) changes during the reentry of the aircraft. In order to obtain the expected value at each moment, a linear approximation method is selected, that is, it is assumed that the energy / speed / altitude changes approximately linearly with a certain variable from the starting point to the end point. Since linear approximation obviously has errors, the present invention will add speed and altitude offsets and increase weight control during the implementation process to ensure the actual constraint effect. The vertical reward R v (s t ,at ) is the sum of energy bonus, speed bonus and height bonus.

[0128] Energy Reward E The design is based on observing the change of the aircraft's mechanical energy with the remaining range. It is found that the mechanical energy changes approximately linearly with the remaining range, so the remaining range is selected to make a linear approximation of the energy change. The expected value of energy E is calculated according to the following formula μ , according to the Gaussian distribution, the error between its actual energy and the expected energy is calculated to provide a reward, and the standard deviation of the Gaussian distribution is used as a hyperparameter to adjust the tolerance to the error:

[0129]

[0130]

[0131] In the formula, E 0 and L 0 They represent the energy and remaining range of the vehicle at the reentry point respectively; Indicates that the mean is E μ , with a standard deviation of δ E From the Gaussian distribution of The probability of V is the bias used to control the speed error, b h is the offset used to adjust the height error, V is the speed of the aircraft, R 0 is the radius of the Earth.

[0132] Speed ​​Reward V The design also observed the change of the aircraft speed over time and found that the speed changes approximately linearly with time, so we chose to make a linear approximation of the speed change based on the flight time. The expected value of the speed V is calculated according to the following formula: μ , according to the Gaussian distribution, the error between its actual speed and the expected speed is calculated to provide a reward, and the standard deviation of the Gaussian distribution is used as a hyperparameter to adjust the tolerance to the error:

[0133]

[0134]

[0135] Where V 0 Indicates the speed of the vehicle at the reentry point; is the expected flight time; Indicates that the mean value is V μ , with a standard deviation of δ V Get V+b from the Gaussian distribution V The probability of V It is the bias used to adjust the speed error.

[0136] Highly rewarding h By observing the change of aircraft altitude with speed, the design found that the altitude will show a large fluctuation in the early stage of flight, and will show an approximately linear change in the later stage of flight. Therefore, the same exponential weight as the lateral reward will be added to the altitude reward, which is almost 0 in the early stage of flight and increases in the later stage of flight, starting to constrain the movement of the aircraft. The increase in altitude reward will avoid the situation where the altitude drops too quickly. Since the speed of the aircraft is greatly affected by the control amount, the present invention finally chooses to use time to make a linear approximate estimate of the altitude similar to the speed reward. The calculation formula is shown below.

[0137]

[0138]

[0139] In the formula, h 0 Indicates the altitude of the vehicle at the reentry point; Indicates that the mean is h μ , with a standard deviation of δ h Get h+b from the Gaussian distribution h The probability of h is the offset used to adjust the height error; The hyperparameter used to control the weight change in this reward is set to 2000km, and h is the current flight altitude of the aircraft.

[0140] Final reward: The mission completion will be determined based on the energy of the aircraft as needed. When the energy of the aircraft drops to the energy of the mid-term handover point, the mission is considered to have been completed.

[0141] At the end of the mission, the error between the state of the aircraft at the end of the calculation and the target state will be used to give the aircraft an appropriate reward. The smaller the error, the higher the reward, which is calculated as follows.

[0142]

[0143] In the formula, ω r ,ω θ ,ω φ ,ω V are the influence coefficients used to balance the impact of aircraft distance, longitude, latitude, and speed amplitude, ε r ,ε θ ,ε φ ,ε V are hyperparameters used to calculate the relative error of aircraft distance, longitude, latitude and speed. r =1.5km,ε θ =0.15°,εφ =0.15°,ε V =100m / s.

[0144] Simulation Verification

[0145] In order to verify the effect of the PPO algorithm, the present invention has carried out simulation verification in three cases. First, the algorithm was tested without disturbance in the re-entry point state, and the change curve of each state quantity during the re-entry process of the aircraft was obtained; then, each state quantity of the re-entry point state was disturbed respectively (that is, only a single state quantity was disturbed each time), and the state error when finally arriving at the middle and final handover points was compared; finally, the re-entry point state was disturbed as a whole, and the state error when finally arriving at the middle and final handover points was also compared. In the experiment, PPO was trained for 800 episodes, and 4096 environments were used in parallel in each episode for 400 steps of environmental interaction. Finally, the optimal model was selected from the last 100 episodes for testing, and the following results were obtained.

[0146] The aircraft starts from a given reentry point state, and the mission goal is to fly to the preset mid-term handover point. The target state of the aircraft at the mid-term handover point is:

[0147] Table 1 Target status of aircraft at the mid-term and final handover points

[0148]

[0149] During the training process, the parameters of the vehicle reentry point are randomly and uniformly sampled according to the parameter range shown in Table 2.

[0150] Table 2 Parameter range of the initial point of reentry of the aircraft

[0151]

[0152] (1) No disturbance at the re-entry point

[0153] The strategy trained by the PPO algorithm is compared with the prediction correction guidance method. The median value of the parameter range in Table 2 is used as the reentry point state, and the state change curve of the aircraft during reentry is obtained as follows: Figure 1-Figure 4 As shown in the figure, it can be seen that the strategy obtained by the PPO algorithm maintains a target accuracy close to or even better than that of the prediction correction guidance, while the roll angle curve is smoother and a completely different flight trajectory is explored, which shows significantly different characteristics from the classic flight trajectory in terms of latitude, longitude and altitude changes.

[0154] The error between the vehicle state and the target state at the end of the mission is shown in the following table. Compared with the predictive correction guidance method, the PPO algorithm achieves better speed control accuracy while maintaining similar position control (longitude and latitude).

[0155] Table 3 Error between the aircraft state and the target state at the last handover point

[0156]

[0157] (2) Reentry point single state quantity disturbance

[0158] In order to test the influence of different state quantity changes on the accuracy of the mid-term handover point, the state quantity of the aircraft at the re-entry point was perturbed by a single state quantity. Among the six state quantities shown in Table 2, 11 points were evenly selected within their parameter ranges for testing, with a total of 66 groups of experiments. The average state error of the mid-term handover point is shown in the following table. It can be seen that the PPO algorithm has a good generalization ability for the perturbation of the single state quantity at the re-entry point.

[0159] Table 4 Experimental results of single state quantity disturbance at reentry point

[0160]

[0161] (3) Full state quantity disturbance

[0162] In order to test the influence of the simultaneous change of multiple state quantities on the accuracy of the mid-term handover point, the state quantity of the aircraft at the reentry point is disturbed as a whole, and the maximum value, minimum value and median value of the six state quantities are taken in the parameter range shown in Table 2 to combine them, and 3 6 =729 groups of tests, the average state errors of the mid- and end-shift handover points are shown in the following table.

[0163] Table 5 Results of full variable disturbance experiment

[0164]

[0165]

[0166] The present invention designs an intelligent aircraft guidance method based on reinforcement learning. Under the condition of satisfying the constraints of dynamic equations, heat flux, overload, dynamic pressure, etc., the aircraft is guided to explore in the environment by setting effective rewards. The aircraft roll angle guidance model is trained by the proximal strategy optimization algorithm in reinforcement learning. The roll angle command is generated online based on real-time state information, which subverts the constraints of traditional guidance methods in terms of lateral corridors and explores completely different flight trajectories.

[0167] The simulation results show that the aircraft intelligent guidance method based on reinforcement learning can achieve control accuracy comparable to that of the predictive correction guidance algorithm, give full play to the wide-range flight advantages of the aircraft, further expand the flight profile, and at the same time have strong robustness to state disturbances, and achieve good control effects in various situations.

[0168] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. An aircraft intelligent guidance method based on reinforcement learning, characterized in that The steps include: S1. Define and obtain the observation space of the aircraft, wherein the observation space includes the aircraft state quantity, the aircraft state differential quantity, the difference between the aircraft state and the state of the mid-term handover point, the Euclidean distance between the longitude and latitude of the aircraft and the longitude and latitude of the mid-term handover point, and the difference between the aircraft energy and the terminal energy; S2. Define the action space of the reinforcement learning strategy model as the roll angle reference of the aircraft, use the proximal strategy to optimize the reinforcement learning model, and generate the roll angle reference of the aircraft online based on the real-time state space; S3, mapping the aircraft's roll angle reference output by the reinforcement learning strategy model to obtain the aircraft's theoretical roll angle; S4. The theoretical roll angle of the aircraft is truncated to obtain the final roll angle instruction actually used to control the aircraft. S5: Set the roll angle command Substitute the dynamics model to update the aircraft state and re-execute steps S1 to S5.

2. The method for intelligent aircraft guidance based on reinforcement learning according to claim 1, characterized in that: The calculation method for truncating the theoretical roll angle is: Among them, σ max is the constraint on the roll angle amplitude, and η is the safety factor.

3. The method for intelligent aircraft guidance based on reinforcement learning according to claim 2 is characterized in that The constraint σ on the roll angle amplitude max for: in, is the heel angle amplitude threshold under heat flux constraint; is the heel angle amplitude threshold under overload constraint; is the roll angle amplitude threshold under dynamic pressure constraint; m is the mass of the aircraft; g is the acceleration of gravity; V is the speed of the aircraft; r is the distance from the center of the earth; S ref is the characteristic area of ​​the aircraft; k Q is the heat flow rate model coefficient; is the maximum allowable heat flux at the stagnation point; C L and C D are lift coefficient and drag coefficient respectively; n max is the maximum permissible overload; g0 is the gravitational acceleration at sea level; q max is the maximum dynamic pressure.

4. The method for intelligent aircraft guidance based on reinforcement learning according to claim 1, characterized in that The proximal policy optimization reinforcement learning model includes an Actor model and a Critic model. The Actor model outputs the action of the aircraft according to the observation space of the aircraft, and the Critic model calculates the reward according to the observation space of the aircraft, and optimizes the Actor model with the maximization of the reward expectation as the objective function.

5. The method for intelligent aircraft guidance based on reinforcement learning according to claim 4 is characterized in that The objective function calculation formula is as follows: In the formula, R(s t ,a t ) is the reward function, indicating that the aircraft is in state s t Execute action a t The expected reward of the immediate reward obtained, including the lateral reward R h (s t ,a t ), vertical reward R v (s t ,a t ) and the final reward R f (s t ,a t ), κ t The discount factor for the current calculation period ranges from 0 to 1.

6. The method for intelligent aircraft guidance based on reinforcement learning according to claim 5, characterized in that The horizontal reward R h (s t ,a t ) is the sum of the latitude and longitude bonus and the direction bonus.

7. The method for intelligent aircraft guidance based on reinforcement learning according to claim 6, characterized in that: The direction reward r dir By calculating the actual velocity direction ψ and the line of sight angle ψ of the aircraft →tgt The difference between the two guides the aircraft to fly to the middle and last handover points along the line of sight. The calculation formula is as follows: Where, L togo is the remaining range of the aircraft; It is a hyperparameter used to regulate weight changes in direction rewards; The weight used to adjust the impact of the directional bonus on the aircraft; represents the heading angle ψ t With sight angle difference; in, is the expected longitude value of the mid-to-final handover point, θ is the actual longitude value of the mid-to-final handover point; φ is the actual longitude value of the mid-to-final handover point, is the expected latitude of the mid- and end-shift handover points, t represents the current calculation cycle, and t-1 represents the previous calculation cycle; The latitude and longitude bonus serves to guide the aircraft to fly close to the mid-term and final handover points, and the calculation formula is as follows: Among them, L togo is the remaining range of the aircraft, L * is the set distance threshold; It is a hyperparameter used to regulate the weight change in longitude and latitude rewards; is the weight used to adjust the influence of longitude and latitude bonus on the aircraft; θ is the actual longitude of the mid-term handover point; φ is the actual longitude of the mid-term handover point; Respectively represent the Euclidean distance between the aircraft longitude and latitude and the mid-end handover point longitude and latitude in the current calculation cycle and the previous calculation cycle.

8. The method for intelligent aircraft guidance based on reinforcement learning according to claim 5, characterized in that The vertical reward R v (s t ,a t ) is the sum of energy bonus, speed bonus and height bonus.

9. The method for intelligent aircraft guidance based on reinforcement learning according to claim 8, characterized in that: The energy reward r E The calculation formula is as follows: In the formula, Indicates that the mean is E μ , with a standard deviation of δ E From the Gaussian distribution of The probability of b V is the bias used to control the speed error, b h is the offset used to adjust the height error, V is the speed of the aircraft, R0 is the radius of the earth, and GM represents the gravitational constant; The speed reward r V The calculation formula is as follows: In the formula, Indicates that the mean value is V μ , with a standard deviation of δ V Get V+b from the Gaussian distribution V The probability of b V is the bias used to adjust the speed error; The height reward r h The calculation formula is as follows: In the formula, Indicates that the mean is h μ , with a standard deviation of δ h Get h+b from the Gaussian distribution h The probability of b h is the offset used to adjust the height error; This is the hyperparameter used to regulate the weight change in this reward, and h is the flight altitude of the aircraft.

10. The method for intelligent aircraft guidance based on reinforcement learning according to claim 9, characterized in that The final reward r tgt When the aircraft arrives at the mid-term and final handover point, the calculation formula is as follows: In the formula, ω r ,ω θ ,ω φ ,ω V are the influence coefficients used to balance the impact of aircraft distance, longitude, latitude, and speed amplitude, ε r ,ε θ ,ε φ ,ε V are the hyperparameters used to calculate the relative error of the aircraft distance, longitude, latitude, and speed. The reward at the end of the rest of the time is 0.

Images