Method for optimizing reentry trajectory of hypersonic variant aircraft driven by reinforcement learning

Abstract

The invention discloses a reinforcement learning driven hypersonic variant aircraft reentry trajectory optimization method, and belongs to the field of hypersonic aircraft and trajectory optimization. The implementation method comprises the following steps: establishing a waverider-like hypersonic variant aircraft parameterized model comprising a waverider fuselage and two deformed wings based on a category shape conversion (CST) method; the influence of factors such as flight height H, flight Mach number Ma, attack angle alpha, inner wing section sweepback x1, outer wing section sweepback x2 and the like is considered, a Kriging agent model is established, and the aerodynamic performance of the hypersonic speed variant aircraft is obtained. Based on a DDPG reinforcement learning algorithm, according to flight characteristics of a hypersonic speed variant aircraft in a reentry section flight process, and in combination with expert experience and dynamics, start and end states, heat flow, overload and dynamic pressure constraints in the flight process, training of a reentry section trajectory optimization model driven by the DDPG algorithm is realized. Through flight simulation prediction of different hypersonic speed variant aircrafts, variant decisions are given in real time, and reentry section trajectory optimization of the hypersonic speed variant aircrafts is realized.

Description

Technical Field

[0001] The invention relates to a method for optimizing the reentry trajectory of a hypersonic variant aircraft driven by reinforcement learning, and belongs to the field of hypersonic aircraft and trajectory optimization. Background Art

[0002] Due to the inherent characteristics of high flight speed and high flight altitude, hypersonic aircraft have strong penetration capability, rapid response capability and excellent concealment capability, and can perform specific combat tasks such as competing for high-level space, attacking long-range targets and monitoring wide-area targets. It is precisely because of the diversity of executable tasks that hypersonic aircraft are a major development direction of countries around the world in the aerospace field in the 21st century. Due to the characteristics of hypersonic aircraft flight missions such as "cross-airspace" and "wide speed range", the design method of layout scheme design, overall scheme design and aerodynamic shape design based on a single mission condition is very mature, but not applicable enough. As an emerging hotspot in the field of aerospace research, variant technology can adaptively adjust the shape according to the flight state in complex environments and emergencies that cannot be predicted in advance to improve the aerodynamic characteristics of the aircraft, optimize flight performance, and improve the overall performance of the entire flight envelope.

[0003] Reentry flight is an extremely important flight process of a hypersonic vehicle in the entire flight process. It starts from the separation point of the rocket and performs unpowered flight until the dive pressure point enters the terminal guidance stage. The reentry phase accounts for the largest proportion in the entire trajectory, has the widest domain, and has the most complex constraints. It is the flight phase that has the greatest impact on the overall trajectory performance. However, the state quantities of hypersonic variant aircraft that introduce variant technology are complicated, and the influencing mechanism is difficult to distinguish, making it difficult for traditional optimization methods to give appropriate variant decisions and achieve good reentry trajectory optimization. In addition, the drastically changing complex environment and the requirements of its own performance limit the implementation of variant schemes, and it is not suitable for large-scale and rapid deformation. Therefore, it is of great significance to design a decision maker that determines the deformation of the aircraft based on state quantities such as flight altitude, speed, angle of attack, and current aerodynamic shape, which can deeply tap the design potential of hypersonic aircraft, improve the flight performance of the aircraft in the reentry trajectory, and enhance its battlefield survivability and mission completion capabilities.

[0004] With the development of artificial intelligence and neural networks, reinforcement learning can be used to perform iterative training step by step through the rewards obtained by the mechanism to perform actions, and make decisions that adapt to complex environments. In decision-making problems, reinforcement learning can well solve decision-making problems including obstacle avoidance of autonomous mobile robots and autonomous action planning of robotic arms. Trajectory optimization problems similar to these problems are also a type of state transition problem. The action is decided by the body being in a certain state to reach the next state, and the benefits obtained from this state change are used to train the decision model. Therefore, the Deep Deterministic Policy Gradient (DDPG) decision algorithm based on policy gradient descent that combines reinforcement learning with neural networks is used to give variant decisions to achieve trajectory optimization of hypersonic variant aircraft in the reentry phase, which has good development prospects. Summary of the invention

[0005] In order to solve the problems of poor flight performance of hypersonic variant aircraft in the reentry phase and long calculation time of traditional optimization algorithms, the present invention discloses a reinforcement learning driven hypersonic variant aircraft reentry trajectory optimization method to solve the following technical problems: realizing parametric modeling of hypersonic variant aircraft and rapid construction of different variant configurations; constructing a Kriging proxy model according to the aerodynamic simulation sampling points of the geometric model to predict aerodynamic performance under different flight configurations and flight environments; training the optimization model based on the DDPG reinforcement learning algorithm, constructing a decision maker with the maximum lift-to-drag ratio as the main goal, and finally obtaining a reentry optimization trajectory that maintains a high lift-to-drag ratio and satisfies the constraints in most states.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] The invention discloses a method for optimizing the reentry trajectory of a hypersonic variant aircraft driven by reinforcement learning. A parametric model of a waverider-like hypersonic variant aircraft including a waverider fuselage and two-stage deformable wings is established based on a class shape transformation method (CST). Considering the influence of factors such as flight altitude H, flight Mach number Ma, angle of attack α, inner wing section sweep angle χ1, outer wing section sweep angle χ2, etc., a Kriging proxy model is established to obtain the aerodynamic performance of the hypersonic variant aircraft. Based on the DDPG reinforcement learning algorithm, according to the flight characteristics of the hypersonic variant aircraft during the reentry flight, combined with expert experience and dynamics, initial and final states, heat flow, overload and dynamic pressure constraints during the flight, the DDPG algorithm-driven reentry trajectory optimization model training is realized. Through flight simulation prediction of different hypersonic variant aircraft under practical applications, variant decisions are given in real time to achieve the optimization of the reentry trajectory of the hypersonic variant aircraft.

[0008] The hypersonic speed variant aircraft reentry phase trajectory optimization method driven by reinforcement learning disclosed in the present invention comprises the following steps:

[0009] Step A: Establish a parametric model of the hypersonic variant aircraft.

[0010] Step A is implemented as follows:

[0011] Step A-1: ​​Construct the baseline shape and variant form. According to the characteristics of the reentry flight environment of the hypersonic variant aircraft, a waverider-like configuration is adopted, that is, a waverider fuselage combined with a hypersonic wing shape. Due to the thin atmosphere during flight, the variable sweep angle in large-scale deformation is used to improve the performance of the variant.

[0012] Step A-2: Parametric description of the geometric model. Based on the standard three-dimensional CST description equation combined with the waverider fuselage shape characteristics, combined with the leading edge line located in the horizontal plane and the symmetric relationship of the longitudinal plane, the Z coordinate expression of the upper and lower surfaces of the initial fuselage shape is obtained, that is, the parametric model of the hypersonic variant aircraft.

[0013]

[0014] In the formula, H′ u is the control parameter of the upper surface of the spanwise section, is the classification function, M′ u is the control parameter of the upper surface of the body axis section, N u ′ is the upper surface thickness control parameter, η is the normalized X and Y coordinates. Similarly, the subscript l is the control parameter related to the lower surface.

[0015] Step B: Establish a six-degree-of-freedom trajectory model for a hypersonic variant vehicle.

[0016] Step B is implemented as follows:

[0017] Step B-1: In the reentry phase flight environment of a hypersonic variant aircraft, it is necessary to consider the changes in the speed of sound and atmospheric density. Therefore, the aerodynamic model of the hypersonic variant aircraft is jointly affected by the flight altitude H, the flight Mach number Ma, the angle of attack α, the inner wing section sweep angle χ1, and the outer wing section sweep angle χ2. According to the flight environment and the structural characteristics of the aircraft, a certain number of sampling points are generated within the variable range using the Latin hypercube test design method:

[0018] [H,Ma,α,χ1,χ2]=lhsdesign(N) (2)

[0019] Where N is the number of sampling points.

[0020] Through a series of automated processes such as drawing grids, importing aerodynamic simulation software to solve aerodynamic forces, and collecting aerodynamic data, multiple sets of discrete lift and drag coefficient aerodynamic coefficient data are obtained; the lift and drag coefficient aerodynamic coefficient data is the lift coefficient C L and the drag coefficient C D .

[0021] When solving the aerodynamic data in step B, the aerodynamic data is solved for the aircraft in similar working conditions through the grid independence test, and the correctness of the collected aerodynamic data is verified through the experimental data comparison method to obtain correct aerodynamic data.

[0022] Step B-2: For the lift coefficient C L , drag coefficient C D Construct Kriging surrogate models respectively.

[0023]

[0024] In the formula, They are the Kriging proxy model fitting functions for the lift coefficient and the drag coefficient respectively.

[0025] Leave one out cross validation (LOOCV) was performed on the established Kringing surrogate model, and the multiple correlation coefficient was given.

[0026]

[0027] Among them, C Li , C Di is the lift coefficient and drag coefficient obtained by the proxy model at each sampling point; if any complex correlation coefficient is less than 0.8, return to step B-1 to increase the number of sampling points; otherwise, proceed to step B-3;

[0028] Step B-3: According to the flight environment characteristics experienced by the hypersonic variant aircraft during the reentry phase, through the description of the coordinate system and the atmospheric model, the state quantities are selected as the rotating earth relative speed v, the track inclination angle γ, the heading angle ψ, the longitude λ, the latitude φ, and the altitude H to construct a six-degree-of-freedom ballistic model of the hypersonic variant aircraft.

[0029]

[0030] In the formula, R e is the radius of the earth, g is the gravitational acceleration, which adopts the nominal value; σ is the roll angle, which is taken as 0; L is the lift acceleration, and D is the drag acceleration, which needs to be calculated; C v2 is the relative velocity v direction component of the involved acceleration term, C γ1 and Cγ2 are the γ-direction components of the track inclination angle of the Coriolis acceleration term and the entrained acceleration term, respectively, C ψ1 and C ψ2 They are the heading angle ψ direction components of the Coriolis acceleration term and the entrained acceleration term, respectively, and the expressions are:

[0031]

[0032] In the formula, ω e is the angular velocity of the Earth’s rotation, using the nominal value.

[0033] The aerodynamic data obtained using formula (3) gives the lift acceleration L and the drag acceleration D.

[0034]

[0035] In the formula, dynamic pressure q = 0.5ρv 2 , ρ is the atmospheric density given by the standard atmospheric model, and S is the reference area of ​​the wing.

[0036] Step C: Determine the objects pointed to by the state quantity, action quantity, constraints on the body and initial and final conditions in the reinforcement learning algorithm; determine the form of the reward function in the reinforcement learning algorithm and the experience pool composed of expert experience and exploration experience; determine the network architecture setting and initial quantity assignment in the reinforcement learning algorithm; train the re-entry trajectory optimization model driven by the reinforcement learning algorithm to obtain the trained re-entry trajectory optimization model driven by the reinforcement learning algorithm, and perform trajectory optimization according to the trained re-entry trajectory optimization model driven by the reinforcement learning algorithm.

[0037] Step C is implemented as follows:

[0038] Step C-1: Set the inner wing section sweep angle χ1, outer wing section sweep angle, altitude H, track inclination angle γ, Mach number Ma and angle of attack α as the basic element state quantity parameters s required in the DDPG reinforcement learning algorithm.

[0039] s=[χ1,χ2,H,γ,Ma,α] (8)

[0040] The change in the inner wing section sweep angle Δχ1 and the change in the outer wing section sweep angle Δχ2 are set as the basic element action amount a required in the DDPG reinforcement learning algorithm.

[0041] a=[Δχ1,Δχ2] (9)

[0042] The state parameter attack angle α changes in stages with the speed as shown in formula (10).

[0043]

[0044] Among them, α max is the maximum angle of attack, αmax ( K ) is the angle of attack at the maximum lift-to-drag ratio, V1 is the initial speed at the maximum angle of attack, V2 is the speed corresponding to the maximum lift-to-drag ratio, and the other parameters are as follows:

[0045]

[0046] The maximum heat flux constraint, overload constraint and dynamic pressure constraint are considered; based on the standard drag acceleration profile, a reentry boundary constraint model is given.

[0047]

[0048] In the formula, q max is the maximum heat flow constraint, n′ max is the overload constraint, q′ max is the dynamic pressure constraint, D q,max is the drag acceleration under the maximum heat flux constraint, D n′,max is the resistance acceleration under overload constraint, D q′,max is the drag acceleration under dynamic pressure constraint.

[0049] Considering the flight characteristics, flight environment characteristics and the limitations of the airframe during the reentry phase of the hypersonic variant aircraft, the take-off parameters of the airframe, namely the take-off speed v0 and the take-off height H0, are given; the initial configuration parameters, namely the inner wing section sweep angle χ 10 , outer wing section sweep angle χ 20 ; The initial posture parameters of the aircraft are the track inclination angle γ0, the heading angle ψ0; the initial position parameters of the aircraft are the longitude λ0, the latitude φ0; the other parameters are the aircraft mass m, the wing reference area S, the earth radius R e , rotation speed ω e and the gravitational acceleration g on the Earth's surface.

[0050] Take the reentry end condition to terminate the training and set the termination speed v s , end height H s .

[0051] Step C-2: Establish the reward function R required for reinforcement learning training, and build an experience pool based on the reward function R that combines expert experience.

[0052] Firstly, considering the objective function of maximum lift-to-drag ratio, the smoothness requirement of deformation and related constraint requirements, the reward function R is established.

[0053] R=λ1C L / D -λ2(Δχ1+Δχ2)-λ3ΔD q,n′,q′ (13)

[0054] Among them, λ1, λ2, λ3 are preset weight coefficients, C L / D is the lift-to-drag ratio, ΔDq,n′,q′ It is the excess of heat flow, overload and dynamic pressure constraints.

[0055] During the training process, the experience pool sampled for training is divided into exploration experience E explore and expert experience expert , introduce parameter α e As the ratio of expert experience sampling to total sampling

[0056]

[0057] Where M is the number of samples taken in the corresponding experience pool.

[0058] Exploration experience E in the experience pool explore , is the experience formed by the interaction between the action output by the neural network and the environment of the agent in the training round. It is stored in each round of training until the upper limit of the experience pool is reached and then updated. expert Since the lift-to-drag ratio is the main reward, the experience of the two control variables of the inner and outer wing section sweep angles that achieve the maximum lift-to-drag ratio under the current state is set.

[0059] Step C-3: Give the reinforcement learning network architecture settings.

[0060] Establish an action network Actor, which consists of two networks Actor_e and Actor_t with the same initial hyperparameters and structure. The former gives the output action a when the input state s is given, and the latter outputs the next action a' when the input next state s' is given. Actor_e is updated in the form of normal update, and Actor_t is updated in the form of soft update. The network architecture includes five layers, including input layer, three hidden layers and output layer. Except for the last two layers, sigmoid is used as the activation function. Relu is used as the activation function between the remaining layers, and the connection mode is full connection between neurons. The number of input neurons is set to the state dimension, and the number of output neurons is set to the action dimension.

[0061] The evaluation network Critic is established, which consists of two networks with the same initial hyperparameters and structure, namely Critic_e and Critic_t. Both networks give output value functions when the state quantity s and the action quantity a are input. Critic_e is updated in the form of normal update, while Critic_t is updated in the form of soft update. The network architecture includes five layers, including input layer, three hidden layers and output layer. Except for the last two layers, sigmoid is used as the activation function. Relu is used as the activation function between the remaining layers, and the connection method is full connection between neurons. The number of input neurons is set to the total dimension of action quantity and state quantity, and the number of output neurons is the dimension of value function.

[0062] Normal update uses the weight training form of gradient descent method to update the weight training as follows:

[0063]

[0064] Among them, L is the damage function, θ is the network hyperparameter, E is the expectation function, R is the reward function, γ e is the discount factor and Q is the value function.

[0065] The normal update format is as follows:

[0066]

[0067] Set the soft update coefficient τ, the original network hyperparameter θ and the updated hyperparameter θ′, and the update form is as follows:

[0068] θ'←τθ+(1-τ)θ',τ≤1 (17)

[0069] After the network architecture is set up, the initial state value s0 and the initial position state value X0 are determined according to the initial data given in step C-1:

[0070]

[0071] Among them, Ma0 and α0 are derived and calculated from v0.

[0072] Step C-4: Perform reentry trajectory optimizer training under the complete DDPG algorithm architecture;

[0073] The steps for network parameter training are as follows:

[0074] At time t, there exists an action value a t-Δt , state value s t With the position state value X t :

[0075]

[0076] The subscript t represents the value of the variable at time t.

[0077] The variable H t ,α t ,Ma t ,x 1t ,x 2t Substitute into formula (3) to obtain the aerodynamic coefficient C at the current moment Lt ,C Dt ; The current aerodynamic coefficient C Lt ,C Dt Substitute into formula (7) to obtain the lift acceleration L at the current moment: t , drag acceleration D t ; Lt , D t With X t Substituting into equation (5), the Runge-Kutta 4th-order integrator is used to obtain the position state value X at the next moment: t+Δt , Δt is the specific time step set; the position state value X t+Δt Medium t+Δt Substitute into formula (10) to obtain α t+Δt , substitute into the standard model to obtain Ma t+Δt ; will s t Get a from the incoming network Actor t , a t As follows:

[0078] a t =[Δχ 1t ,Δχ 2t ] (20)

[0079] There is a relationship:

[0080]

[0081] A t Substituting into equation (21), the Runge-Kutta 4th-order integrator is used to obtain the next moment χ 1t+Δt ,x 2t+Δt , integrated position state value X t+Δt Parameter H has been obtained in t+Δt ,γ t+Δt Given t+Δt ; Set the action value a t With the state value s t The value function is passed into the network Critic, and the network Actor is updated to complete the training process at time t. At t = 0, the following formula is obtained:

[0082]

[0083] After the initial value is passed in to start training, the network parameter training steps are repeated until the training end condition is reached, that is:

[0084]

[0085] After the training conditions are met, this training process is recorded as one training, and the number of training times i = i + 1; set the training iteration upper limit i max , when i=i max The training is ended when , and the re-entry trajectory optimization model driven by the trained reinforcement learning algorithm is obtained. The trajectory optimization is performed according to the re-entry trajectory optimization model driven by the trained reinforcement learning algorithm.

[0086] Step D: Consider the reentry flight of hypersonic variant aircraft under different terminal conditions, determine the specific characteristic parameters of the aircraft, and optimize the trajectory according to the reentry trajectory optimization model driven by the reinforcement learning algorithm trained in step C, so as to improve the accuracy and efficiency of reentry trajectory optimization and realize high-precision variant decision control of hypersonic variant aircraft during reentry flight. Afterwards, compare the simulation analysis of other methods under the same conditions to highlight the flight benefits of the established optimization model.

[0087] Step D is implemented as follows:

[0088] Step D-1: Optimize the trajectory of three different simulation conditions. The hypersonic variant aircraft has a small trajectory difference in the initial descent phase due to the slight difference in aerodynamic force, and the initial point is the rocket-rocket separation point, which is transported by a booster rocket to a near-space at an altitude of about 60km. The simulation termination condition is that the hypersonic variant aircraft reaches the dive pressure point determined by the mission characteristics, and the aircraft is ready to dive and enter the terminal guidance phase to reach the target mission point. The initial conditions of the design simulation are the same as the initial conditions of the training, and the different trajectories of the terminal termination conditions are optimized to give the variant configuration to the hypersonic aircraft during the reentry phase flight. Flight benefits.

[0089] Step D-2: Further explore the established DDPG algorithm driven reentry trajectory optimization model to compare the benefits obtained by the hypersonic variant aircraft during the reentry flight process, and conduct flight simulation based on multiple methods. For the optimization method, SLSQP, Constrained Optimization by LinearApproximation (COBYLA) and Trust Region Constrained Algorithms (trust-constr) are used to calculate the angle corresponding to the maximum lift-to-drag ratio under the established lift coefficient proxy model and drag coefficient proxy model, and substitute it into the motion equation to calculate the state quantity at the next moment, and give the comparison optimization simulation results. For the direct control method, the pseudo-spectral method is used, which performs well in solving trajectory optimization problems. The LG points of the roots of the N-order Legendre polynomial are used for the matching points. When the starting and ending conditions and flight state parameters are the same as those of the previous simulation, the flight results of the entire simulation are compared.

[0090] Beneficial effects:

[0091] 1. In view of the current research status of the lack of analytical models for hypersonic variant aircraft, the present invention discloses a method for optimizing the reentry trajectory of a hypersonic variant aircraft driven by reinforcement learning, establishes a parametric model of a hypersonic variant aircraft based on the CST method, determines the wing deformation configuration with variable sweep as the main deformation form, and calculates aerodynamic data with the help of engineering software ICEM and high-precision aerodynamic simulation software SU2. On this basis, a reentry six-degree-of-freedom trajectory model is given to improve the overall modeling of the hypersonic variant aircraft and improve the modeling accuracy of the hypersonic variant aircraft.

[0092] 2. The present invention discloses a reinforcement learning driven hypersonic variant aircraft reentry trajectory optimization method, which replaces the high-precision aerodynamic analysis model of the hypersonic variant aircraft with a proxy model to achieve rapid prediction of the aerodynamic parameters of the aircraft during flight, thereby reducing the number of analyses of the time-consuming aerodynamic analysis model required for optimization and improving the efficiency of reentry trajectory optimization.

[0093] 3. The present invention discloses a method for optimizing the reentry trajectory of a hypersonic variant aircraft driven by reinforcement learning. Aiming at the requirements of reentry trajectory optimization of a hypersonic variant aircraft, the invention combines the applicable scenarios and specific characteristics of various algorithms in reinforcement learning and adopts the DDPG algorithm that can be applied to continuous states and continuous actions. The established DDPG algorithm-driven hypersonic variant aircraft reentry trajectory optimization model is trained in combination with expert experience to achieve millisecond-level variant decision control of hypersonic variant aircraft during reentry flight, which is versatile, efficient and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 This is a schematic diagram of the overall layout of a hypersonic variant aircraft, where: Figure 1 (a) is the side view of the initial shape of the fuselage. Figure 1 (b) is the rear view of the initial shape of the fuselage. Figure 1 (c) is a top view of the initial shape of the fuselage. Figure 1 (d) is the top view of the half mold of the whole machine. ;

[0095] Figure 2 It is a schematic diagram of the wing shape transformation of a hypersonic variant aircraft;

[0096] Figure 3 Flowchart for pneumatic data collection;

[0097] Figure 4 A reentry trajectory optimization flow chart built for DDPG;

[0098] Figure 5 The comparison chart of various parameters before and after trajectory optimization at the termination speed v1 = 3000m / s and the termination height H1 = 30km;

[0099] Figure 6 The comparison chart of various parameters before and after trajectory optimization at the termination speed v1 = 3500m / s and the termination height H1 = 35km;

[0100] Figure 7 This is a comparison chart of various parameters before and after trajectory optimization at the termination speed v1 = 2500m / s and the termination height H1 = 25km;

[0101] Figure 8 This is a comparison chart of lift-to-drag ratio under the same termination conditions;

[0102] Fig. 9 This is a comparison chart of parameters before and after SLSQP optimization;

[0103] Fig.10 This is a comparison chart of various parameters before and after COBYLA optimization;

[0104] Fig.11 This is a comparison chart of parameters before and after trust-constr optimization. DETAILED DESCRIPTION

[0105] In order to better illustrate the purpose and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0106] like Figure 4 As shown, the present embodiment discloses a method for optimizing the reentry trajectory of a hypersonic variant aircraft driven by reinforcement learning, and the specific implementation steps are as follows:

[0107] Step A: Establish a parametric model of a hypersonic variant aircraft. During the reentry phase, different flight configurations need to be adopted according to different flight environments to obtain the most suitable aerodynamic shape. Based on the reference shape, determine the achievable variant forms and give different variant configurations. In order to accurately draw the shape of a hypersonic variant aircraft and quickly draw it after changing the geometric parameters, it is necessary to parametrically describe the shape of the aircraft.

[0108] Step A is implemented as follows:

[0109] Step A-1: ​​Construct the baseline shape and variant form. According to the characteristics of the reentry flight environment of the hypersonic variant aircraft, a waverider-like configuration is adopted, that is, a waverider fuselage is combined with a hypersonic wing shape such as Figure 1 As shown in the figure. Due to the thin atmosphere during flight, the variable sweep angle in large-scale deformation is used to improve the deformation performance. For the sweep angle deformation, the shear-type rigid-flexible coupling skin and the slide-rail skeleton structure are used to achieve the effect as shown in the figure. Figure 2 shown.

[0110] Step A-2: Parametric description of the geometric model. In order to accurately draw the shape of the hypersonic variant aircraft and quickly draw it after changing the geometric parameters, it is necessary to parametrically describe the shape of the aircraft. Based on the standard three-dimensional CST description equation combined with the characteristics of the waverider-like fuselage shape, combined with the symmetry relationship between the leading edge line in the horizontal plane and the longitudinal plane, the Z coordinate expression of the upper and lower surfaces of the initial shape of the fuselage is obtained.

[0111]

[0112] In the formula, the surface control parameter H′ on the spanwise section is u =0.446, is the category function, and the surface control parameter M′ on the body axis section u =0.752, upper surface thickness control parameter N' u =0.796, the control parameter of the lower surface of the body axis section M′ l =0.973, upper surface thickness control parameter N′ l =1.948, η is the normalized X and Y coordinates.

[0113] Step B: Establish a six-degree-of-freedom trajectory model for a hypersonic variant vehicle.

[0114] Step B is implemented as follows:

[0115] Step B-1: In the reentry phase flight environment of the hypersonic variant aircraft, it is necessary to consider the changes in the speed of sound and atmospheric density. Therefore, the aerodynamic model of the hypersonic variant aircraft is jointly affected by the flight altitude h∈[30,60]km, the flight Mach number Ma∈[8,25], the angle of attack α∈[10°,25°], the inner wing section sweep angle χ1∈[18°,30°], and the outer wing section sweep angle χ2∈[0°,χ1]. According to the flight environment and the characteristics of the aircraft structure, a certain number of sampling points are generated within the variable range using the Latin hypercube test design method:

[0116] [H,Ma,α,χ1,χ2]=lhsdesign(N) (25)

[0117] The number of initial sampling points N is set to 400.

[0118] A series of automated processes are carried out through drawing grids based on the obtained sampling point information, importing aerodynamic simulation software to solve aerodynamic forces, and collecting aerodynamic data. Figure 3 As shown, multiple sets of discrete lift and drag coefficient aerodynamic coefficient data are obtained; the lift and drag coefficient aerodynamic coefficient data are lift coefficient C L and the drag coefficient C D .

[0119] When solving aerodynamic data, the correctness of the collected aerodynamic data is verified by grid independence test, solving aerodynamic data for aircraft with similar working conditions and comparing with experimental data.

[0120] Step B-2: Since the design variables of the established variant aerodynamic model are of high dimension and the polynomial expression between the lift and drag coefficient and each variable cannot be clearly defined, the lift coefficient C L , drag coefficient C D Construct Kriging surrogate models respectively.

[0121]

[0122] In the formula, They are the Kriging proxy model fitting functions for the lift coefficient and the drag coefficient respectively.

[0123] In order to determine the accuracy of the proxy model in predicting aerodynamic performance, the established Kringing proxy model is subjected to leave-one-out validation and the complex correlation coefficient is given:

[0124]

[0125] Among them, C Li , C Di is the lift coefficient and drag coefficient obtained by the proxy model at each sampling point; if any complex correlation coefficient is less than 0.8, return to step B-1 to increase the number of sampling points; otherwise, proceed to step B-3;

[0126] The complex correlation coefficient R of the drag coefficient proxy model is obtained 2 =0.891, lift coefficient proxy model complex correlation coefficient R 2 =0.919, both above 0.8, which is considered to be able to provide relatively accurate aerodynamic prediction and achieve complete data support.

[0127] Step B-3: According to the flight environment characteristics experienced by the hypersonic variant aircraft during the reentry phase, through the description of the coordinate system and the atmospheric model, the state quantities are selected as the rotating earth relative speed v, the track inclination angle γ, the heading angle ψ, the longitude λ, the latitude φ, and the altitude H to construct a six-degree-of-freedom ballistic model of the hypersonic variant aircraft.

[0128]

[0129] Where, the radius of the earth is R e is 6378km, and the acceleration due to gravity g is 9.81m / s 2 ; The roll angle σ is 0; L is the lift acceleration, D is the drag acceleration, which needs to be calculated; C v2 is the relative velocity v direction component of the involved acceleration term, Cγ1 and C γ2 are the γ-direction components of the track inclination angle of the Coriolis acceleration term and the entrained acceleration term, respectively, C ψ1 and C ψ2 They are the heading angle ψ direction components of the Coriolis acceleration term and the entrained acceleration term, respectively, and the expressions are:

[0130]

[0131] Where, the Earth's rotation angular velocity ω e 7.29e-5rad / s

[0132] The aerodynamic data obtained using formula (3) gives the lift acceleration L and the drag acceleration D.

[0133]

[0134] In the formula, dynamic pressure q = 0.5ρv 2 , ρ is the atmospheric density given by the standard atmospheric model. S is the reference area of ​​the wing.

[0135]

[0136] Where, ρ0 = 1.225 kg / m 3 ,h s =(RT0) / g0 is the constant coefficient related to atmospheric height, which is 7254.2m.

[0137] Step C: Determine the state quantity, action quantity, constraints on the body and the initial and final conditions in the reinforcement learning algorithm; determine the form of the reward function in the reinforcement learning algorithm and the experience pool composed of expert experience and exploration experience; determine the network architecture setting in the reinforcement learning algorithm and the initial quantity assignment; finally, give the training steps of the re-entry trajectory optimization model driven by the reinforcement learning algorithm to complete the training. The process is as follows Figure 5 shown.

[0138] Step C is implemented as follows:

[0139] Step C-1: Set the inner wing section sweep angle χ1, outer wing section sweep angle, altitude H, track inclination angle γ, Mach number Ma and angle of attack α as the basic element state quantity parameters s required in the DDPG reinforcement learning algorithm.

[0140] s=[χ1,χ2,H,γ,Ma,α] (32)

[0141] The change in the inner wing section sweep angle Δχ1 and the change in the outer wing section sweep angle Δχ2 are set as the basic element action amount a required in the DDPG reinforcement learning algorithm.

[0142] a=[Δχ1,Δχ2] (33)

[0143] The angle of attack α, a key parameter affecting the aerodynamic effect of hypersonic variant aircraft, is not included in the training of the reinforcement learning DDPG algorithm. The angle of attack changes with speed in stages as follows:

[0144]

[0145] Among them, the maximum angle of attack α max The maximum lift-to-drag ratio is 20°, and the attack angle α max(K) The maximum angle of attack initial speed is V1 = 5500m / s, the maximum lift-to-drag ratio corresponding speed is V2 = 5000m / s, and the other parameters are as follows:

[0146]

[0147] The maximum heat flux constraint, overload constraint and dynamic pressure constraint are considered; based on the standard drag acceleration profile, a reentry boundary constraint model is given.

[0148]

[0149] Where, the maximum heat flux constraint q max =1.5×10 7 W / m 2 , overload constraint n′ max =2, dynamic pressure constraint q′ max =90KPa, D q,max is the drag acceleration under the maximum heat flux constraint, D n′,max is the resistance acceleration under overload constraint, D q′,max is the drag acceleration under dynamic pressure constraint.

[0150] Considering the flight characteristics, flight environment characteristics and the limitations of the airframe during the reentry phase of the hypersonic variant aircraft, the airframe take-off parameters are given: take-off speed v0 = 6200m / s, take-off altitude H0 = 60km; the airframe initial configuration parameters are the inner wing sweep angle χ 10 =18°, outer wing section sweep angle χ 20 =0°; initial posture parameters of the aircraft: track inclination angle γ0 = 0°, heading angle ψ0 = 0°; position parameters of the aircraft: longitude λ0 = 0°, latitude φ0 = 0°; other parameters: aircraft mass m = 5000kg, wing reference area S = 5.47m 2 、Earth radius R e , rotation speed ω e and the gravitational acceleration g on the Earth's surface.

[0151] Take the reentry end condition to terminate the training and set the termination speed v s =3000m / s, end height H s =30km.

[0152] Step C-2: Establish the reward function R required for reinforcement learning training to build an experience pool that combines expert experience.

[0153] Firstly, considering the objective function of maximum lift-to-drag ratio, the smoothness requirement of deformation and related constraint requirements, the reward function R is established.

[0154] R=λ1C L / D -λ2(Δχ1+Δχ2)-λ3ΔD q,n′,q′ (37)

[0155] Among them, the preset weight coefficients λ1=1, λ2=0.2, λ3=10, C L / D is the lift-to-drag ratio, ΔD q,n′,q′ It is the excess of heat flow, overload and dynamic pressure constraints.

[0156] For the experience pool setting, which is an important component of the reinforcement learning DDPG algorithm, considering the large span of the state space and the complex variables of the hypersonic variant aircraft during the reentry phase, and the long-term continuous action makes the action space very large, it is difficult for the intelligent agent to effectively understand the whole process through simple exploration learning. An experience pool combined with expert experience is established for training.

[0157] During the training process, the experience pool sampled for training is divided into exploration experience E explore and expert experience expert , introduce the parameter α of the ratio of expert experience sampling to total sampling e 0.2

[0158]

[0159] Among them, the total number of samples in the corresponding experience pool is M = M(E expert )+M(E explore ) is 1024.

[0160] Exploration experience E in the experience pool explore , is the experience formed by the interaction between the action output by the neural network and the environment of the agent in the training round. It is stored in each round of training until the upper limit of the experience pool is reached and then updated. expert Since the lift-to-drag ratio is the main reward, the experience of the two control variables of the inner and outer wing section sweep angles that achieve the maximum lift-to-drag ratio under the current state is set.

[0161] In the round of collecting expert experience, the neural network still outputs and executes the actions. At the same time, SLSQP is used to search for the sweep angle corresponding to the maximum lift-to-drag ratio under the aerodynamic proxy model at each time length, and the expert experience of this round is stored.

[0162] Step C-3: Give the reinforcement learning network architecture settings.

[0163] The action network Actor is established, which consists of two networks Actor_e and Actor_t with the same initial hyperparameters and structures. The former gives the output action a when the state quantity s is input, and the latter outputs the next action a' when the next state quantity s' is input. Actor_e is updated in the form of normal update, and Actor_t is updated in the form of soft update. The network architecture includes five layers, including input layer, three hidden layers and output layer. Except for the last two layers, sigmoid is used as the activation function. Relu is used as the activation function between the remaining layers, and the connection method is full connection between neurons. The number of input neurons is set to the state dimension of 6, and the number of output neurons is set to the action dimension of 2.

[0164] The evaluation network Critic is established, which consists of two networks with the same initial hyperparameters and structure, namely Critic_e and Critic_t. Both networks give output value functions when the state quantity s and the action quantity a are input. Critic_e is updated in the form of normal update, while Critic_t is updated in the form of soft update. The network architecture includes five layers, including input layer, three hidden layers and output layer. Except for the last two layers, sigmoid is used as the activation function. Relu is used as the activation function between the remaining layers, and the connection method is full connection between neurons. The number of input neurons is set to the total dimension of action quantity and state quantity 8, and the number of output neurons is the dimension of value function 1.

[0165] Normal update uses the weight training form of gradient descent method to update the weight training as follows:

[0166]

[0167] Among them, L is the damage function, θ is the network hyperparameter, E is the expectation function, R is the reward function, γ e is the discount factor and Q is the value function.

[0168] The normal update format is as follows:

[0169]

[0170] Set the soft update coefficient τ = 0.2, the original network hyperparameter θ and the updated hyperparameter θ′, and the update form is as follows:

[0171] θ'←τθ+(1-τ)θ',τ≤1 (41)

[0172] After the network architecture is set up, the initial state value s0 and the initial position state value X0 are determined according to the initial data given in step C-1:

[0173]

[0174] Among them, Ma0 and α0 are derived and calculated from v0.

[0175] Step C-4: Perform reentry trajectory optimizer training under the complete DDPG algorithm architecture;

[0176] The steps for network parameter training are as follows:

[0177] At time t, there exists an action value a t-Δt , state value s t With the position state value X t :

[0178]

[0179] The subscript t represents the value of the variable at time t.

[0180] The variable H t ,α t ,Ma t ,x 1t ,x 2t Substitute into formula (3) to obtain the aerodynamic coefficient C at the current moment Lt ,C Dt ; The current aerodynamic coefficient C Lt ,C Dt Substitute into formula (7) to obtain the lift acceleration L at the current moment: t , drag acceleration D t ; L t , D t With X t Substituting into equation (5), the Runge-Kutta 4th-order integrator is used to obtain the position state value X at the next moment: t+Δt , Δt is the specific time step set; the position state value X t+Δt Medium t+Δt Substitute into formula (10) to obtain α t+Δt , substitute into the standard model to obtain Ma t+Δt ; will s t Get a from the incoming network Actor t , a t As follows:

[0181] a t =[Δχ 1t ,Δχ 2t ] (44)

[0182] There is a relationship:

[0183]

[0184] A t Substituting into equation (21), the Runge-Kutta 4th-order integrator is used to obtain the next moment χ 1t+Δt ,x 2t+Δt , integrated position state value X t+Δt Parameter H has been obtained in t+Δt ,γ t+Δt Given t+Δt ; Set the action value a t With the state value s t The value function is passed into the network Critic, and the network Actor is updated to complete the training process at time t. At t = 0, the following formula is obtained:

[0185]

[0186] After the initial value is passed in to start training, the network parameter training steps are repeated until the training end condition is reached, that is:

[0187]

[0188] After the training conditions are met, this training process is recorded as one training, and the number of training times i = i + 1; set the training iteration upper limit i max =5000, when i=i max The training is ended at this time, and a reinforcement learning driven hypersonic variant aircraft re-entry trajectory optimizer is output, thereby forming a reinforcement learning driven hypersonic variant aircraft re-entry trajectory optimization method.

[0189] Step D: Consider the reentry flight of hypersonic variant aircraft under different terminal conditions, determine the specific characteristic parameters of the aircraft, optimize the trajectory with the help of the reentry trajectory optimization model driven by the DDPG algorithm trained in step C, and give variant decisions in real time. Afterwards, compare the simulation analysis of other methods under the same conditions to highlight the flight benefits of the established optimization model.

[0190] Step D is implemented as follows:

[0191] Step D-1: Optimize the trajectory of three different simulation conditions. The hypersonic variant aircraft has a small trajectory difference in the initial descent phase due to the slight difference in aerodynamic force, and the initial point is the separation point of the rocket, which is transported by the booster rocket to the near space at an altitude of about 60km. The simulation termination condition is that the hypersonic variant aircraft reaches the dive pressure point determined by the mission characteristics, and the aircraft is ready to dive down to enter the terminal guidance phase and reach the target mission point. The initial conditions of the design simulation are the same as the initial conditions of the training, and the different trajectories of the terminal termination conditions are optimized to give the variant configuration to the hypersonic aircraft during the reentry phase. The termination conditions are set to the termination speed v1 = 2500, 3000, 3500m / s, corresponding to the termination altitude H1 = 25, 30, 35km.

[0192] Step D-2: In order to further explore the benefits of the established DDPG algorithm-driven reentry trajectory optimization model on the hypersonic variant aircraft during the reentry flight, flight simulations based on various optimization methods are compared. For the optimization method, SLSQP, COBYLA and trust-constr are used to calculate the angle corresponding to the maximum lift-to-drag ratio under the established lift coefficient proxy model and drag coefficient proxy model, and the angle is substituted into the motion equation to calculate the state quantity at the next moment, and the optimization simulation results are compared. For the direct control method, the pseudo-spectral method is used, which performs well in solving trajectory optimization problems. The LG points of the roots of the N-order Legendre polynomial are used for the matching points. When the starting and ending conditions and flight state parameters are the same as those of the previous simulation, the flight results of the entire simulation are compared.

[0193] In order to better reflect the effectiveness and engineering practicality of the present invention, the present invention is further explained below by taking the reentry trajectory optimization problem of a hypersonic variant aircraft as an example with reference to the accompanying drawings and tables.

[0194] In this case, the overall layout of the hypersonic variant aircraft is shown in Figure 1 ,in, Figure 1 (a) is the side view of the initial shape of the fuselage. Figure 1 (b) is the rear view of the initial shape of the fuselage. Figure 1 (c) is a top view of the initial shape of the fuselage. Figure 1 (d) is a top view of the half mold of the whole machine. The comparison of parameters before and after trajectory optimization under different termination conditions is shown in Figure 5 to Figure 7 ,in, Figures 5 to 7 (a) is the altitude-range diagram, Figures 5 to 7 (b) is the altitude-speed diagram, Figures 5 to 7 (c) is the speed-time diagram, Figure 5 (d) is the projectile inclination angle-time diagram. The comparison of parameters before and after trajectory optimization under different optimization algorithms is shown in Figures 9 to 11 in, Figures 9-11(a) is the altitude-range diagram, Figures 9-11 (b) is the altitude-speed diagram, Figures 9-11 (c) is the speed-time diagram, Figures 9-11 (d) is the missile body inclination angle-time diagram. The range and lift-to-drag ratio improvements before and after optimization under different termination conditions are shown in Table 1. The range and lift-to-drag ratio comparison of different algorithms under the same termination conditions is shown in Table 2. The range and lift-to-drag ratio of the Gaussian pseudo-spectral method configuration and other configurations are shown in Table 3.

[0195] Table 1 Comparison of range and lift-to-drag ratio of hypersonic variant aircraft before and after optimization

[0196]

[0197] Table 2 Comparison of range and lift-to-drag ratio of hypersonic variant aircraft before and after optimization of different algorithms

[0198]

[0199] Table 3 Comparison of range and lift-to-drag ratio between Gaussian pseudospectral configuration and other configurations

[0200]

[0201] The above optimization results show that the present invention can obtain a trajectory of a hypersonic variant aircraft with a range as extended as possible during the reentry phase with a relatively low computing cost, and can achieve millisecond-level variant decision control.

[0202] The specific description above further illustrates in detail the purpose, technical solutions and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention, which is used to explain the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A reinforcement learning driven hypersonic variant aircraft reentry trajectory optimization method, characterized by: The steps include: Step A: Establish a parametric model of a hypersonic variant aircraft; Step A is implemented as follows: Step A-1: ​​Construct the baseline shape and variant form; according to the characteristics of the re-entry flight environment of the hypersonic variant aircraft, adopt a waverider-like configuration, that is, a waverider fuselage combined with a hypersonic wing shape; due to the thin atmosphere during flight, use a variable sweep angle in large-scale deformation to improve the performance of the variant; Step A-2: Parameterize the geometric model; based on the standard three-dimensional CST description equation combined with the waverider-like fuselage shape characteristics, combined with the symmetry relationship between the leading edge line in the horizontal plane and the longitudinal plane, the Z coordinate expression of the upper and lower surfaces of the initial fuselage shape is obtained, that is, the parametric model of the hypersonic variant aircraft; In the formula, H′ u is the surface control parameter on the spanwise section, c(·) is the category function, M′ u is the control parameter of the upper surface of the body axis section, N u ′ is the upper surface thickness control parameter, η is the normalized X and Y coordinates, and similarly, the subscript l is the lower surface related control parameter; Step B: Establish a six-degree-of-freedom trajectory model of a hypersonic variant aircraft; Step B is implemented as follows: Step B-1: In the reentry phase flight environment of the hypersonic variant aircraft, the changes in the speed of sound and atmospheric density need to be considered. Therefore, the aerodynamic model of the hypersonic variant aircraft is jointly affected by the flight altitude H, the flight Mach number Ma, the angle of attack α, the inner wing section sweep angle χ1, and the outer wing section sweep angle χ2. According to the flight environment and the characteristics of the aircraft structure, the Latin hypercube test design method is used to generate a predetermined number of sampling points within the variable range: [H,Ma,α,χ1,χ2]=lhsdesign(N) (2)where N is the number of sampling points; Through a series of automated processes such as drawing grids, importing aerodynamic simulation software to solve aerodynamic forces, and collecting aerodynamic data, multiple sets of discrete lift and drag coefficient aerodynamic coefficient data are obtained; the lift and drag coefficient aerodynamic coefficient data is the lift coefficient C L and the drag coefficient C D ; Step B-2: Calculate the lift coefficient C L , drag coefficient C D Construct Kriging surrogate models respectively; In the formula, are the Kriging proxy model fitting functions of lift coefficient and drag coefficient respectively; The established Kringing surrogate model is subjected to leave one out cross validation LOOCV (Leave one out cross validation) and the multiple correlation coefficient is given; Among them, C Li , C Di is the lift coefficient and drag coefficient obtained by the proxy model at each sampling point; if any complex correlation coefficient is less than 0.8, return to step B-1 to increase the number of sampling points; otherwise, proceed to step B-3; Step B-3: According to the flight environment characteristics experienced by the hypersonic variant aircraft during the reentry phase, through the description of the coordinate system and the description of the atmospheric model, the state quantities are selected as the rotating earth relative speed v, the track inclination angle γ, the heading angle ψ, the longitude λ, the latitude φ, and the altitude H, and a six-degree-of-freedom trajectory model of the hypersonic variant aircraft is constructed; In the formula, R e is the radius of the earth, g is the gravitational acceleration, which adopts the nominal value; σ is the roll angle, which is taken as 0; L is the lift acceleration, and D is the drag acceleration, which needs to be calculated; C v2 is the relative velocity v direction component of the involved acceleration term, C γ1 and C γ2 are the γ-direction components of the track inclination angle of the Coriolis acceleration term and the entrained acceleration term, respectively, C ψ1 and C ψ2 They are the heading angle ψ direction components of the Coriolis acceleration term and the entrained acceleration term, respectively, and the expressions are: In the formula, ω e is the angular velocity of the Earth's rotation, using the nominal value; The aerodynamic data obtained by formula (3) gives the lift acceleration L and the drag acceleration D; In the formula, dynamic pressure q = 0.5ρv 2 , ρ is the atmospheric density given by the standard atmospheric model, S is the reference area of ​​the wing; Step C: Determine the objects pointed to by the state quantity, action quantity, constraints on the body and initial and final conditions in the reinforcement learning algorithm; determine the form of the reward function in the reinforcement learning algorithm and the experience pool composed of expert experience and exploration experience; determine the network architecture setting and initial quantity assignment in the reinforcement learning algorithm; train the reentry trajectory optimization model driven by the reinforcement learning algorithm to obtain the trained reentry trajectory optimization model driven by the reinforcement learning algorithm, and perform trajectory optimization according to the trained reentry trajectory optimization model driven by the reinforcement learning algorithm; Step C is implemented as follows: Step C-1: Set the inner wing section sweep angle χ1, the outer wing section sweep angle, the height H, the track inclination angle γ, the Mach number Ma and the angle of attack α as the basic element state quantity parameters s required in the DDPG reinforcement learning algorithm; s=[χ1,χ2,H,γ,Ma,α] (8) The change of the inner wing section sweep angle Δχ1 and the change of the outer wing section sweep angle Δχ2 are set as the basic element action amount a required in the DDPG reinforcement learning algorithm; a=[Δχ1,Δχ2] (9) The state parameter attack angle α changes with the speed in stages as shown in formula (10): Among them, α max is the maximum angle of attack, α max ( K ) is the angle of attack at the maximum lift-to-drag ratio, V1 is the initial speed at the maximum angle of attack, V2 is the speed corresponding to the maximum lift-to-drag ratio, and the other parameters are as follows: According to the maximum heat flow constraint, overload constraint and dynamic pressure constraint, and based on the standard drag acceleration profile, a reentry boundary constraint model is given. In the formula, q max is the maximum heat flow constraint, n′ max is the overload constraint, q′ max is the dynamic pressure constraint, D q,max is the drag acceleration under the maximum heat flux constraint, D n′,max is the resistance acceleration under overload constraint, D q′,max is the drag acceleration under dynamic pressure constraint; According to the flight characteristics, flight environment characteristics and the limitations of the hypersonic variant aircraft during the reentry phase, the take-off parameters of the aircraft are given, including the take-off speed v0 and the take-off height H0; the initial configuration parameters include the inner wing section sweep angle χ 10 , outer wing section sweep angle χ 20 ; The initial posture parameters of the aircraft are the track inclination angle γ0, the heading angle ψ0; the initial position parameters of the aircraft are the longitude λ0, the latitude φ0; the other parameters are the aircraft mass m, the wing reference area S, the earth radius R e , rotation speed ω e and the gravitational acceleration g on the Earth’s surface; Take the reentry end condition to terminate the training and set the termination speed v s , end height H s ; Step C-2: Establish the reward function R required for reinforcement learning training, and build an experience pool combined with expert experience based on the reward function R; Considering the objective function of maximum lift-to-drag ratio, the smoothness requirement of deformation and related constraint requirements, the reward function R is established; R=λ1C L / D -λ2(Δχ1+Δχ2)-λ3ΔD q,n′,q′ (13) Among them, λ1, λ2, λ3 are preset weight coefficients, C L / D is the lift-to-drag ratio, ΔD q,n′,q′ It is the excess of heat flow, overload and dynamic pressure constraints; During the training process, the experience pool sampled for training is divided into exploration experience E explore and expert experience expert , introduce parameter α e As the ratio of expert experience sampling to total sampling Where M is the number of samples taken in the corresponding experience pool; Exploration experience E in the experience pool explore , is the experience formed by the interaction between the action output by the neural network and the environment of the agent in the training round. It is stored in each round of training until the upper limit of the experience pool is reached and then updated. The expert experience E in the experience pool expert Since the lift-to-drag ratio is the main reward, the experience of the two control variables of the inner and outer wing section sweep angles that achieve the maximum lift-to-drag ratio under the current state is set; Step C-3: Give the reinforcement learning network architecture settings; Establish an action network Actor and an evaluation network Critic. The network Critic outputs a value function when inputting a state quantity s and an action quantity a, which is used to update the network Actor. The network Actor consists of two networks Actor_e and Actor_t with the same initial hyperparameters and structures. Actor_e outputs an action quantity a when inputting a state quantity s; Actor_t outputs the next action quantity a′ when inputting the next state quantity s′. After the network architecture is set up, the initial state value s0 and the initial position state value X0 are determined according to the initial data given in step C-1: Among them, Ma0 and α0 are derived and calculated from v0; Step C-4: Perform reentry trajectory optimizer training under the complete DDPG algorithm architecture; The steps for network parameter training are as follows: At time t, there exists an action value a t-Δt , state value s t With the position state value X t : The subscript t represents the value of the variable at time t. The variable H t ,α t ,Ma t ,x 1t ,x 2t Substitute into formula (3) to obtain the aerodynamic coefficient C at the current moment Lt ,C Dt ; The current aerodynamic coefficient C Lt ,C Dt Substitute into formula (7) to obtain the lift acceleration L at the current moment: t , drag acceleration D t ; L t , D t With X t Substituting into equation (5), the Runge-Kutta 4th-order integrator is used to obtain the position state value X at the next moment: t+Δt , Δt is the specific time step set; the position state value X t+Δt Medium t+Δt Substitute into formula (10) to obtain α t+Δt , substitute into the standard model to obtain Ma t+Δt ; will s t Get a from the incoming network Actor t , a t As follows: a t =[Dx 1t ,Dx 2t ] (17) Get the relationship: A t Substituting into equation (18), the Runge-Kutta 4th-order integrator is used to obtain the next moment χ 1t+Δt ,x 2t+Δt , integrated position state value X t+Δt Parameter H has been obtained in t+Δt ,γ t+Δt Given t+Δt ; Set the action value a t With the state value s t The value function is passed into the network Critic, and the network Actor is updated to complete the training process at time t. At t = 0, the following formula is obtained: After the initial value is passed in to start training, the network parameter training steps are repeated until the training end condition is reached, that is: After the training conditions are met, this training process is recorded as one training, and the number of training times i = i + 1; set the training iteration limit i max , when i=i max The training is ended when , and the re-entry trajectory optimization model driven by the trained reinforcement learning algorithm is output, and the trajectory optimization is performed according to the re-entry trajectory optimization model driven by the trained reinforcement learning algorithm.

2. A method for optimizing the reentry trajectory of a hypersonic morphing vehicle driven by reinforcement learning as claimed in claim 1, characterized in that: The method also includes step D: considering the reentry flight of a hypersonic variant aircraft under different terminal conditions, determining specific characteristic parameters of the aircraft, and optimizing the trajectory according to the reentry trajectory optimization model driven by the reinforcement learning algorithm trained in step C, so as to improve the accuracy and efficiency of the reentry trajectory optimization and realize high-precision variant decision control of the hypersonic variant aircraft during the reentry flight.

3. A method for optimizing the reentry trajectory of a hypersonic morphing vehicle driven by reinforcement learning as claimed in claim 1, characterized in that: When solving the aerodynamic data in step B, the aerodynamic data is solved for the aircraft in similar working conditions through the grid independence test, and the correctness of the collected aerodynamic data is verified through the experimental data comparison method to obtain correct aerodynamic data.

4. A method for optimizing the reentry trajectory of a hypersonic variant vehicle driven by reinforcement learning as claimed in claim 1, characterized in that: The network architecture of the Actor in C-4 includes five layers, namely the input layer, three hidden layers and the output layer. Except for the last two layers, which use sigmoid as the activation function, the remaining layers use relu as the activation function, and the connection method is full connection between neurons; the number of input neurons is set to the state dimension, and the number of output neurons is set to the action dimension.

5. A method for optimizing the reentry trajectory of a hypersonic morphing vehicle driven by reinforcement learning as claimed in claim 1, characterized in that: The evaluation network Critic in C-4 is composed of two networks with the same initial hyperparameters and structure, namely Critic_e and Critic_t. Both networks give output value functions when the input state s and action a are input; Critic_e is updated in the form of normal update, while Critic_t is updated in the form of soft update; the network architecture includes five layers, including input layer, three hidden layers and output layer. Except for the last two layers, sigmoid is used as the activation function, and relu is used as the activation function between the remaining layers. The connection method is full connection between neurons; the number of input neurons is set to the total dimension of action and state, and the number of output neurons is the dimension of value function.

6. A method for optimizing the reentry trajectory of a hypersonic variant vehicle driven by reinforcement learning as claimed in claim 1, 4 or 5, characterized in that: Actor_e is updated in the normal way, and Actor_t is updated in the soft way. Normal update uses the weight training form of gradient descent method to update the weight training as follows: Among them, L is the damage function, θ is the network hyperparameter, E is the expectation function, R is the reward function, γ e is the discount factor, Q is the value function; The normal update format is as follows: Set the soft update coefficient τ, the original network hyperparameter θ and the updated hyperparameter θ′, and the soft update form is as follows: θ'←τθ+(1-τ)θ',τ≤1 (23) .

Images