A hyper-sonic vehicle control parameter optimization method based on deep Q learning

Abstract

The application provides a control parameter optimization method for a hypersonic aircraft based on a deep Q network, and the method comprises the following steps: establishing a hypersonic aircraft dynamics model; decomposing the hypersonic aircraft dynamics model into a speed subsystem and an altitude subsystem; designing a speed subsystem adaptive controller; designing an altitude subsystem adaptive controller; and optimizing the design parameters a in steps 3 and 4 by using a deep Q learning theory h 、a γ 、a α 、a Q , and designing a reward function. According to the needs of a user, the application can dynamically optimize the parameters of the controller of the hypersonic aircraft by designing the reward function, so that the control system can exert maximum efficiency.

Description

Technical Field

[0001] This invention relates to a method for optimizing flight control parameters, specifically a method for optimizing control parameters of a hypersonic vehicle based on a deep Q-network. Background Technology

[0002] Deep Q-Network (DQN) is an artificial intelligence algorithm that has gained widespread popularity in recent years due to its ease of use. The parameter settings of the controller for hypersonic vehicles have a significant impact on control performance and even system stability; therefore, using intelligent algorithms to optimize the parameters of hypersonic vehicle controllers shows promising application prospects.

[0003] Traditional hypersonic vehicle optimization methods (K. Gao, J. Song, X. Wang, and H. Li, “Fractional-order proportional-integral-derivative linear active disturbance rejection control design and parameter optimization for hypersonic vehicles with actuator faults,” Tsinghua Science and Technology, vol. 26, no. 1, pp. 9-23, Feb. 2021.) can achieve parameter optimization using frequency domain analysis, but they can only optimize parameters at one operating point at a time. Hypersonic vehicles have a large flight envelope spanning multiple domains, thus these methods lack intelligence. Existing hypersonic vehicle parameter adjustment methods (a hypersonic vehicle expert intelligent control method and vehicle, CN201910547381.8) first construct an expert system, pre-establishing the correspondence between the real-time deformation state of the hypersonic vehicle and the control parameters of the PID controller within the expert system. During flight, the control parameters are adjusted according to the state of the hypersonic vehicle. This is a rule-based parameter tuning method, which, while possessing some intelligence, lacks autonomy during the parameter tuning process.

[0004] Hypersonic vehicles have a large flight envelope, with a wide range of speeds and altitudes, and face more complex and variable flight environments compared to conventional aircraft. Therefore, designing an effective control parameter optimization method is crucial for improving controller performance. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method for optimizing control parameters of hypersonic vehicles based on deep Q-learning, the specific steps of which are as follows:

[0006] Step 1: Establish the following hypersonic vehicle dynamics model:

[0007]

[0008] Where: m, I yy g represents the mass, moment of inertia, and gravitational acceleration of the aircraft, respectively; V, h, γ, α, θ, and Q represent the velocity, altitude, track angle, angle of attack, pitch angle, and pitch velocity, respectively, and α = θ - γ. These are the derivatives of velocity, altitude, track angle, pitch angle, pitch angular velocity, and elastic state, respectively. Let η1 and η2 represent the first and second order elastic states, respectively. These are the derivatives of the first-order elastic state and the second-order elastic state, respectively. It is the second derivative of the i-th elastic state; ζ i ω i , respectively, represent the damping and natural vibration frequencies of the elastic modal coordinates; T, D, L, M, I yy N i These are thrust, drag, lift, pitching moment, moment of inertia, and the i-th generalized force;

[0009] Set the initial state and reference signal for the hypersonic vehicle;

[0010] Step 2: Decompose the hypersonic vehicle dynamics model (1) into a velocity subsystem (2) and an altitude subsystem (3):

[0011]

[0012]

[0013] In the formula: F V , Φ, d V , d h F γ d γ F α d α F Q δ e d Q These are the subsystem functions, fuel equivalence ratio, velocity disturbance, derivative of angle of attack, altitude disturbance, track angle subsystem function, track angle disturbance, angle of attack subsystem function, angle of attack disturbance, pitch rate subsystem function, elevator deflection, and pitch rate disturbance, respectively. γ This indicates the relationship between the product of two parameters;

[0014] Step 3: Design the adaptive controller for the velocity subsystem:

[0015]

[0016]

[0017] Where: Φ, k V , These are the derivatives of the fuel equivalence ratio, speed controller parameters, speed adaptive parameters, and speed adaptive parameters, respectively. V =VV ref It is the speed tracking error, e V It is a function of time t, where T1 is the flight time, and V ref It is a speed reference signal; t V >o is the design parameter, k V >0 will be obtained through training a deep Q-learning network; the uniformly continuous bounded function τ(t) satisfies:

[0018]

[0019] In the formula: τ1>0, τ2>o are design parameters. It is the derivative of the function τ(t);

[0020] Step 4: Design the adaptive controller for the altitude subsystem:

[0021]

[0022]

[0023] Where: χ γ , χ α , χ Q δ e These are the virtual control laws for track angle, angle of attack, pitch rate, and elevator deflection, a h a γ b γ , a α b α , a Q b Q , These are, respectively, the altitude controller parameter, the first parameter of the track angle controller, the second parameter of the track angle controller, the adaptive parameter of the track angle, the derivative of the adaptive parameter of the track angle, the first parameter of the angle of attack controller, the second parameter of the angle of attack controller, the adaptive parameter of the angle of attack, the derivative of the adaptive parameter of the angle of attack, the first parameter of the pitch rate controller, the second parameter of the pitch rate controller, the adaptive parameter of the pitch rate, and the derivative of the adaptive parameter of the pitch rate; a h aγ a α a Q This will be determined through deep Q-learning; e h =hh ref It is the height tracking error, h ref For high reference signal, It is the derivative of the high-reference signal, e γ =γ-χ γ It is the tracking error of the track angle, e α =α-χ α It is the angle-of-attack tracking error, e Q =Q-χ Q It is the pitch angular velocity tracking error;

[0024] Step 5: Optimize the design parameter 'a' in Steps 3 and 4 using deep Q-learning theory. h a γ a α a Q Design the reward function:

[0025] r = -w V |e V |-w h |e h |-w Φ Φ-w e1 |Δδ e |-w e2 |δ e | (9)

[0026] In the formula: w V w h w Φ w e1 w e2 All are weighting parameters, Δδ e δ e These represent the change in elevator deflection angle and the magnitude of the elevator deflection angle, respectively.

[0027] The deep reinforcement learning proposed in this invention can continuously explore control performance under different control parameters based on sample data obtained from simulation and through continuous interaction with the environment. After training, a mapping from the flight environment and flight state to control parameters can be obtained, and the controller parameters can be adaptively adjusted according to the external environment and its own state, thereby maximizing the controller performance.

[0028] The advantages of this invention are:

[0029] In traditional hypersonic vehicle adaptive controller design methods, the gain coefficient is fixed after the controller design is completed, which limits the controller's performance to some extent. To maximize controller performance, this invention proposes a dynamic parameter optimization / adaptive parameter method that adapts to changes in the flight environment and flight state, thereby improving controller efficiency. This method allows for the design of a reward function based on user needs, dynamically optimizing the controller parameters of the hypersonic vehicle and ensuring the control system achieves maximum efficiency. Attached Figure Description

[0030] Figure 1 A schematic diagram of control parameter optimization is shown;

[0031] Figure 2 The curve shows the change in the benefit of deep Q-learning. As the number of training iterations increases, the benefit of the Q-network gradually increases and tends to converge, indicating that the application of deep Q-learning is effective.

[0032] Figure 3 The speed tracking performance curve is shown below. Figure 3 (a) shows the speed reference signal and the speed change curves under three conditions. Figure 3 (b) shows the speed tracking error in three scenarios;

[0033] Figure 4 To closely track the performance curve, where Figure 4 (a) shows the altitude reference signal and altitude change curves for three scenarios. Figure 4 (b) shows the height tracking error in three scenarios;

[0034] Figure 5 For the track angle, angle of attack, and pitch angular velocity, where Figure 5 (a) shows the track angle. Figure 5 (b) shows the angle of attack. Figure 5 (c) shows the pitch angular velocity;

[0035] Figure 6 For elastic state and control input, where Figure 6 (a) shows the first-order elastic state. Figure 6 (b) shows the second-order elastic state. Figure 6 (c) shows the fuel equivalence ratio control input. Figure 6 (d) shows the elevator deflection angle control input;

[0036] Figure 7 The adaptive parameter variation curve is shown. Detailed Implementation

[0037] This invention provides a method for optimizing control parameters of a hypersonic vehicle based on a deep Q-network. The method's technical approach is as follows: Deep reinforcement learning is used to optimize an adaptive flight controller, enabling it to dynamically change its gain coefficient based on the external flight environment and its own flight state, thereby maximizing the controller's performance. The external flight environment and the vehicle's flight state are used as inputs, and the gain coefficient of the adaptive flight controller is used as the output. A reward function (e.g., the integral of the absolute value of the tracking error over a certain simulation duration) is set according to mission requirements. The policy function network is continuously trained, and the network parameters are continuously updated using gradient descent. Figure 1 As shown, after multiple iterations of training, a policy network is obtained that integrates the external flight environment and the flight state with the gain coefficients of the adaptive flight controller, enabling dynamic adjustment of controller parameters and maximizing controller performance.

[0038] The specific steps of the method of the present invention are as follows:

[0039] Step 1: Establish the following hypersonic vehicle dynamics model:

[0040]

[0041] Where: m, I yy g represents the mass, moment of inertia, and gravitational acceleration of the aircraft, respectively; V, h, γ, α, θ, and Q represent the velocity, altitude, track angle, angle of attack, pitch angle, and pitch velocity, respectively, and α = θ - γ. These are the derivatives of velocity, altitude, track angle, pitch angle, pitch angular velocity, and elastic state, respectively. Let η1 and η2 represent the first and second order elastic states, respectively. ζ represents the derivatives of the first-order elastic state and the second-order elastic state, respectively; i ω i , respectively, represent the damping and natural vibration frequencies of the elastic modal coordinates; T, D, L, M, I yy N i These are thrust, drag, lift, pitching moment, moment of inertia, and the i-th generalized force.

[0042] Set the initial state and reference signal for the hypersonic vehicle.

[0043] Step 2: Decompose the hypersonic vehicle dynamics model (1) into a velocity subsystem (2) and an altitude subsystem (3):

[0044]

[0045]

[0046] In the formula: F V, Φ, d V , d h F γ d γ F α d α F Q δ e d Q These are the subsystem functions, fuel equivalence ratio, velocity disturbance, derivative of angle of attack, altitude disturbance, track angle subsystem function, track angle disturbance, angle of attack subsystem function, angle of attack disturbance, pitch rate subsystem function, elevator deflection, and pitch rate disturbance, respectively. γ It's a multiplicative relationship.

[0047] Step 3: Design the adaptive controller for the velocity subsystem:

[0048]

[0049]

[0050] Where: Φ, k V , These are the derivatives of the fuel equivalence ratio, speed controller parameters, speed adaptive parameters, and speed adaptive parameters, respectively. V =VV ref It is the speed tracking error, e V It is a function of time t, where T1 is the flight time, and V ref It is a speed reference signal; t v >0 is a design parameter, k V >0 will be obtained through training a deep Q-learning network. A uniformly continuous bounded function τ(t) satisfies:

[0051]

[0052] In the formula: τ1>0, τ2>0 are design parameters. It is the derivative of the function τ(t).

[0053] Step 4: Design the adaptive controller for the altitude subsystem:

[0054]

[0055]

[0056] Where: χ γ , χ α , χ Q δ e a h , aγ b γ , a α b α , a Q b Q , These are: virtual control law for track angle, virtual control law for angle of attack, virtual control law for pitch rate, elevator deflection, altitude controller parameters, derivative of altitude reference signal, first parameter of track angle controller, second parameter of track angle controller, adaptive parameter of track angle, derivative of adaptive parameter of track angle, first parameter of angle of attack controller, second parameter of angle of attack controller, adaptive parameter of angle of attack, derivative of adaptive parameter of angle of attack, first parameter of pitch rate controller, second parameter of pitch rate controller, adaptive parameter of pitch rate, derivative of adaptive parameter of pitch rate; a h a γ a α a Q This will be determined through deep Q-learning; e h =hh ref It is the height tracking error, h ref For high reference signal, e γ =γ-χ γ It is the tracking error of the track angle, e α =α-χ α It is the angle-of-attack tracking error, e Q =Q-χ Q It is the pitch angular velocity tracking error.

[0057] Step 5: Optimize the design parameter 'a' in Steps 3 and 4 using deep Q-learning theory. h a γ a α a Q Design the reward function:

[0058] r = -w V |e V |-w h |e h |-w Φ Φ-w e1 |Δδ e |-w e2 |δ e | (9)

[0059] In the formula: w V w h w Φ w e1 w e2 All are weighting parameters, Δδ e δe These represent the change in elevator deflection angle and the magnitude of the elevator deflection angle, respectively.

[0060] According to the specific implementation steps of the technical solution of the present invention, the following embodiment is given.

[0061] Step 1: Establish the hypersonic vehicle dynamic model as shown in equation (1). The initial state of the hypersonic vehicle is set as V0 = 7699.5ff / s, h0 = 84999.5ff, γ0 = 0°, α0 = 1.6325°, Q0 = 0° / s, η1 = 0.97, η2 = 0.7967. V0, h0, γ0, α0, and q0 are the initial values ​​of V, h, γ, α, and Q, respectively.

[0062] Let V increase from 7700 ft / s to 7800 ft / s, and h increase from 85000 ft / s to 85300 ft / s, V ref and h ref The following is given by a filter with a damping of 0.95 and a natural frequency of 0.03 rad / s:

[0063]

[0064]

[0065] In the formula: V ref (s) and h ref Vc(s) is the input of the filter, Vc(s) and hc(s) are the outputs of the filter, and s is the Laplace transform factor.

[0066] Step 2: Decompose the hypersonic vehicle dynamics model (1) into a velocity subsystem (2) and an altitude subsystem (3).

[0067] Step 3: Design an adaptive controller for the velocity subsystem as shown in equations (4) and (5). Some parameters of the controller are selected as follows:

[0068] Step 4: Design an adaptive controller for the height subsystem as shown in equations (7) and (8). Some parameters of the controller are selected as follows: [b γ b α b q ] = [1, 1, 1], [c h c γ c α c q = [0.0005, 0.05, 0.05, 0.05].

[0069] Step 5: Design the reward function as shown in equation (9), with the weight parameters selected as: [w V wh w Φ w e1 w e2 ] = [1, 1, 1, 1, 1].

[0070] To demonstrate the advantages of this invention, consider the following three scenarios (Scenario 1 and Scenario 2 represent two sets of parameters that are better obtained using conventional parameter tuning methods):

[0071] Case 1: Control parameters not using deep Q-learning training, the control parameters are selected as [a] h a γ a α a Q ] = [8, 24, 24, 24];

[0072] Case 2: Control parameters not using deep Q-learning training, the control parameters are selected as [a] h a γ a α a Q ] = [7, 20, 26, 28];

[0073] Case 3: Control parameters obtained by deep Q-learning training.

[0074] Figure 2 The curve shows the return as a function of the number of training iterations. As the number of training iterations increases, the return gradually increases and then tends to converge. From... Figure 3 and Figure 4 It is evident that the speed tracking error and altitude tracking error can asymptotically converge to zero in all three cases. Furthermore, the speed tracking error and altitude tracking error in case 3 exhibit smaller fluctuations than in the other two cases, indicating that the proposed control method can achieve better control performance. It is also evident that the tracking error overshoot is smaller when using the present invention, demonstrating superior tracking performance. Figure 5 and Figure 6 The results show that the track angle, angle of attack, pitch rate, elastic state, and control input curves in case 3 are smoother than those in the other two cases. This indicates that compared to traditional parameter tuning methods, the parameter optimization method of this invention enables the controller to perform more effectively, and the proposed control method can achieve better transient control performance. Figure 7 The proposed method demonstrates that the adaptive parameters are bounded. After multiple iterations of training, a policy network is obtained that integrates the external flight environment and the flight state with the gain coefficients of the adaptive flight controller, enabling dynamic adjustment of the controller parameters and maximizing controller performance.

Claims

1. A method for optimizing control parameters of a hypersonic vehicle based on deep Q-learning, characterized in that, The specific steps are as follows: Step 1: Establish the following hypersonic vehicle dynamics model: Where: m, I yy g represents the mass, moment of inertia, and gravitational acceleration of the aircraft, respectively; V, h, γ, α, θ, and Q represent the velocity, altitude, track angle, angle of attack, pitch angle, and pitch velocity, respectively, and α = θ - γ. These are the derivatives of velocity, altitude, track angle, pitch angle, pitch angular velocity, and elastic state, respectively. Let η1 and η2 represent the first and second order elastic states, respectively. These are the derivatives of the first-order elastic state and the second-order elastic state, respectively. It is the second derivative of the i-th elastic state; ζ i ω i , respectively, represent the damping and natural vibration frequencies of the elastic modal coordinates; T, D, L, M, I yy N i These are thrust, drag, lift, pitching moment, moment of inertia, and the i-th generalized force; Set the initial state and reference signal for the hypersonic vehicle; Step 2: Decompose the hypersonic vehicle dynamics model (1) into a velocity subsystem (2) and an altitude subsystem (3): In the formula: F V , Φ, d V , d h F γ d γ F α d α F Q δ e d Q These are the subsystem functions, fuel equivalence ratio, speed disturbance, derivative of angle of attack, altitude disturbance, track angle subsystem function, track angle disturbance, angle of attack subsystem function, angle of attack disturbance, pitch rate subsystem function, elevator deflection, and pitch rate disturbance, respectively. Vγ represents the relationship between the product of the two parameters. Step 3: Design the adaptive controller for the velocity subsystem: Where: Φ, k V , These are the derivatives of the fuel equivalence ratio, speed controller parameters, speed adaptive parameters, and speed adaptive parameters, respectively. V =VV ref It is the speed tracking error, e V It is a function of time t, where T1 is the flight time, and V ref It is a speed reference signal; ι V >0 is a design parameter, k V >0 will be obtained through training a deep Q-learning network; A uniformly continuous and bounded function τ(t) satisfies: In the formula: τ1>0, τ2>0 are design parameters. It is the derivative of the function τ(t); Step 4: Design the adaptive controller for the altitude subsystem: Where: χ γ , χ α , χ Q δ e These are the virtual control laws for track angle, angle of attack, pitch rate, and elevator deflection, a h a γ b γ , a α b α , a Q b Q , These are, respectively, the altitude controller parameter, the first parameter of the track angle controller, the second parameter of the track angle controller, the adaptive parameter of the track angle, the derivative of the adaptive parameter of the track angle, the first parameter of the angle of attack controller, the second parameter of the angle of attack controller, the adaptive parameter of the angle of attack, the derivative of the adaptive parameter of the angle of attack, the first parameter of the pitch rate controller, the second parameter of the pitch rate controller, the adaptive parameter of the pitch rate, and the derivative of the adaptive parameter of the pitch rate; a h a γ a α a Q This will be determined through deep Q-learning; e h =hh ref It is the height tracking error, h ref For high reference signal, It is the derivative of the high-reference signal, e γ =γ-χ γ It is the tracking error of the track angle, e α =α-χ α It is the angle-of-attack tracking error, e Q =Q-χ Q It is the pitch angular velocity tracking error; Step 5: Optimize the design parameter 'a' in Steps 3 and 4 using deep Q-learning theory. h a γ a α a Q Design the reward function: r=-w V |e V |-w h |e h |-w Φ F-w e1 |Dd e |-w e2 |d e | (9) In the formula: w V w h w Φ w e1 w e2 All are weighting parameters, Δδ e δ e These represent the change in elevator deflection angle and the magnitude of the elevator deflection angle, respectively.

Images