A vertical landing phase performance recovery control method for short vertical propulsion systems based on reinforcement learning

Abstract

The patent gives a short vertical propulsion system vertical landing phase performance recovery control method based on reinforcement learning. The optimization method considers the degradation of the short vertical propulsion system in the design of the controller through the reinforcement learning algorithm. Through the pre-training of the intelligent agent by reinforcement learning, the multiple thrust outputs of the short vertical propulsion system can still maintain stability under the disturbance of component degradation, helping the short vertical aircraft to maintain the balance of the pitch attitude in the vertical landing phase under the condition of degradation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of aero-engines, in particular to a short vertical propulsion system vertical landing phase performance recovery control method based on reinforcement learning. BACKGROUND

[0002] For short take-off and vertical landing aircraft and its propulsion system, the aerodynamic rudder cannot play a role in the hovering state, and the height and attitude control can only completely rely on the multi-source thrust generated by the two lift source nozzles of the propulsion system. However, in this process, if the rotating parts of the propulsion system have performance degradation, such as high-temperature over-temperature of the high-pressure turbine, intake distortion of the lift fan, etc., it will cause thrust loss and change of thrust distribution, and the loss of thrust will change the speed of the aircraft in the height channel, and the change of thrust distribution will cause the imbalance of the pitch attitude, making the aircraft deviate from the designed landing point and even cause the aircraft to stall. The current performance recovery control method is to compensate for the loss of thrust to maintain the thrust of the aircraft unchanged to ensure that the height channel is not affected, but there is little attitude recovery control method for short vertical propulsion systems. In addition, the current performance recovery control method includes model-based method and model-free method, the model-based method depends heavily on model confidence, and if model degradation is considered, the controller designed based on the model-based method will have a particularly high order, and the calculation amount is too large to be practically applied. The model-free performance recovery control method such as PID cannot consider model degradation in the control process, and can only control the parameters that need to be compensated through fuel compensation thrust in a one-to-one manner, which is easy to cause thrust coupling, overshoot in short vertical propulsion systems, and simple fuel increase will affect the attitude balance. Therefore, this paper proposes a model-free performance recovery control method for short vertical propulsion systems based on reinforcement learning, which adds the degradation factor as the state to the reinforcement learning agent training, and takes the control amount of the compensation thrust as the action of the agent, and then uses the agent obtained by offline training of reinforcement learning as the controller, thereby realizing the model-free performance recovery control considering degradation.

[0003] The present application provides a short vertical propulsion system vertical landing phase performance recovery control method based on reinforcement learning, which considers the degradation of the short vertical propulsion system in the design of the controller through the reinforcement learning algorithm, and ensures that the multiple thrust outputs of the short vertical propulsion system remain stable under the disturbance of component degradation through the agent trained offline by reinforcement learning, helping the short vertical aircraft to maintain the balance of the pitch attitude online in the vertical landing phase even in the case of degradation. SUMMARY

[0004] In order to solve the problem of high order, strong coupling and serious overshoot of the traditional short vertical propulsion system vertical landing phase performance recovery controller, a short vertical propulsion system performance recovery control method based on reinforcement learning is proposed.

[0005] The object of the present application is achieved by the technical solutions described below.

[0006] The application discloses a short-vertically-thrust propelling system performance recovery control method based on reinforcement learning, and comprises the following steps.

[0007] Step 1: initialize the comment deep neural network parameter Q(s, a|θ Q ) in the agent, wherein s is the state output from the short-vertically-thrust propelling system, the state comprises thrust and, thrust ratio, lift fan efficiency factor, fan efficiency factor, high-pressure compressor efficiency factor, high-pressure turbine efficiency factor and low-pressure turbine efficiency factor, namely s=[TT, TS, η LF , η FAN , η HPC , η HPT , η LPT ]′, a is the action output from the reinforcement learning, the action comprises fuel flow and lift fan inlet guide vane angle, a=[W f , IGV]′, θ Q is the collection of weight and bias parameters in the neural network, the purpose of the comment deep neural network is to learn an optimal θ Q function, so that a Q value can be evaluated according to the state s and the action a to represent the state-action pair value, and step 2 is entered;

[0008] Step 2: initialize the action deep neural network parameter μ(s|θ μ ) in the agent, wherein s is the state output from the short-vertically-thrust propelling system, θ μ is the collection of weight and bias parameters in the neural network, and the purpose of the action deep neural network is to learn an optimal θ μ function so that the output action a can be mapped according to the state s, and step 3 is entered;

[0009] Step 3: set the input of the short-vertically-thrust propelling system as the output a of the action deep neural network, namely the short-vertically-thrust propelling system is taken as the environment explored by the agent, and the reward function is calculated according to the parameter change caused by the output of the action deep neural network, and the formula is as follows:

[0010]

[0011] In the formula, t represents simulation time, r represents reward, represents the error of thrust and, namely the performance recovery error of the height channel, The error representing the thrust ratio is the performance recovery error of the attitude channel. f(T4) is the over-temperature penalty term. That is, if the temperature is not exceeded, f(T4) = 0, and if the temperature is exceeded, f(T4) = -100. After setting the reward, an exploration is conducted to generate initial experience, and then step 4 is entered.

[0012] Step 4: Perform random exploration based on the initialization parameters to obtain the initial state s1 = [TT, TS, η] LF ,η FAN ,η HPC ,η HPT ,η LPT Proceed to step 5;

[0013] Step 5: Select the action at the current time t based on the current strategy and exploration noise, using the following formula:

[0014]

[0015] In the formula, a t It is the action at time t, μ(s) t |θ μ ) is the current state s t In action deep neural network θ μ The output mapped under the parameter settings, It is to explore noise, and to use a t The current reward r is calculated by inputting the data into the short vertical propulsion system based on the reward function set in step 3. t At the same time, a new state s is obtained. t+1 , state-action pair (s t ,a t ,r t ,s t+1 Store it in the experience playback buffer, and then proceed to step 6;

[0016] Step 6: Sample a mini-batch of state-action pairs N×(s) from the experience replay area. t ,a t ,r t ,s t+1 ), where N represents the batch size. The input of this sampling batch is fed into the deep neural network for commenting to calculate the loss function. The calculation formula is as follows:

[0017]

[0018] In the formula, i is the number of iterations in the inner batch, L is the loss function, and θ represents the objective function to be minimized. Q That is, the parameters of the deep neural network, Q(s) i ,a i |θ Q) is the current state-action pair (s i i ) is the value under the current neural network parameters, y i is the learning objective of the neural network, whose formula is as follows:

[0019]

[0020] In the formula, i is the cycle number of the internal batch, r(s i ,a i ) is the reward under the current sampling batch, γ is the decay coefficient, and θ Q′ represents the parameters of the critic target neural network, that is, the deep neural network θ Q is expected to approximate the target neural network θ Q′ as much as possible by reducing the loss function. Q′ Compared with the frequently updated main network, the target network θ Q′ has a lower update frequency and updates the parameters from the main network in a weighted manner, as follows:

[0021] θ Q′ = ρθ Q

[0022] In the formula, ρ is a hyperparameter between 0 and 1 that determines the degree of soft update, and then the parameter update of the action deep neural network is performed, and step 7 is entered.

[0023] Step 7: Perform policy update through the data obtained by the current batch sampling, and the formula is as follows:

[0024]

[0025] In the formula, i is the cycle number of the internal batch, L is the loss function, θ Q is the parameter of the critic deep neural network, θ μ is the parameter of the action deep neural network, s i represents the current state, represents the action taken in the current state s i , and α represents the learning rate. represents the loss gradient of Q with respect to action a, represents the loss gradient of μ with respect to θ μ , and the Q(s, μ(s t |θ μ ​As the estimation value of the Q value function, it can be seen that the policy gradient on which the policy improvement depends is derived by multiple times of derivation of the Q value function, the method of updating the policy target network parameter is to change the action policy mu in the direction in which the Q value function increases along the mapping gradient of the value function Q, and then the action deep neural network parameter is updated:

[0026] θ μ′ = ρθ μ′ + (1-ρ)θ μ

[0027] In the formula, p is a hyperparameter between 0 and 1, which determines the degree of soft update, then the batch iteration is performed, if i

[0028] Step 8: judge the relationship between the current time t and the total number of rounds M, if t LF FAN HPC HPT LPT , the output of the intelligent agent is the control input a = [W f , IGV] of the short vertical propulsion system, so as to realize the model-free performance recovery control design considering performance degradation. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The embodiment of the application is a specific implementation step of the content "a short vertical propulsion system performance recovery control method in a vertical landing phase based on reinforcement learning". DETAILED DESCRIPTION

[0030] Technical scheme:

[0031] The embodiment provides a short vertical propulsion system performance recovery control method in a vertical landing phase based on reinforcement learning, which comprises the following steps:

[0032] Step 1: initialize the comment deep neural network parameter Q (s, a | θ Q ) in the intelligent agent, wherein s is the state output from the short vertical propulsion system, and the state includes thrust, thrust ratio, lift fan efficiency factor, fan efficiency factor, high-pressure compressor efficiency factor, high-pressure turbine efficiency factor and low-pressure turbine efficiency factor, that is, s = [TT, TS, η LF , η FAN , η HPC , η HPT , η LPT ​​​​], a is the action output from reinforcement learning, the action includes fuel flow and lift fan inlet guide vane angle, a = [W f , IGV], θ Q is the collection of weight and bias parameters in neural network, the purpose of critic deep neural network is to learn an optimal θ Q function, so that a Q value can be evaluated according to state s and action a to represent the value of state-action pair, go to step 2;

[0033] Step 2: initialize action deep neural network parameters μ(s|θ μ ) in agent, where s is the state output from short vertical propulsion system, θ μ is the collection of weight and bias parameters in neural network, the purpose of action deep neural network is to learn an optimal θ μ function, so that the output action a can be mapped according to the state s, go to step 3;

[0034] Step 3: set the input of short vertical propulsion system as the output a of action deep neural network, that is, short vertical propulsion system as the environment explored by agent, calculate the reward function according to the parameter change caused by the output of action deep neural network, the formula is as follows:

[0035]

[0036] In the formula, t represents simulation time, r represents reward, represents the error of thrust and, that is, the performance recovery error of height channel, represents the error of thrust ratio, that is, the performance recovery error of attitude channel, f(T4) is the over-temperature penalty term, that is, if not over-temperature f(T4) = 0, if over-temperature f(T4) = -100, after setting the reward, go to exploration once to generate initial experience, go to step 4;

[0037] Step 4: random exploration according to the initial parameters, obtain the initial state s1 = [TT, TS, η LF , η FaN , η HPC , η HPT , η LPT ], go to step 5;

[0038] Step 5: select the action of current time t according to the current policy and exploration noise, the formula is as follows:

[0039]

[0040] In the formula, a t is the action at time t, μ(s t | θμ ) is the current state s t In action deep neural network θ μ The output mapped under the parameter settings, It is to explore noise, and to use a t The current reward r is calculated by inputting the data into the short vertical propulsion system based on the reward function set in step 3. t At the same time, a new state s is obtained. t+1 , state-action pair (s t ,a t ,r t ,s t+1 Store it in the experience playback buffer, and then proceed to step 6;

[0041] Step 6: Sample a mini-batch of state-action pairs N×(s) from the experience replay area. t ,a t ,r t ,s t+1 ), where N represents the batch size. The input of this sampling batch is fed into the deep neural network for commenting to calculate the loss function. The calculation formula is as follows:

[0042]

[0043] In the formula, i is the number of iterations in the inner batch, L is the loss function, and θ represents the objective function to be minimized. Q That is, the parameters of the deep neural network, Q(s) i ,a i |θ Q ) is the current state-action pair (s i ,a i The value of y under the current neural network parameters. i This is the learning objective of the neural network, and its formula is as follows:

[0044]

[0045] In the formula, i is the number of iterations in the internal batch, and r(s) i ,a i ) is the reward for the current sampling batch, γ is the decay coefficient, and θ is the decay coefficient. Q′ This represents the parameters of the target neural network, specifically the parameters of the deep neural network θ that we want to reduce by lowering the loss function. Q Approximate the target neural network θ as closely as possible Q′ Compared to the frequently updated main network, the target network θ Q′ It has a lower update frequency and uses a weighted approach to update parameters from the main network, as shown below:

[0046] θ Q′= pQ + (1 - p)Q Q′ Q

[0047] where p is a hyper-parameter between 0 and 1, which determines the degree of soft update, and then the parameter update of the action deep neural network is performed to enter step 7;

[0048] Step 7: Perform policy update through the data obtained by the current batch sampling, and the formula is as follows:

[0049]

[0050] where i is the number of internal batch iterations, L is the loss function, Q Q is the parameter of the comment deep neural network, Q μ is the parameter of the action deep neural network, s i represents the current state, represents the action taken in the current state s i , and a represents the learning rate, represents the loss gradient of Q with respect to action a, represents the loss gradient of μ with respect to Q μ , by selecting Q(s, μ(s t | Q μ )) in the current Q value function as the estimated value of the Q value function, it can be seen that the policy gradient relied on by the policy improvement is derived from the Q value function multiple times, and the method of updating the policy target network parameter is to change the action policy μ in the direction of increasing the Q value function along the mapping gradient of the value function Q, and then perform the parameter update of the action deep neural network:

[0051] Q μ′ = pQ + (1 - p)Q μ′ μ

[0052] where p is a hyper-parameter between 0 and 1, which determines the degree of soft update, and then perform internal batch iteration, if i < N, return to step 6 for neural network update, if i = N, enter step 8;

[0053] Step 8: Determine the relationship between the current time t and the total number of rounds M, if t < M, return to step 3, if t = M, the offline training is completed, and the agent is configured to the environment after offline training, the input of the agent is the state s1 = [TT, TS, η LF , η FAN , η HPC , η HPT , η LPT ] of the short vertical propulsion system, and the output is the control input a = [W​​f IGV', thereby achieving model-free performance recovery control design considering performance degradation.

[0054] Compared with the prior art, the short-vertically-thrust propulsion system vertical landing phase performance recovery control method based on reinforcement learning obtained by the embodiment can consider degradation without an accurate model, and realize performance recovery of height and attitude of the short-vertically-thrust propulsion system in the vertical landing phase considering performance degradation.

Claims

1. A method for performance recovery control during the vertical descent phase of a short vertical propulsion system based on reinforcement learning, characterized in that... Includes the following steps: Step 1: Initialize the parameters of the deep neural network for commenting in the agent. ,in This refers to the output state from the short vertical / short vertical propulsion system, which includes the thrust sum, thrust ratio, lift fan efficiency factor, fan efficiency factor, high-pressure compressor efficiency factor, high-pressure turbine efficiency factor, and low-pressure turbine efficiency factor. , These are actions output from reinforcement learning, including fuel flow rate and lift fan inlet guide vane angle. , It is the set of weights and bias parameters in a neural network. The purpose of commenting on deep neural networks is to learn an optimal one. Functions that can be adapted to the state and actions Evaluate one Use the value to represent this state-action pair value, then proceed to step 2; Step 2: Initialize the parameters of the action deep neural network in the agent. ,in This is the state output from the short vertical propulsion system. It is the set of parameters for weights and biases in a neural network. The goal of an action-oriented deep neural network is to learn an optimal... The function can then be based on the state. Mapping the output action Proceed to step 3; Step 3: Set the input of the short vertical propulsion system to the output of the action deep neural network. In other words, the short vertical propulsion system serves as the environment for the agent's exploration. The reward function is calculated based on the parameter changes caused by the output of the deep neural network of the action in the short vertical propulsion system, as shown in the following formula: In the formula, Represents the simulation time. Represents awards, The error representing the thrust sum, i.e., the performance recovery error of the altitude channel, The error representing the thrust ratio, i.e., the performance recovery error of the attitude channel, It's an over-temperature penalty, meaning if the temperature is not exceeded... If the temperature exceeds the limit After setting the rewards, conduct an exploration to generate initial experience, then proceed to step 4; Step 4: Perform random exploration based on the initialization parameters to obtain the initial state. Proceed to step 5; Step 5: Select the current time based on the current strategy and exploration noise. The action is represented by the following formula: In the formula, yes Momentary actions The current state In action deep neural networks The output mapped under the parameter settings, It is to explore noise, The current reward is calculated by inputting the data into the short vertical propulsion system based on the reward function set in step 3. At the same time, a new state is obtained. State-action pairs Store it in the experience replay buffer, and then proceed to step 6; Step 6: Sample a mini-batch of state-action pairs from the experience replay area. ,in The value represents the batch size. This sample batch is fed into the deep neural network for commenting to calculate the loss function, using the following formula: In the formula, It is the number of loops within the internal batch. It is the loss function, representing the objective function to be minimized. That is, commenting on the parameters of deep neural networks. It is a current state-action pair Value under the current neural network parameters This is the learning objective of the neural network, and its formula is as follows: In the formula, It is the number of loops within the internal batch. This is the reward for the current sampling batch. It is the attenuation coefficient. This represents the parameters of the target neural network, i.e., the parameters of the deep neural network that are intended to improve by reducing the loss function. Approximate the target neural network as closely as possible Compared to the frequently updated main network, the target network... It has a lower update frequency and uses a weighted approach to update parameters from the main network, as shown below: In the formula, It is a hyperparameter between 0 and 1 that determines the degree of soft update. Then, the parameters of the action deep neural network are updated, and the process proceeds to step 7. Step 7: Update the strategy using the data obtained from the current batch sampling, using the following formula: In the formula, It is the number of loops within the internal batch. It is a loss function. These are parameters for commenting on deep neural networks. These are the parameters of the action deep neural network. Indicates the current state. Indicates the current state The actions taken Indicates the learning rate. express Relative to action The loss gradient, express Compared to The loss gradient is obtained by selecting the current... Value functions As The estimated value of the value function shows that the policy gradient upon which policy improvement depends is determined by... The value function is derived by taking multiple derivatives; the method for updating the policy objective network parameters is along the value function. The mapping gradient is upward, causing the action policy to... Towards The direction of the value function increase changes, and then the parameters of the deep neural network are updated for the action: In the formula, It is a hyperparameter between 0 and 1 that determines the degree of soft updates, followed by in-batch iterations. Return to step 6 to update the neural network. Proceed to step 8; Step 8: Determine the current time Total number of rounds Size relationship, if Return to step 3, if The offline training is then complete. The trained agent is then configured into the environment, and the agent's input is the state of the short vertical propulsion system. The output is the control input for the short vertical propulsion system. This enables model-free performance recovery control design that takes into account performance degradation.

Images