A method for direct thrust control of an aero-engine based on reinforcement learning

Abstract

The application discloses an aero-engine direct thrust control method based on reinforcement learning, adopts a reinforcement learning deep deterministic policy gradient algorithm to design an engine direct thrust control end-to-end upper controller, constructs reinforcement learning model state information based on flight altitude, Mach number, relative thrust error and engine state variables, and obtains complex global decisions online through the reinforcement learning model. The method uses traversal sampling as an initialization dataset to guarantee initialization stability, uses a sample pool and a fixed target network to improve learning convergence. When the engine working condition changes or the engine deteriorates, the method realizes online optimization of engine performance under the premise of safe and stable engine operation, overcomes the limitation that traditional thrust control technology needs offline optimization, fully excavates the performance potential of the engine, and meets target thrust requirements.

Description

Technical Field

[0001] This invention specifically relates to a direct thrust control method for aero-engines based on reinforcement learning, belonging to the field of direct thrust control technology for aero-engines. Background Technology

[0002] For aero engines, their primary function is to provide thrust to the aircraft. Traditional sensor-based control methods cannot directly control key parameters such as thrust or surge margin. Therefore, direct thrust control is a key feature of the new generation of combat aircraft, which aims to maximize the engine's performance in different operating modes. This is one of the core technologies of intelligent aero engine control and has attracted widespread attention from engineers.

[0003] The thrust control strategy module is a crucial component of the direct thrust control system. Traditional aero-engine control systems, in their initial design, sacrifice engine performance potential to maintain a large safety margin, ensuring stable engine operation. However, with the rapid development of advanced aero-engine technology, the performance requirements for aero-engines during flight are increasingly stringent. The safety margin initially reserved in the design has, to some extent, suppressed the engine's performance potential. Traditional methods for obtaining thrust control strategies often employ optimization algorithms such as linear programming (LP), sequential quadratic programming (SQP), genetic algorithms (GA), and particle swarm optimization to obtain optimal commands offline. Therefore, when engine operating conditions change or the engine deteriorates, offline optimal commands may no longer meet the optimal performance targets under the current state, exhibiting limitations.

[0004] Deep reinforcement learning, as a novel machine learning-based control algorithm, has demonstrated excellent performance in intelligent control fields such as robotics and unmanned aerial vehicles (UAVs). Unlike other machine learning algorithms, deep reinforcement learning combines the feature extraction capabilities of deep neural networks with the decision-making and control capabilities of reinforcement learning. It utilizes computerized human knowledge to continuously interact with and accumulate experience with the controlled object through intelligent perception and intelligent decision-making, enabling the intelligent system to learn and solve task strategies through self-trial and error in a given environment, thereby improving the autonomy, robustness, and intelligence level of control tasks. It has shown excellent performance in tasks in fields such as robot control, computer vision, natural language processing, and autonomous driving, which use high-dimensional data as input. It has received key attention and research in the control of aircraft such as aviation, aerospace, and missiles, and is a future research topic in aircraft dynamics and control. The engine control algorithm based on deep reinforcement learning has the following advantages: (1) It does not require complex manual parameter design; (2) It has adaptive learning capabilities; (3) It uses deep neural networks, which have strong model expressive capabilities and can directly map the perceived state to the engine control signal; (4) It does not require a large amount of data to be provided in advance. Through self-sampling and self-learning methods, it samples data from the environment while training its own strategy. Currently, there is a lack of research on applying deep reinforcement learning to the field of direct thrust control of aero-engines. Summary of the Invention

[0005] Purpose of the invention:

[0006] To overcome the shortcomings of existing technologies and make up for the limitations of traditional thrust control strategies that require offline optimization, and to fully tap the performance potential of engines, this invention provides a direct thrust control method for aero-engines based on reinforcement learning. This method uses reinforcement learning to design an end-to-end upper-level controller for direct thrust control of the engine. When the engine's operating conditions change or the engine deteriorates, the controller can provide complex global decisions online to meet the target thrust requirements.

[0007] Technical solution:

[0008] A reinforcement learning-based direct thrust control method for aero-engines includes the following steps:

[0009] 1. A direct thrust control method for aero-engines based on reinforcement learning, characterized by comprising the following steps:

[0010] Step S1: Select the aircraft's flight altitude H and flight Mach number M. a The Markov process state information S of the reinforcement learning model for direct thrust control of aero-engine is constructed using the thrust relative error ΔF and engine state variable Y, and the action information a is constructed by selecting engine adjustable variable U.

[0011] Step S2: Based on the trained reinforcement learning model, input the current state information S of the direct thrust control reinforcement learning model. t Output action information a t ;

[0012] Step S3, based on action information a t The control quantity U of the engine's adjustable variable is obtained and applied to the engine to change its state.

[0013] Step S4: Obtain the next-time state information S of the direct thrust control reinforcement learning model. t+1 Proceed to step S2;

[0014] The aforementioned aero-engine direct thrust control reinforcement learning model is trained using a deep deterministic policy gradient algorithm, comprising two Actor-Critic networks: an Online network and an Estimation Network (Target network). Each AC network includes an Actor action network and a Critic evaluation network. The training steps are as follows:

[0015] Step 1), based on variable flight altitude H and variable flight Mach number M a The engine state variable Y is randomly sampled by varying the relative thrust error ΔF and the engine adjustable variable U to form the initial state dataset {S} of the direct thrust control reinforcement learning model.

[0016] Step 2), set the pool capacity to M, set the action network exploration noise variance to N, and initialize the number of interaction data m = 0;

[0017] Step 3) Randomly select state data from the dataset {S} to initialize the state information S at the start of the round;

[0018] Step 4), based on the current state information S t Action information a is obtained through the Actor Action Network. t It acts on the engine, causing a change in the engine's state and returning a reward value r. t and the state information S at the next moment t+1 ;

[0019] Step 5), transfer the interaction data [S] t a t r t S t+1 Store the data in the sample pool, and let m = m + 1;

[0020] Step 6): If m > M, then randomly select a portion of historical interaction data based on the historical data in the sample pool to update the network parameters of the Target network.

[0021] Step 7): If the number of training steps in a round reaches the set value or the engine state variable Y exceeds the limit value, the training round ends; otherwise, proceed to step 4.

[0022] Step 8), then jump to step 3 until the preset number of training sessions is reached, then the training ends.

[0023] In some preferred embodiments, the formula for calculating the relative thrust error ΔF in step S1 is:

[0024] ΔF=F r -F

[0025] In the formula, F r is the thrust command value, and F is the thrust value calculated by the high-precision airborne real-time adaptive model based on tracking filter correction.

[0026] In some preferred embodiments, the engine state variable Y mentioned in step S1 consists of variables that determine the engine state, including the high-pressure rotor speed N. h Low-pressure rotor speed N l Fan pressure ratio π Fan Compressor pressure ratio π Com High-pressure turbine pressure ratio π Hpt Low-pressure turbine pressure ratio π Lpt That is, Y = [N h N l , π Fan , π Com π Hpt , π Lpt ].

[0027] In some preferred embodiments, the engine adjustable variable U mentioned in step S1 includes the fuel flow rate W. f Guide vane angle α, nozzle area A8, i.e., U = [W f ,α,A8].

[0028] In some preferred embodiments, the formula for calculating the reward value r in step 4 is:

[0029]

[0030] In the formula, r0 is a constant reward, |ΔF| m For thrust indicators, For fuel consumption rate indicators, SMC≤SMC min The compressor's minimum surge margin is a constraint limitation. To constrain the minimum oil-gas ratio, W f ≤W fmin orW f ≥W fmaxThe maximum and minimum adjustable range of fuel flow is limited by A8 ≤ A 8min orA8≥A 8max To limit the maximum and minimum adjustable range of the nozzle area, α VSV ≤α VSVmin orα VSV ≥α VSVmax The maximum and minimum adjustable range of the guide vane angle is limited by k1, k2, and k3, which are weighting coefficients.

[0031] Beneficial effects of this invention:

[0032] This invention employs reinforcement learning to design an end-to-end upper-level controller for direct thrust control of an engine. Using a sample pool and a fixed target network makes the learning process more stable and ensures convergence. When engine operating conditions change or the engine deteriorates, under the premise of safe and stable engine operation, the controller can provide complex global decisions online, achieving online optimization of engine performance to meet target thrust requirements. This overcomes the shortcomings of existing technologies, compensates for the limitations of traditional thrust control strategies that require offline optimization, and fully taps the engine's performance potential. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a specific embodiment of the method of the present invention.

[0034] Figure 2 Schematic diagram of direct thrust control principle

[0035] Figure 3 Actor-Critic Framework Structure Diagram for Reinforcement Learning

[0036] Figure 4 Schematic diagram for updating the gradient algorithm for deep deterministic policies

[0037] Figure 5 Training results of reinforcement learning model for direct thrust control of aero-engines Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] See appendix Figure 1 Here is a flowchart of the method of the present invention, which specifically includes the following steps:

[0040] Step S1: Select the aircraft's flight altitude H and flight Mach number M. aThe Markov process state information S of the reinforcement learning model for direct thrust control of aero-engine is constructed using the thrust relative error ΔF and engine state variable Y, and the action information a is constructed by selecting engine adjustable variable U.

[0041] Step S2: Based on the trained reinforcement learning Actor network, input the current state information S of the direct thrust control reinforcement learning model. t Output action information a t ;

[0042] Step S3, based on action information a t The control quantities of the engine's adjustable variables are obtained and applied to the engine to change its state.

[0043] Step S4: Obtain the next-time state information S of the direct thrust control reinforcement learning model. t+1 Proceed to step S2;

[0044] The aforementioned aero-engine direct thrust control reinforcement learning model is trained using a deep deterministic policy gradient algorithm, comprising two Actor-Critic networks: an Online network and an Estimation Network (Target network). Each AC network includes an Actor action network and a Critic evaluation network. The training steps are as follows:

[0045] Step 1), based on variable flight altitude H and variable flight Mach number M a The engine state variable Y is randomly sampled by varying the relative thrust error ΔF and the engine adjustable variable U to form the initial state dataset {S} of the direct thrust control reinforcement learning model.

[0046] Step 2), set the pool capacity to M, set the Actor network exploration noise to N, and initialize the number of interactive data m = 0;

[0047] Step 3) Randomly select state data from the dataset {S} to initialize the state information S at the start of the round;

[0048] Step 4), based on the current state information S t Action information a is obtained through the Actor Action Network. t It acts on the engine, causing a change in the engine's state and returning a reward value r. t and the state information S at the next moment t+1 ;

[0049] Step 5), transfer the interaction data [S] t a t r t S t+1Store the data in the sample pool, and let m = m + 1;

[0050] Step 6): If m > M, then randomly select a portion of historical interaction data based on the historical data in the sample pool to update the network parameters of the Target network.

[0051] Step 7): If the number of training steps in a round reaches the set value or the engine state variable Y exceeds the limit value, the training round ends; otherwise, proceed to step 4.

[0052] Step 8), then jump to step 3 until the preset number of training sessions is reached, then the training ends.

[0053] To more clearly illustrate the invention method for direct thrust control of aero-engines based on reinforcement learning, the steps of each embodiment are described in detail below with reference to the accompanying drawings.

[0054] Step S1: Select the aircraft's flight altitude H and flight Mach number M. a The Markov process state information S of the direct thrust control reinforcement learning model of the aero-engine is constructed using the thrust relative error ΔF and engine state variable Y, and the action information a is constructed by selecting the engine adjustable variable U.

[0055] The formula for calculating the relative thrust error ΔF is as follows:

[0056] ΔF=F r -F

[0057] In the formula, F r is the thrust command value, and F is the thrust value calculated by the high-precision airborne real-time adaptive model based on tracking filter correction.

[0058] The engine state variable Y consists of variables that determine the engine state, including the high-pressure rotor speed N. h Low-pressure rotor speed N l Fan pressure ratio π Fan Compressor pressure ratio π Com High-pressure turbine pressure ratio π Hpt Low-pressure turbine pressure ratio π Lpt ,Right now:

[0059] Y = [N] h N l , π Fan , π Com , π Hpt , π Lpt ]

[0060] Engine adjustable variables U, including fuel flow rate W f Guide vane angle α, tail nozzle area A8, that is:

[0061] U = [Wf , α, A8]

[0062] Secondly, a direct thrust control reinforcement learning model is built based on the deep deterministic policy gradient algorithm, including two Actor-Critic networks: an online network and an estimating network (target network). Each AC network includes an Actor action network and a Critic evaluation network. In this embodiment, the deep deterministic policy gradient algorithm uses a deep neural network to approximate the policy network (Actor) and the value network (Critic), and employs reinforcement learning for training. The Actor-Critic framework structure is shown in the attached figure. Figure 3 As shown, it contains two neural networks: a policy network Actor that generates actions, with network parameters θ, input state S, and output action a, represented as:

[0063] a = μ θ (S)

[0064] An evaluation network called Critic judges the quality of an action 'a' generated by a policy network called Actor, assigning a score. Its network parameters are ω, and it takes the state S and action 'a' as inputs to obtain the evaluation index Q, denoted as Q. ω (S, a).

[0065] Taking the reinforcement learning model for direct thrust control of an engine as an example, the policy network Actor selects actions based on probability: inputting engine state information S, it outputs action a; the evaluation network Critic evaluates the actions based on the Actor's actions: inputting state information S and action a, it outputs an index evaluating the quality of the action. The formula for calculating the maximum future reward Q-value is:

[0066] Q ω (s t a t )=r(s t a t )+γQ ω (s t+1 a t+1 )

[0067] In the formula, the subscript t represents the variable at the current time, t+1 represents the variable at the next time, r is the reward function under the state and action at time t, and γ is the discount factor.

[0068] In this embodiment, the discount factor γ is set to 0.9, the number of training sessions is set to 200, and the number of training steps in each round is set to 2000.

[0069] The training steps for its direct thrust control reinforcement learning model are as follows:

[0070] Step 1), based on variable flight altitude H and variable flight Mach number M a The engine state variable Y is randomly sampled by varying the relative thrust error ΔF and the engine adjustable variable U, forming the initial state dataset {S} of the direct thrust control reinforcement learning model.

[0071] Because aero-engines have strong nonlinearity, the state initialization of reinforcement learning models based on engine direct thrust control cannot use random initialization of each variable within a certain range. Otherwise, it is easy to cause the steady-state calculation to fail to converge during initialization. Therefore, traversal sampling is used as the initialization dataset to ensure the rationality of initialization and the stability of the training process.

[0072] Step 2), set the sample pool capacity to M = 10000, set the Actor network exploration noise variance to N = 3, and initialize the number of interaction data m = 0.

[0073] Step 3): Based on the dataset {S} obtained in Step 1), randomly select state data from {S} to initialize the state information S at the start of the round.

[0074] Step 4), based on the current state information S t Action information a is obtained through the Actor Action Network. t It acts on the engine, causing a change in the engine's state and returning a reward value r. t and the state information S at the next moment t+1 .

[0075] The formula for calculating the reward value r in step 4) is as follows:

[0076]

[0077] In the formula, r0 is a constant reward, |ΔF| m For thrust indicators, For fuel consumption rate indicators, SMC≤SMC min The compressor's minimum surge margin is a constraint limitation. To constrain the minimum oil-gas ratio, W f ≤W fmin orW f ≥W fmax The maximum and minimum adjustable range of fuel flow is limited by A8 ≤ A 8min orA8≥A 8max To limit the maximum and minimum adjustable range of the nozzle area, α VSV ≤α VSVmin orα VSV ≥α VSVmax The maximum and minimum adjustable range of the guide vane angle is limited by k1, k2, and k3, which are weighting coefficients.

[0078] Step 5), transfer the interaction data |S t a t r t S t+1 Store the data in the sample pool, and let m = m + 1;

[0079] As attached Figure 4 As shown, the update of the policy network Actor parameter θ is based on the score of the evaluation network Critic (parameter ω), which is achieved by maximizing the score Q, i.e. minimizing the loss function Loss. The Loss calculation formula is:

[0080] Loss=-Q ω (s, a) = -Q ω (s, μ) θ (s)).

[0081] The Loss function is a nested compound function. In the update of the policy network Actor (parameter θ), only the parameter θ is updated, and the parameter ω of the evaluation network Critic is not updated.

[0082] The evaluation network updates its Critic parameter ω by minimizing the loss function Loss, enabling the network to obtain the correct score Q based on the feedback reward function. This closely approximates the correct future maximum reward calculated using the time difference method. The Loss calculation formula is as follows:

[0083]

[0084] In the formula, s, a, and r represent the current state, action, and reward value, while s' and a' represent the next state and action.

[0085] Since the Actor's update depends on the Critic's score, updating the Critic and Actor together makes the network more difficult to converge. Therefore, a fixed target network is used to make the learning process more stable and ensure convergence. Sample pooling is used to break data correlation and accelerate convergence. The fixed target network replicates the structure of the Actor and Critic networks, corresponding to the target network target_P (parameters...). ) and target network target_Q network (parameters) The parameters of the target network are not updated in real time. Instead, the parameters of the current AC network are updated into the target network after a certain number of steps to cut off the correlation and improve convergence.

[0086] Step 6): If m > M, then randomly select a portion of historical interaction data based on the historical data in the sample pool to update the network parameters of the Target network.

[0087] Set the number of training sessions to 200 and the number of training steps per round to 2000.

[0088] Step 7): If the number of training steps in a round reaches the set value of 2000 or the engine state variable Y exceeds the limit value, the training round ends; otherwise, proceed to step 4.

[0089] Considering the performance limitations of the engine during operation, to ensure that the engine does not surge or stall, and to maintain the controllable range of the control variables, the training round is stopped when one of the engine's state variables exceeds the limit value.

[0090] Step 8), then jump to step 3, until the preset number of training times of 200 is reached, then the training ends.

[0091] See appendix Figure 5 The training results are for the reinforcement learning model of direct thrust control of aero-engines. As the number of training iterations increases, the reinforcement learning model eventually converges.

[0092] Step S2: Based on the trained Actor network, input the current state information S of the direct thrust control reinforcement learning model. t Output action information a t .

[0093] A trained deep reinforcement learning network is an independent function. With a trained deep neural network, the state information of the direct thrust control reinforcement learning model can be input, and the optimal controller parameters can be obtained online through the independent function without having to solve linear or nonlinear optimization problems online.

[0094] Step S3, based on action information a t The control quantities of the engine's adjustable variables are obtained and applied to the engine to change its state.

[0095] Step S4: Obtain the next-time state information S of the direct thrust control reinforcement learning model. t+1 Proceed to step S2.

[0096] This invention employs reinforcement learning to design an end-to-end upper-level controller for direct thrust control of an engine. Using a sample pool and a fixed target network makes the learning process more stable and ensures convergence. When engine operating conditions change or the engine deteriorates, under the premise of safe and stable engine operation, the controller can provide complex global decisions online, achieving online optimization of engine performance to meet target thrust requirements. This overcomes the shortcomings of existing technologies and compensates for the limitations of traditional thrust control strategies that require offline optimization, fully tapping the engine's performance potential while ensuring safe and stable engine operation.

[0097] This invention is not limited to the above embodiments. Based on the technical solutions disclosed in this invention, those skilled in the art can make some simple modifications, equivalent changes and alterations to some of the technical features without creative effort, all of which fall within the scope of the technical solutions of this invention.

Claims

1. A method for aeroengine direct thrust control based on reinforcement learning, characterized in that, Comprise the following steps: Step S1, select the aircraft flight height H, flight Mach number M a , thrust relative error ΔF, engine state variable Y Markov process state information S of building aero-engine direct thrust control reinforcement learning model, select engine adjustable variable U to build action information a; Step S2, based on the trained reinforcement learning Actor network, input the state information S of the direct thrust control reinforcement learning model at the current moment t , output action information a t ; Step S3, based on the action information a t obtaining a control amount of an engine variable U, which acts in the engine, so that the engine state changes; Step S4, obtain the state information S of the next moment under the direct thrust control reinforcement learning model t+1 , jump to step S2; The aero-engine direct thrust control reinforcement learning model adopts a deep deterministic policy gradient algorithm for training, and comprises two Actor-Critic networks: a reality network and an estimation network, each AC network comprising an Actor action network and a Critic evaluation network, and the training steps are: Step 1), based on variable flight altitude H, variable flight Mach number M a , variable thrust relative error AF, variable engine adjustable variable U, random sampling of engine state variable Y, to constitute the initialization state data set {S} of the direct thrust control reinforcement learning model; Step 2), set the pool capacity to M, set the action network exploration noise variance to N, and initialize the interaction data number m=0; Step 3), randomly select state data from the data set {S}, and initialize the state information S at the beginning of the round; Step 4), based on the current state information S t , through the Actor action network, the action information a t , acts on the engine, so that the engine state changes, and the reward value r t and the state information S t+1 of the next moment are returned; Step 5), store the interaction data [S t , a t , r t , S t+1 ] in the sample pool, let m = m + 1; Step 6), if m>M, a part of the historical interaction data is randomly extracted based on the sample pool historical data to update the network parameters of the estimation network; Step 7), if the round training step reaches the set value or the engine state variable Y exceeds the limit value, the round training is ended, otherwise jump to step 4; Step 8), jump to step 3 until the preset training number is reached, and then the training is ended.

2. The reinforcement learning based aeroengine direct thrust control method of claim 1, wherein, The thrust relative error ΔF calculation formula in step S1 is: ΔF = F r - F In the formula, F r F is a thrust command value, and F is a thrust value calculated based on a high-precision onboard real-time adaptive model of a tracking filter correction.

3. The reinforcement learning based aeroengine direct thrust control method of claim 1, wherein, The engine state variable Y described in step S1 is composed of variables that determine the engine state, including high-pressure rotor speed N h , low-pressure rotor speed N1, fan pressure ratio π Fan , compressor pressure ratio π Com , high-pressure turbine pressure ratio π Hpt , low-pressure turbine pressure ratio π Lpt , that is, Y = [N h , N l , π Fan π Com , π Hpt , π Lpt ] 4. The reinforcement learning based aeroengine direct thrust control method of claim 1, wherein, The engine of step S1 can adjust variables U, including fuel flow W f , vane angle a, and nozzle area A8, i.e., U = [W f , a, A8].

5. The reinforcement learning based aeroengine direct thrust control method of claim 1, wherein, The reward value r calculation formula in step 4 is: where r0is a constant value reward, |ΔF| m is a thrust index, is a fuel consumption index, SMC≤SMC min is a compressor minimum surge margin constraint limit, is a minimum fuel-air ratio constraint limit, W f ≤W fmin orW f ≥W fmax is a maximum and minimum adjustable range limit of fuel flow, A8≤A 8min orA8≥A 8max is a maximum and minimum adjustable range limit of nozzle area, α VSV ≤α VSVmin orα VSV ≥α VSVmax is a maximum and minimum adjustable range limit of guide vane angle, k1, k2, k3 are weight coefficients.

Images