An intelligent control method for aircraft engines based on data-driven simulation environment

Abstract

The present invention relates to an intelligent control method for an aeroengine based on a data-driven simulation environment. An open-loop experiment is conducted on an aeroengine for which an intelligent controller needs to be designed, and the original data of the aeroengine is collected; a NARX neural network is trained using a training set, and an aeroengine neural network model is obtained after verification by a test set data; a deep deterministic policy gradient DDPG algorithm is used to perform interactive training with an aeroengine data-driven simulation environment to obtain an aeroengine control strategy; the control strategy is adjusted in actual interaction until an optimal control strategy is obtained; and the obtained optimal aeroengine control strategy is deployed on embedded hardware such as a Raspberry Pi to obtain an aeroengine intelligent controller. The present invention combines a data-driven simulation environment with a deep reinforcement learning algorithm, and the designed reinforcement learning controller has high control accuracy, and the control strategy obtained by training is transferred to a real engine system for good control effect.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft engine control and relates to an aircraft engine intelligent control method based on a data-driven simulation environment. The data-driven simulation environment is combined with a deep reinforcement learning algorithm to achieve high-precision intelligent control of the aircraft engine. Background Art

[0002] Aircraft engines are typical multivariable, strongly coupled, and time-varying control systems. As aircraft performance improves, the structure of engines becomes more and more complex. Traditional control methods are effective in solving linear and deterministic problems, but they can no longer meet the increasing control performance requirements in aircraft engine control. Artificial intelligence-based control methods have opened up new solutions. Reinforcement learning is an important branch of artificial intelligence algorithms and has made great progress in recent years. Deep reinforcement learning algorithms that combine deep learning with reinforcement learning have been widely used in engineering optimization and control problems due to their ability to solve continuous problems, such as robot control, signal processing, pattern recognition, and traffic signal control, but they are still in their infancy in the field of aircraft engine control.

[0003] Since reinforcement learning algorithms have low learning efficiency and safety issues when directly applied to aircraft engine systems, researchers usually train deep reinforcement learning algorithms in a simulated environment and then migrate the strategy to the real world. Therefore, an accurate simulation environment is particularly important for the application of deep reinforcement learning algorithms in aircraft engine control. The aircraft engine simulation environment based on the engine working mechanism relies heavily on the characteristics of engine components, has low model accuracy, and cannot perform comprehensive analytical modeling under many conditions. It is difficult to accurately reflect the operating characteristics of the real engine, which poses a huge challenge to the implementation of deep reinforcement learning algorithms in the field of aircraft engine control. Therefore, the current application of deep reinforcement learning algorithms in the field of aircraft engines and the construction of accurate simulation environments have certain limitations and research gaps. Summary of the invention

[0004] Technical issues to be solved

[0005] In order to avoid the shortcomings of the prior art, the present invention proposes an intelligent control method for aircraft engines based on a data-driven simulation environment, which solves the problem that reinforcement learning controllers designed based on mechanism models or mathematical models are difficult to implement in practice, so that the control strategy designed based on the deep reinforcement learning algorithm can be successfully migrated to the actual aircraft engine control system, thereby greatly improving the control accuracy, response speed and robustness of the aircraft engine control system.

[0006] Technical Solution

[0007] An intelligent control method for an aeroengine based on a data-driven simulation environment is characterized by the following steps:

[0008] Step 1: Conduct an open-loop experiment on the aircraft engine for which an intelligent controller is required, and collect the fuel flow rate W of the aircraft engine. f and the rotor speed N p Data, select the fuel flow rate as the control signal, and the rotor speed as the controlled signal;

[0009] Step 2: After denoising the collected data, normalize it:

[0010]

[0011]

[0012] Where: W f 、N p is the original data of fuel flow and rotor speed, W f ′、N p ′ is the normalized data of fuel flow and rotor speed, maxW f 、minW f is the maximum and minimum value of the original data of fuel flow, maxN p 、minN p The maximum and minimum values ​​of the original data of the rotor speed are obtained, and the aero-engine experimental data set is divided into a training set and a test set according to a ratio of 7:3;

[0013] Step 3: Use the training set to train the NARX neural network, and after verification by the test set data, obtain the aircraft engine neural network model;

[0014] The NARX neural network is represented by a nonlinear difference equation of external input:

[0015] N p ′(k)=f[N p ′(k-1), N p ′(k-2),...,W f ′(k-1),W f ′(k-2)...]

[0016] Where f(·) represents a specific nonlinear function, N p ′(tn y ) represents the corresponding engine output parameters and the corresponding delay n y The value of W f ′(tn u ) represents the corresponding engine input parameters and the corresponding delay n u Numeric value;

[0017] Step 4: Based on the aircraft engine data-driven model, define the action space A according to the control structure t , define the state space to represent the operating state S of the engine t , define the reward function R t Guide the intelligent agent to achieve control objectives and form a data-driven simulation environment for aircraft engines;

[0018] Step 5: Use the deep deterministic policy gradient (DDPG) algorithm to interactively train with the aircraft engine data-driven simulation environment to obtain the aircraft engine control strategy;

[0019] Step 6: Based on the control strategy in step 5, interactive training of the DDPG algorithm and the aircraft engine is carried out in the experiment, and the control strategy is adjusted in the actual interaction until the optimal control strategy is obtained; the optimal aircraft engine control strategy is deployed on embedded hardware such as Raspberry Pi to obtain an aircraft engine intelligent controller.

[0020] The NARX neural network model in step 3 outputs d at the i-th hidden layer point at a certain time k. i for:

[0021]

[0022] In the formula, f sig (·) is the activation function used by the hidden layer nodes; n u is the external input delay length; n y w is the output feedback delay; ir (k) is the weight between the i-th hidden layer node and the input u(tr) at time k; w il (k) is the weight between the i-th hidden layer node and the output feedback y(tl) at time k; b i is the threshold of the i-th hidden layer node; the j-th output layer node output y j (k) is:

[0023]

[0024] In the formula, w ji (k) is the weight from the i-th hidden layer node to the j-th output layer node at time k; T j is the jth output layer node threshold; N is the number of hidden layer nodes.

[0025] The action space, state space and reward function of the aero-engine data-driven simulation environment in step 5 may be: action space A t = [u(t)] or A t =[k p , k i], where u(t) represents the aircraft engine control signal, k p , k i Represents the PI controller parameters, and defines the state space to represent the operating state S of the engine t =[e N , ∫e N 、N p ], where e N represents the error between the reference rotor speed and the actual rotor speed, ∫e N Represents the integral of the rotor speed error, N p represents the rotor speed and defines the reward function Guide the intelligent agent to achieve control objectives and form an aero-engine data-driven simulation environment.

[0026] The interactive training process of the deep deterministic policy gradient DDPG algorithm and the aircraft engine data-driven simulation environment in step 5 is as follows:

[0027] Step a: Before the interaction, first set the training end conditions of the DDPG algorithm and the number of interaction steps per round, initialize the soft update factor ρ, the reward discount factor γ, and initialize the Critic network Q(s, a|θ Q ) parameter θ Q , actor network π(s|θ π ) parameter θ π , initialize the target network parameters Q′ and π′, and assign θ Q ′←θ Q ,θ π ′←θ π , initialize the experience pool D;

[0028] Step b: Initialize the random process N at the beginning of each round t , used to add exploration to the action, N t Choose the Omstein-Uhlenbeck process, which satisfies the following stochastic differential equation:

[0029] dX t =θ(π-X t )dt+σdW t

[0030] Where X t is a random variable, θ>0, x, σ>0 are parameters, W t It is a Brownian process;

[0031] Step c: receiving the initial state S0 of the aircraft engine;

[0032] Step d: In each round of interaction, the DDPG algorithm first selects action A t :

[0033] A t =π(S t |θ π )+N t

[0034] Step e: The aircraft engine data-driven simulation environment returns the reward value R t , transfer to the next state S t+1 ;

[0035] Step f: Store state transfer data pair (S t , A t , R t , S t+1 ) to the experience pool D, let

[0036] Y i =R i +γQ′(S t+1 ,π′(S t+1 |θ π ′)|θ Q ′)

[0037] Step g: Update the Critic network by minimizing the loss function:

[0038]

[0039] Step h: Update the Actor network by policy gradient

[0040]

[0041] Step i: Update the target network by exponential smoothing method:

[0042] θ Q ′←ρθ Q +(1-ρ)θ Q '

[0043] θ π ←ρθ π +(1-ρ)θ π

[0044] When the deep reinforcement learning algorithm converges and the single-round reward value reaches the training end condition, the training ends;

[0045] Step j: When the Actor neural network of the deterministic policy function has converged, the Actor neural network is the aircraft engine control strategy optimized by the deep reinforcement learning algorithm.

[0046] Beneficial Effects

[0047] The present invention proposes an intelligent control method for aircraft engines based on a data-driven simulation environment. For aircraft engines that need to design intelligent controllers, open-loop experiments are conducted to collect the original data of aircraft engines; the NARX neural network is trained using a training set, and after verification by a test set data, an aircraft engine neural network model is obtained; a deep deterministic policy gradient DDPG algorithm is used to perform interactive training with the aircraft engine data-driven simulation environment to obtain an aircraft engine control strategy; the control strategy is adjusted in actual interaction until the optimal control strategy is obtained; the obtained optimal aircraft engine control strategy is deployed on embedded hardware such as Raspberry Pi to obtain an aircraft engine intelligent controller. The present invention combines a data-driven simulation environment with a deep reinforcement learning algorithm to achieve the design of an intelligent controller for aircraft engines based on reinforcement learning.

[0048] Compared with the general controller design method, it has the following advantages: (1) Compared with the general aviation engine simulation environment based on mechanism model and mathematical model, the data-driven simulation environment has higher accuracy and better real-time operation. The designed reinforcement learning controller has high control accuracy, and the control strategy obtained by training can be transferred to the real engine system for good control effect. (2) The control method proposed in the present invention utilizes the trial-and-error learning ability of the reinforcement learning algorithm to learn the optimal control strategy in the continuous interaction between the aviation engine data-driven simulation environment and the deep reinforcement learning algorithm, which can realize the adaptive control of the aviation engine and greatly improve the control accuracy, response speed and robustness of the aviation engine control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 :Flowchart for establishing a data-driven simulation environment for aircraft engines

[0050] Figure 2 : Schematic diagram of NARX neural network structure

[0051] Figure 3 : Schematic diagram of the intelligent control structure of the aircraft engine proposed by the present invention

[0052] Figure 4 : Schematic diagram of DDPG algorithm optimizing PI controller

[0053] Figure 5 :Actor neural network structure

[0054] Figure 6 : Critic neural network structure

[0055] Figure 7 : Comparison of experimental results between the intelligent controller designed based on the method of the present invention and the PI controller DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The embodiment of the present invention takes the intelligent control of the NT-50A micro turbojet engine as an example to further describe the present invention in detail:

[0058] SS1: The NT50A micro turbojet engine consists of an inlet, compressor, combustion chamber, turbine, tail nozzle and other structures. Based on the monotonicity, stability, sensitivity, measurability and controllability of the engine controlled parameters, the fuel flow rate is selected as the control signal and the rotor speed is selected as the performance parameter. The fuel flow rate and rotor speed are used as the input and output variables of the neural network respectively;

[0059] SS2: The process of establishing the data-driven simulation environment for aircraft engines Figure 1 As shown, firstly, by inputting PWM pulses into the fuel pump of the NT50A micro turbojet engine, a fuel flow signal Wf with continuous step changes is obtained, and the corresponding step response of the engine, that is, the rotor speed signal N p , collect the above data and process the data;

[0060] SS3: If there are abnormal situations such as data loss and dislocation in the collected data, data replacement, elimination and filtering operations are performed to complete the preprocessing of the original data;

[0061] SS4: Since the engine fuel flow and rotor speed are data of different dimensions, the preprocessed input and output data are normalized:

[0062]

[0063]

[0064] Where W f 、N p is the original data of fuel flow and rotor speed, maxW f 、minW f is the maximum and minimum value of the original data of fuel flow, maxN p 、minN p The maximum and minimum values ​​of the original data of rotor speed;

[0065] SS5: After the above normalization process, the micro turbojet engine experimental data set is obtained, and the data set is divided into a training set and a test set in a ratio of 7:3;

[0066] SS6: In this embodiment, the NARX neural network model is used to construct the data-driven simulation environment of the NT-50A micro turbojet engine. NARX (Nonlinear Auto Regressive models with Exogenous Inputs) is the full name of the external input nonlinear autoregressive neural network. It is a general recursive neural network composed of a delay module and feedback on the basis of a static multilayer perceptron. Its structure is as follows Figure 2 As shown;

[0067] SS7: Model input is denoted by X narx =[W f ′(k)] T , is the fuel flow of the micro turbojet engine at time k, and the output of the model is recorded as Y narx =[N p ′Ck)] T , is the rotor speed of the micro turbojet engine at time k;

[0068] SS8: The NARX neural network has two hidden layers, with 3 and 10 neuron nodes in each layer respectively, and the hidden layer activation function is the Tansig function:

[0069]

[0070] SS9: NARX neural network can be represented by a nonlinear difference equation with external input:

[0071] N p ′(k)=f[N p ′(k-1), N p ′(k-2),...,W f ′(k-1),W f ′(k-2)....]

[0072] SS10: At a certain time k, the i-th hidden layer outputs d i for:

[0073]

[0074] where f tansig (·) is the Tansig function used by the hidden layer nodes, For the external input delay, is the output feedback delay, w ir (k) is the i-th hidden layer node at time k and the input W f The weight between ′(tr), w il (k) is the i-th hidden layer node and output feedback quantity N at time k p′(kl), b i is the threshold of the i-th hidden layer node;

[0075] SS11: The jth output layer node outputs y jk (k) is:

[0076]

[0077] In the formula, w ji (k) is the weight from the i-th hidden layer node to the j-th output layer node at time k, T j is the jth output layer node threshold, N is the number of hidden layer nodes;

[0078] SS12: Train the NARX neural network using training set data;

[0079] SS13: Use the test data to verify the accuracy of the trained NARX neural network model. If the accuracy meets the requirements, the modeling process ends; if the accuracy does not meet the requirements, adjust the relevant parameters and repeat the above steps until the output accuracy of the model meets the requirements.

[0080] SS14: After the NARX neural network model is trained, the underlying controller is used to directly control the aircraft engine data-driven model, and the deep reinforcement learning algorithm optimizes the underlying controller parameters. The control structure is shown in Figure 3 , where the underlying controller is a PI controller;

[0081] SS15: Define the action space as A = [k p , k i ], where k p , k i are the parameters of the low-level controller, i.e., the PI controller;

[0082] SS16: Define the state space as S = [e N , ∫e N 、N p ], where e N It indicates the error between the reference speed and the actual speed, and characterizes the actual operating state of the aircraft engine;

[0083] SS17: Define the reward function as Guide the agent to achieve control goals;

[0084] SS18: The neural network model, action space, state space, and reward function together constitute the data-driven simulation environment of aircraft engines;

[0085] SS19: From the action space and state space defined above, we can see that this control problem is a continuous state action problem. Therefore, this example selects the deep deterministic policy gradient (DDPG) algorithm. The core structure of this algorithm is the Actor neural network and the Critic neural network. The principle of optimizing the PI controller based on the DDPG algorithm is as follows: Figure 4 As shown, the Actor neural network structure is as follows Figure 5 As shown, the structure of the Critic neural network is as follows Figure 6 As shown;

[0086] SS20: The interaction process between the DDPG algorithm and the data-driven simulation environment is as follows:

[0087] SS21: The end condition of deep reinforcement learning algorithm training is set as the cumulative reward value of a single round reaches 95% of the maximum cumulative reward value. The simulation time of a single round is 20s, and the sampling interval is 0.025s. Based on this, the number of single-round simulation interactions is 800 times, and the maximum cumulative reward value of a single round is:

[0088]

[0089] SS22: Initialize soft update factor ρ = 0.001, reward discount factor γ = 0.90, initialize critic network Q(s, a|θ Q ) parameter θ Q , actor network π(s|θ π ) parameter θ π , initialize the target network parameters Q′ and π′, and assign θ Q ′←θ Q ,θ π ′←θ π , initialize the experience pool D

[0090] SS23: Initialize the random process N at the beginning of each round of interaction between the data-driven aviation engine simulation environment and the DDPG algorithm t , used to add exploration to the action, N t Choose the Omstein-Uhlenbeck process, which satisfies the following stochastic differential equation:

[0091] dX t =θ(π-X t )dt+σdW t

[0092] Where X t is a random variable, θ>0, x, σ>0 are parameters, W t It is a Brownian process;

[0093] SS24: Receive the initial state S0 of the aircraft engine;

[0094] SS25: In each round of interaction, the DDPG algorithm first selects action A t :

[0095] A t =π(S t |θ π )+N t

[0096] SS26: Action A t =[k p , k i ], and get the fuel flow rate of the micro turbojet engine:

[0097] W f =k p ×e N +k i ×∫e N

[0098] The aircraft engine data-driven simulation environment returns the reward value R t , transfer to the next state S t+1 ;

[0099] SS27: Storage state transfer data pair (S t , A t , R t , S t+1 ) to D command

[0100] Y i =R i +γQ′(S t+1 ,π′(S t+1 |θ π ′)|θ Q ′)

[0101] SS28: Update the Critic network by minimizing the loss function:

[0102]

[0103] SS29: Updating the Actor Network via Policy Gradients:

[0104]

[0105] SS30: Update the target network by exponential smoothing method:

[0106] θ Q ′←ρθ Q +(1-ρ)θ Q '

[0107] θ π′←ρθ π +(1-ρ)θ π '

[0108] SS31: In each round, the deep reinforcement learning algorithm and the data-driven simulation environment interact for 800 steps in a cycle. In each interaction, the Actor neural network is updated to output the strategy, and the Critic neural network is updated to estimate the Q value. The training ends when the training converges and the cumulative reward value of a single round reaches 95% of the maximum cumulative reward value.

[0109] SS32: At this point, the Actor neural network as a deterministic policy function has converged. The Actor neural network is the optimal strategy for aircraft engine control optimized by the deep reinforcement learning algorithm.

[0110] SS33: The output of the Actor neural network, i.e. the parameters of the PI controller, is applied to the NT50A micro turbojet engine to obtain the following experimental results: Figure 7 As shown, in terms of control response speed.

[0111] According to the experimental results Figure 7 The control method proposed in the present invention is significantly faster than the traditional PID control. In the speed ranges of 40000-50000r / min and 60000-70000r / min, the rise time of the control method proposed in the present invention is nearly 20s faster than that of the PID controller. The adjustment time of the RL-PID controller in each speed range is nearly 12s faster than that of the PID controller. These results prove that the controller designed according to the method proposed in the present invention has a strong adaptive fast tracking capability.

Claims

1. An intelligent control method for an aircraft engine based on a data-driven simulation environment, characterized in that Here are the steps: Step 1: Conduct an open-loop experiment on the aircraft engine for which an intelligent controller is required, and collect the fuel flow rate W of the aircraft engine. f and the rotor speed N p Data, select the fuel flow rate as the control signal, and the rotor speed as the controlled signal; Step 2: After denoising the collected data, normalize it: Where: W f 、N p is the original data of fuel flow and rotor speed, W f ′、N p ′ is the normalized data of fuel flow and rotor speed, maxW f 、minW f is the maximum and minimum value of the original data of fuel flow, maxN p 、minN p The maximum and minimum values ​​of the original data of the rotor speed are obtained, and the aero-engine experimental data set is divided into a training set and a test set according to a ratio of 7:3; Step 3: Use the training set to train the NARX neural network, and after verification by the test set data, obtain the aircraft engine neural network model; The NARX neural network is represented by a nonlinear difference equation of external input: N p ′(k)=f[N p ′(k-1),N p ′(k-2),…,W f ′(k-1),W f ′(k-2)…] Where f(·) represents a specific nonlinear function, N p ′(tn y ) represents the corresponding engine output parameters and the corresponding delay n y The value of W f ′(tn u ) represents the corresponding engine input parameters and the corresponding delay n u Numeric value; Step 4: Based on the aircraft engine data-driven model, define the action space A according to the control structure t , define the state space to represent the operating state S of the engine t , define the reward function R t Guide the intelligent agent to achieve control objectives and form a data-driven simulation environment for aircraft engines; Step 5: Use the deep deterministic policy gradient (DDPG) algorithm to interactively train with the aircraft engine data-driven simulation environment to obtain the aircraft engine control strategy; Step 6: Based on the control strategy in step 5, interactive training of the DDPG algorithm and the aircraft engine is carried out in the experiment, and the control strategy is adjusted in the actual interaction until the optimal control strategy is obtained; the optimal aircraft engine control strategy is deployed on the Raspberry Pi embedded hardware to obtain the aircraft engine intelligent controller.

2. The intelligent control method for an aircraft engine based on a data-driven simulation environment according to claim 1 is characterized in that: The NARX neural network model in step 3 outputs d at the i-th hidden layer point at a certain time k. i for: In the formula, f sig (﹒) is the activation function used by the hidden layer nodes; n u n is the external input delay length; y w is the output feedback delay; ir (k) is the weight between the i-th hidden layer node and the input u(t–r) at time k; w il (k) is the weight between the i-th hidden layer node and the output feedback y(t–l) at time k; b i is the threshold of the i-th hidden layer node; the j-th output layer node output y j (k) is: In the formula, w ji (k) is the weight from the i-th hidden layer node to the j-th output layer node at time k; T j is the jth output layer node threshold; N is the number of hidden layer nodes.

3. The intelligent control method for an aircraft engine based on a data-driven simulation environment according to claim 1 is characterized in that: The action space, state space and reward function of the aero-engine data-driven simulation environment in step 5 may be: action space A t = [u(t)] or A t =[k p , k i ], where u(t) represents the aircraft engine control signal, k p , k i Represents the PI controller parameters, and defines the state space to represent the operating state S of the engine t =[e N , ∫e N 、N p ], where e N represents the error between the reference rotor speed and the actual rotor speed, ∫e N Represents the integral of the rotor speed error, N p represents the rotor speed and defines the reward function Guide the intelligent agent to achieve control objectives and form an aero-engine data-driven simulation environment.

4. The intelligent control method for an aircraft engine based on a data-driven simulation environment according to claim 1 is characterized in that: The interactive training process of the deep deterministic policy gradient DDPG algorithm and the aircraft engine data-driven simulation environment in step 5 is as follows: Step a: Before the interaction, first set the training end conditions of the DDPG algorithm and the number of interaction steps per round, initialize the soft update factor ρ, the reward discount factor γ, and initialize the Critic network Q(s,a|θ Q ) parameter θ Q , actor network π(s|θ π ) parameter θ π , initialize the target network parameters Q′ and π′, and assign θ Q′ ←θ Q ,θ π′ ←θ π , initialize the experience pool D; Step b: Initialize the random process N at the beginning of each round t , used to add exploration to the action, N t Choose the Ornstein-Uhlenbeck process, which satisfies the following stochastic differential equation: dX t =θ(π-X t )dt+σdW t Where X t is a random variable, θ>0, x, σ>0 are parameters, W t It is a Brownian process; Step c: receiving the initial state S0 of the aircraft engine; Step d: In each round of interaction, the DDPG algorithm first selects action A t : A t =π(S t |θ π )+N t Step e: The aircraft engine data-driven simulation environment returns the reward value R t , transfer to the next state S t+1 ; Step f: Store state transfer data pair (S t ,A t ,R t ,S t+1 ) to the experience pool D, let Y i =R i +γQ′(S t+1 ,π′(S t+1 |θ π′ )|θ Q′ ) Step g: Update the Critic network by minimizing the loss function: Step h: Update the Actor network by policy gradient Step i: Update the target network by exponential smoothing method: θQ′←ρθ Q +(1-p)θ Q′ i π′ ←ρθ π +(1-p)θ π′ When the deep reinforcement learning algorithm converges and the single-round reward value reaches the training end condition, the training ends; Step j: When the Actor neural network of the deterministic policy function has converged, the Actor neural network is the aircraft engine control strategy optimized by the deep reinforcement learning algorithm.

Images