Aero-engine bleed air temperature fault-tolerant control method based on reinforcement learning

Abstract

The application discloses an aero-engine bleed air temperature fault-tolerant control method based on reinforcement learning, which is used for continuously and stably outputting the bleed air temperature when efficiency loss and additive failure occur. A controller is designed by introducing a first-order low-pass filter based on a DDPG algorithm of reinforcement learning. The application realizes the stability and good fault-tolerant performance of an engine bleed air temperature control system, and simulation examples prove that the method has certain effectiveness and good performance.

Description

Technical Field

[0001] This invention relates to the field of engine control, and more specifically to a fault-tolerant control method for bleed air temperature in aero-engines. Background Technology

[0002] In recent years, the engine has become the most critical component of an aircraft, constituting a typical nonlinear system. Ensuring its safe and precise control is crucial for smooth flight and successful mission completion. During actual flight, engine failures can have serious consequences, causing the aircraft to deviate from its trajectory or even crash. To detect failures early and mitigate their impact on the aircraft, researchers have conducted studies on engine failure detection, engine failure diagnosis, and engine fault-tolerant control. However, these studies are all based on precise mathematical models of the engine, and considering the effects of disturbances and uncertainties, obtaining accurate models for real-world systems becomes challenging.

[0003] To address the fault tolerance problem in engines, numerous scholars both domestically and internationally have proposed various algorithms. Patent CN116820065A invented a fault-tolerant control method for aero-engine systems with parameter uncertainties based on sliding mode, its main contribution being real-time compensation for system faults, enabling the system to remain stable and achieve the expected H∞ performance index even during faults, thereby improving system reliability. Patent CN114943180A invented an intelligent fault-tolerant control method for engine temperature sensors under electromagnetic shock environments, improving the electromagnetic pulse protection capability of the engine control system. Furthermore, the temperature sensor model is a dynamic model that can be updated in real time, avoiding model accuracy degradation and ensuring fault-tolerant control effectiveness. However, both of these patents require designing fault-tolerant control methods based on mathematical models of the aero-engine or temperature sensor. In practical engineering, accurate mathematical models of aero-engines and temperature sensors are difficult to obtain. This patent proposes a reinforcement learning-based method, a data-driven fault-tolerant control method that does not require mathematical models, reducing the difficulty of use and increasing practical application value. Moreover, no research results on the fault tolerance problem of aero-engine bleed air temperature control systems have been publicly published yet. Summary of the Invention

[0004] In view of the shortcomings of the prior art, this invention proposes a fault-tolerant control method for bleed air temperature of aero-engines based on reinforcement learning, which mainly consists of the following steps:

[0005] Step 1, the fault-tolerant control of the aero-engine bleed air temperature control system based on reinforcement learning according to claim 1, characterized in that the mathematical model of the bleed air temperature control system in step 1 is as follows:

[0006]

[0007]

[0008] Among them, the hot-side fluid (outlet air) has a specific heat C h The mass flow rate is W bleed The inlet temperature is T hi The outlet temperature is T ho The cold-side fluid (intake air) has a specific heat C. c The mass flow rate is W ram The inlet temperature and outlet temperature are respectively T ci and T co H h and H c These represent the overall heat transfer coefficients on the hot and cold sides, respectively;

[0009] Mass flow rate W through the valve v It is calculated using the equations for an isentropic expansion process and is applicable to variable area conduction:

[0010]

[0011] Among them, P u and P d These are the upstream and downstream pressures (psi), T u The upstream temperature is g = 32.174 ft / s. 2 It is the acceleration due to gravity, R = 1717 ft. 2 / (s 2 ·R) is the gas constant of air, γ=1.4 is the ratio of isobaric specific heat to isochoric specific heat; A v It is the opening area of ​​the valve, calculated using the following formula:

[0012]

[0013] β v =x k / u

[0014] Where is the valve diameter, is the valve opening angle, and is the valve opening coefficient.

[0015]

[0016] Where f S =8.40, f D =3.524; Actuator failure is modeled as an equivalent deviation:

[0017] u kf =a*(u k +b)

[0018]

[0019]

[0020] Among them, u kf The numbers indicate actuator faults, with 'a' indicating a multiplication fault and 'b' indicating an addition fault.

[0021] 3. According to the aero-engine bleed air temperature control system of claim 1, the next step is to set the relevant parameters for reinforcement learning and train the DDPG algorithm;

[0022] Reinforcement learning is a field of machine learning in which an agent interacts with an environment and learns a policy π that maximizes the cumulative reward; the environment is modeled as a Markov decision process (MDP) consisting of state S, action A, and transition probabilities P(s). t+1 |s t a t ), reward function R(s) t a t And the definition of the discount factor γ;

[0023] The core objective is to find an optimal policy π that maximizes the expected cumulative reward, expressed as:

[0024]

[0025] Among them, V π (s t Let represent the state-value function under policy π, where represents the expected reward starting from state S and subsequently following π: s t a represents the current state at time t. t Indicates that in state s t The selectable actions, π(a) t |s t ) indicates that in state s t The policy for selecting action a is R(s). t a t ) indicates that in state s t Take action a t The immediate reward obtained later, γ represents the discount factor, indicating the importance of future rewards; P(s) t+1 |s t a t ) indicates that in state s t Take action a t Then transition to the next state s t+1 The probability, V π (s t +1) indicates that under policy π, from the next state s t+1 Initial expected cumulative reward;

[0026] This statement provides the state-value function V at different time steps t. π (s t The evolution of () is more clearly represented, taking into account the current state, action selection, immediate reward, and the value of the next state:

[0027]

[0028] Among them, Q π (s t a t ) represents the action-value function, a t+1 Indicates that in state s t+1 The next action selected, π(a) t +1|s t +1) indicates that in state s t+1 Select action a t+1 The strategy, Q π (s t+1 a t+1 ) indicates that under policy π, from the next state s t+1 and action a t+1 The initial expected cumulative reward; this reflects the recursive definition of action values ​​in reinforcement learning, where the value of the current state and action depends on the immediate reward and the value after transitioning to the next state;

[0029] DDPG (Deep Deterministic Policy Gradient) is a method for simultaneously learning a deterministic policy μ. θ (s) and Q-value function Q w The actor-critic algorithm for (s, a) is designed for environments with continuous action spaces and utilizes a depth function approximator that can handle large state spaces.

[0030] Actor (Strategy) Model:

[0031] Deterministic strategy μ θ The decision-making process outputs an action without exploring noise; it updates J(θ) by increasing the gradient of the expected return.

[0032]

[0033] in, It is the gradient of the objective function J(θ) with respect to the parameters θ of the actor network; E is the expectation operation; s ~ ρ μ This refers to the policy distribution ρ from the actor network. μ Mid-sampling state s; It is the gradient of the critic network's action-value function Q with respect to action a, where w represents the parameters of the critic network; The action μ is selected using the actor network in the gradient.θ (s) Replace action a; The purpose of this equation is to update the parameters of the actor network based on the expectation of the product of the actor's policy gradient and the commentator's action value gradient, aiming to maximize the cumulative reward; During training, this gradient guides the actor network to update its policy in a direction more favorable to achieving higher expected rewards;

[0034] Critic (Value) Model:

[0035] Critics use neural networks with weights w to approximate the action-value function Q. w It is trained by minimizing a loss function, which is the mean squared Bellman error.

[0036]

[0037] Q w (s t a t ) is the commentator network's estimated action-value function based on the current state and action; y is the target value:

[0038] y = (r t +γQ w′ (s t+1 μ θ′ (s t+1 ))

[0039] Target network:

[0040] DDPG utilizes the target network Q w′ and μ θ′ To stabilize training; the target network is updated in the direction of the parameters of the learning network through soft updates:

[0041] w′←τw+(1-τ)w′

[0042] θ′←τθ+(1-τ)θ′

[0043] Where τ is the soft update rate; the equation represents that the parameters w′ / θ′ of the target commentator network are gradually updated by mixing with the parameters w / θ of the current commentator network;

[0044] These equations work together to perform soft updates, gradually bringing the target network parameters closer to the current network parameters in each update; the soft update rate τ controls the rate at which the target network parameters change relative to the current network parameters; this is done to slow down the changes in the target network and ensure a relatively stable target.

[0045] Exploration mechanism:

[0046] The exploration involves adding noise N to the actor's strategy. t To be introduced:

[0047] μ′(s t )=μ θ (s t )+N t

[0048] Where N t Typically, sampling is performed from a noisy process, μ(s) t ) is the actor network in state s t The action produced below, μ θ (s t ) is the actor network in state s t The policy output provided by the actor network provides the necessary exploration for policy learning, enabling it to learn from a diverse set of actions. Therefore, the entire equation represents that the actions generated by the actor network are a combination of deterministic policy output and a certain amount of random noise. This noise is introduced to encourage the exploration of new actions and enhance the algorithm's exploratory behavior.

[0049] 4. The aero-engine bleed air temperature control system according to claim 1 combines the trained results with a first-order low-pass filter. The first-order low-pass filter used is described below:

[0050] The first-order low-pass filtering algorithm, also known as the exponentially weighted moving average (EWMA), is a simple and effective method for smoothing time series data. It is typically used to reduce high-frequency noise while preserving low-frequency components. The core equation of this algorithm is as follows:

[0051] Y(n)=αX(n)+(1-α)Y(n-1)

[0052] Where Y(n) is the current output at time n, X(n) is the current input at time n, Y(n-1) is the previous output at time n-1, and α is a smoothing factor that determines the weights assigned to the current input (0≤α≤1).

[0053] Effectively training an agent to follow the trajectory of control signals requires careful consideration of various design factors when building the environment: these factors can be broadly categorized into agent-related and environment-related considerations.

[0054] Agent-related factors involve the composition of the observation vector and the strategy for defining rewards; on the other hand, environment-related factors include the training strategy, the signals used for training, the initial conditions of the environment, and the criteria for terminating a round; these factors need to be carefully handled to optimize the training process and enhance the agent's ability to navigate and control the trajectory of the signal.

[0055] The structure of the observation vector is as follows: s = [y, e, ∫e.dt] THere, y represents the actual realized flow, e is the error relative to the reference value, and the integral of the error over time; the integral of the error captures the evolution of the error accumulated over time, provides a means to calculate the total error accumulated over time, and guides the agent towards minimizing it; this integral of the error is frequently used as a key observation input in the training of reinforcement learning (RL) controllers:

[0056] The expression below shows a continuously varying reward, e as a function of the error; λ is a small constant used to avoid division by zero errors:

[0057]

[0058] The training mechanism of Deep Deterministic Policy Gradient (DDPG) involves iterative optimization of the actor and critic networks. It begins with the initialization of network parameters and a replay buffer used to store experience. The actor introduces exploratory noise when choosing actions, and the experience generated by the interaction with the environment is stored in the buffer. Then, a small batch of experience is sampled from the experience to update the critic and actor networks, using the critic's mean squared Bellman error. The target network is soft-updated. This process is repeated iteratively to gradually optimize the network, learning an effective policy in the continuous action space through deterministic policy optimization and Q-value estimation.

[0059] Compared with existing technologies, the beneficial effects of the present invention are reflected in the following aspects:

[0060] (1) The reinforcement learning-based control method proposed in this invention has the advantages of fast tracking speed and high control accuracy, and ensures that the output temperature recovers to the desired position quickly after a fault occurs.

[0061] (2) The control method based on reinforcement learning proposed in this invention has a smaller signal oscillation amplitude after being combined with a first-order low-pass filter;

[0062] (3) The reinforcement learning-based control method proposed in this invention demonstrates better stability after a fault occurs compared with PID. Attached Figure Description

[0063] To better demonstrate the advantages of the method designed in this invention, PID control and reinforcement learning methods were compared to address the fault tolerance problem of the bleed air temperature control system. The results show that the method proposed in this invention has better stability and fault tolerance performance.

[0064] Figure 1 This is a diagram illustrating the working principle of the network in the DDPG algorithm.

[0065] Figure 2A comparison of the opening degree under different simulation delays when the training delay is 0 and the low-pass filter coefficient is 0.01;

[0066] Figure 3 A comparison of the opening degree under different simulation delays when the training delay is 0 and the low-pass filter coefficient is 0.05;

[0067] Figure 4 A comparison of the opening degree for different simulation delays when the training delay is 2.5 and the low-pass filter coefficient is 0.01;

[0068] Figure 5 A comparison of the aperture size under different simulation delays when the training latency is 2.5 and the low-pass filter coefficient is 0.05:

[0069] Figure 6 A comparison of the opening degree for different simulation delays when the training delay is 5 and the low-pass filter coefficient is 0.01;

[0070] Figure 7 A comparison of the opening degree for different simulation delays when the training delay is 5 and the low-pass filter coefficient is 0.05;

[0071] Figure 8 The diagram shows the opening tracking performance under different control methods of reinforcement learning and PID, with a training latency of 2.5 and a low-pass filter coefficient of 0.05, under fault-free conditions.

[0072] Figure 9 The output temperature tracking effect is shown under different control methods, namely reinforcement learning and PID, in a fault-free condition.

[0073] Figure 10 The diagram shows the opening tracking performance under different control methods of reinforcement learning and PID control when the training latency is 2.5 and the low-pass filter coefficient is 0.05 under fault conditions.

[0074] Figure 11 The graph shows the output temperature tracking effect under different control methods, namely reinforcement learning and PID, in the event of a fault. Detailed Implementation

[0075] The present invention will be further explained in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0076] To enable those skilled in the art to better understand the implementation of this invention, Matlab 2022b software will be used for simulation to verify its reliability.

[0077] Table 1 DDPG Hyperparameters

[0078] Critics' learning rate <![CDATA[1e -0 3 ]]> Actor learning rate <![CDATA[1e -0 4 ]]> Commentator Hidden Layer 1 50 fully connected layers Commentator Hidden Layer 2 25 fully connected layers Action pathway neurons 25 fully connected layers upper bound of action path Hyperbolic tangent layer γ 0.9 Batch size 64 OUP variance 1.5 OUP variance decay rate <![CDATA[1e -0 5 ]]>

[0079] Table 2. Parameter and Initial Value Settings

[0080] <![CDATA[P amb =10.11]]> <![CDATA[P cin =11.43]]> <![CDATA[P ho =45.107]]> <![CDATA[P co =10.11]]> <![CDATA[W ram =0.0198]]> <![CDATA[W bleed =0.0183]]> <![CDATA[D ram =1.5]]>

[0081] The results show that the fault-tolerant control of the engine bleed air temperature control system based on reinforcement learning proposed in this paper can achieve the target output temperature and immediately recover to the desired trajectory under fault conditions. Compared with traditional PID control, the method proposed in this invention can not only achieve the target output temperature more quickly, but also solve the problem of recovery to stability under different fault conditions. The simulation results verified the effectiveness and feasibility of the fault-tolerant performance of reinforcement learning, which meets expectations.

[0082] Finally, it should be noted that this invention does not explain in detail the common knowledge recognized by those skilled in the art. The above description is only a specific embodiment of this invention and is not intended to limit this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A fault-tolerant control method for bleed air temperature of aero-engines based on reinforcement learning, comprising the following steps: Step 1: Establish a mathematical model for the bleed air temperature control system of an aero-engine; Step 2: Set the relevant parameters for reinforcement learning and train the DDPG algorithm; Step 3: Based on Step 2, combine the trained results with a first-order low-pass filter; Step 4: Introduce multiplicative and additive faults to verify the fault-tolerant performance of reinforcement learning; The aforementioned reinforcement learning-based fault-tolerant control method for bleed air temperature in aero-engines is characterized by the following mathematical model of the bleed air temperature control system in step 1: Among them, the hot-side fluid outlet air has specific heat. The mass flow rate is The inlet temperature is The outlet temperature is The cold-side fluid intake air has a specific heat. The mass flow rate is The inlet temperature and outlet temperature are respectively and ; and These represent the overall heat transfer coefficients on the hot and cold sides, respectively; Mass flow rate through the valve It is calculated using the equations for an isentropic expansion process and is applicable to variable area conduction: in, and These are the upstream and downstream pressures (psi), respectively. It's the temperature upstream. =32.174 It is gravitational acceleration. =1717 It is the gas constant of air. It is the ratio of specific heat at constant pressure to specific heat at constant volume; It is the opening area of ​​the valve, calculated using the following formula: in, It is the diameter of the valve. It refers to the valve's opening angle. It is the valve opening coefficient: in , Actuator failure is modeled as an equivalent deviation: in, This represents the control input containing an actuator fault at time k. Indicates a multiplicative fault. This indicates an additive fault.

2. According to the reinforcement learning-based fault-tolerant control method for bleed air temperature of aero-engines as described in claim 1, the next step is to set the relevant parameters for reinforcement learning and train the DDPG algorithm; Reinforcement learning is a field of machine learning in which an agent interacts with an environment and learns a policy π that maximizes cumulative reward; the environment is modeled as a Markov decision process (MDP) consisting of state S, action A, and transition probabilities. reward function And the definition of the discount factor γ; The core objective is to find an optimal strategy. This strategy maximizes the expected cumulative reward, expressed as: in, The state-value function represents the state value function under policy π, and the function represents the state value function. Begin with and then follow the expected return of π; This represents the current state at time t. Indicates the state The actions that can be selected are... Indicates the state Select Action strategy, Indicates the state China takes action The instant reward obtained afterward This represents the discount factor, indicating the importance of future rewards. Indicates the state China takes action Then transition to the next state The probability, This indicates the next state under policy π. Initial expected cumulative reward; This statement provides the state-value function at different time steps t. The evolution is represented more clearly, taking into account the current state, action selection, immediate reward, and the value of the next state: in, Represents the action value function. Indicates the state The next action selected in [the context] Indicates the state Select Action strategy, This indicates the next state under policy π. and actions The initial expected cumulative reward; this reflects the recursive definition of action values ​​in reinforcement learning, where the value of the current state and action depends on the immediate reward and the value after transitioning to the next state; DDPG (Deep Deterministic Policy Gradient) is a method for simultaneously learning deterministic policies. and Q-value function The actor-critic algorithm; it is designed for environments with continuous action spaces and utilizes a depth function approximator that can handle large state spaces; Actor Strategy Model: Deterministic strategy The decision-making process outputs actions without exploring noise; it updates by ascending the gradient of expected return. : in, It is the objective function Parameters relative to the actor network The gradient; It is the expected operation; This refers to the strategy distribution from the actor network. Mid-sampling status ; It is the commentator network action value function Relative to action The gradient of, where Parameters representing the network of critics; Actions selected using an actor network within a gradient. Replace action a; the purpose of this equation is to update the parameters of the actor network based on the expectation of the product of the actor's policy gradient and the commentator's action value gradient, aiming to maximize the cumulative reward; during training, this gradient guides the actor network to update its policy in a direction more favorable to achieving higher expected rewards; Critic Value Model: Critics use weighted Neural networks are used to approximate the action value function. It is trained by minimizing a loss function, which is the mean squared Bellman error. in It is the commentator network's estimated action-value function based on the current state and action; y is the target value: Target network: DDPG utilizes target networks as well as To stabilize training; the target network is updated in the direction of the parameters of the learning network through soft updates: in, It is the soft update rate; the equation represents the parameters of the target commentator network. / Gradually through parameters with the current critic network / Update using a hybrid approach; These equations together perform soft updates, gradually bringing the target network parameters closer to the current network parameters in each update; soft update rate Controlling the rate of change of the target network parameters relative to the current network parameters is done to slow down the changes in the target network and ensure a relatively stable target. Exploration mechanism: The exploration involves adding noise to the actors' strategies. To be introduced: in It involves sampling from a noisy process. It is the actor network in the state The actions produced below The actor's network is in a state. The policy output provided by the actor network provides the necessary exploration for policy learning, enabling it to learn from a diverse set of actions; therefore, the entire equation represents that the actions generated by the actor network are a combination of deterministic policy output and random noise; this noise is introduced to encourage the exploration of new actions and enhance the algorithm's exploratory behavior.

3. The reinforcement learning-based fault-tolerant control method for aero-engine bleed air temperature according to claim 1 combines the trained results with a first-order low-pass filter. The first-order low-pass filter used is described below: The first-order low-pass filtering algorithm, also known as the exponentially weighted moving average (EWMA), is a simple and effective method for smoothing time series data. It reduces high-frequency noise while preserving low-frequency components. The core equation of this algorithm is as follows: in It is time The current output, It is time The current input, It is time Previous output, It is a smoothing factor that determines the weights assigned to the current input. ; Effectively training an agent to follow the trajectory of control signals requires careful consideration of various design factors when building the environment; these factors can be divided into agent-related and environment-related considerations. Agent-related factors involve the composition of the observation vector and the strategy for defining rewards; on the other hand, environment-related factors include the training strategy, the signals used for training, the initial conditions of the environment, and the criteria for terminating a round; these factors need to be carefully handled to optimize the training process and enhance the agent's ability to navigate and control the trajectory of the signal. The structure of the observation vector is as follows: s= ,in Represents the actual traffic volume. It is the error relative to the reference value, and the integral of the error over time; the integral of the error captures the evolution of the error accumulated over time, provides a means of calculating the total error accumulated over time, and guides the agent in the direction of minimizing it; this integral of the error is often used as a key observation input in the training of reinforcement learning (RL) controllers. The expression below shows a continuously varying reward. As a function of error; It is a small constant used to avoid division by zero errors: The training mechanism of Deep Deterministic Policy Gradient (DDPG) involves iterative optimization of the actor and critic networks. It begins with the initialization of network parameters and a replay buffer used to store experience. The actor introduces exploratory noise when choosing actions, and the experience generated by the interaction with the environment is stored in the buffer. A small batch of experience is then sampled from the experience to update the critic and actor networks, using the critic's mean squared Bellman error. The target network is then soft-updated. This process is repeated iteratively to gradually optimize the network, learning an effective policy in the continuous action space through deterministic policy optimization and Q-value estimation.

Images