Aero-engine transition state control method based on hierarchical security reinforcement learning

Abstract

The application discloses an aero-engine transition state control method based on layered security reinforcement learning, relates to the field of aero-engine control, and comprises the following steps: collecting aero-engine transition state time sequence data, establishing a linear variable parameter model of the aero-engine, and taking the linear variable parameter model as a prediction model; designing an inner controller, wherein the inner controller is composed of a PI controller and a predictive safety filter; designing an outer controller, wherein the outer controller is a reinforcement learning controller based on a deep deterministic policy gradient algorithm; building the inner controller and the outer controller to form a closed-loop control architecture, deploying the control architecture under different transition state acceleration tasks, and outputting control variables in real time through the prediction model. The application solves the problems of traditional control methods, such as complexity in high-dimensional nonlinear systems, heavy calculation burden, and difficulty in completely releasing engine performance, and solves the problem of standard reinforcement learning control methods in ensuring engine safety.

Description

Technical Field

[0001] This invention relates to the field of aero-engine control, and more specifically to a transient control method for aero-engines based on hierarchical safety reinforcement learning. Background Technology

[0002] The transient control scheme for aero-engines has undergone several stages of development along with the continuous advancement of aero-engine technology, from the initial experience-based control strategies to modern, efficient intelligent control methods. With the progress of aero-engine technology, control requirements have become increasingly complex. In high-dimensional nonlinear systems, the limitations of traditional control methods have become increasingly apparent, prompting the industry to propose and apply new control methods.

[0003] Currently, the design methods for transient control programs in aero-engines can be broadly categorized as follows:

[0004] The first type is the approximate determination method, which approximates the transient operating condition using the equilibrium equations when the aero-engine is in a stable operating state. Its core is to ignore the dynamic changes during the transient process and simplify the dynamic problem into a static problem. However, due to the dynamic characteristics of the transient state, such as rotor inertia and thermal inertia, the equilibrium equations in the stable state cannot accurately reflect the real situation of the transient state, resulting in low design accuracy. In addition, in practical applications, a lot of corrections to the stable state equations are required to approximate the transient state, which is a complex process and therefore has not been widely used.

[0005] The second category is model-based methods, which mainly include the following two types:

[0006] ① The power extraction method and its derivatives are based on the core idea of ​​"transforming dynamic problems into steady-state problems." The power extraction method was first proposed by Professor Chen Yuchun. Based on the engine steady-state characteristic calculation model, it adds rotor power extraction to bring the engine's steady-state operating point (line) closer to the surge boundary. Considering other constraints, it uses an appropriate control law description form to achieve the design of the optimal transitional control law. This method has the advantages of being fast and effective, but its essence is a "bang-bang" strategy that seeks to maximize rotor acceleration within parameter constraints. To maintain a unique solution, it only supports single fuel flow scheduling and is difficult to adapt to complex scenarios of multi-variable control. Later, Jia Linyuan combined the power extraction method with the inverse algorithm. A new method for designing transient control laws for variable geometry engines—the inverse transient algorithm—was proposed. However, since it does not consider volume effects (such as the influence of combustion chamber and duct volume on airflow) and multi-rotor coupling (interaction between different rotors), its accuracy can still be further improved. The Army, drawing on the basic idea of ​​the power extraction method, proposed a dynamic stabilization method for the design of optimal transient control laws for aero-engines. This method extracts the rate of change of all state variables from the engine dynamic characteristic calculation model to make the deviation of the common working equation set tend to 0, thereby stabilizing the transient simulation. However, this method cannot accurately give suitable initial values ​​for state variables and requires repeated iterations, which increases the computational cost and time.

[0007] ② Mathematical optimization methods, the core idea of ​​which is to call a general trajectory optimization solver on the engine characteristic model and find the optimal transient control law through optimization algorithms. For example, the constraint variable metric method is suitable for systems with relatively simple structures. For complex nonlinear systems such as aero engines, more mathematical analysis is required to establish a suitable optimization model. The effect of the penalty function method is very sensitive to the setting of the penalty function coefficient. When the penalty function coefficient is too large, it may lead to instability in the optimization process. When it is too small, the convergence speed is slow and it may converge to a solution that does not meet the constraints. Quadratic sequence programming (QP) does not perform well for non-convex problems. The improved quadratic sequence programming (SQP) algorithm is more complex and sensitive to the initial guess value, and may get trapped in local optima. Therefore, although these methods can theoretically achieve optimal control, they each have their limitations.

[0008] The third category is data-driven methods, such as reinforcement learning control methods. In recent years, with the development of artificial intelligence, reinforcement learning methods have been extensively studied in various fields, including the transient state control of aero-engines. Its core idea is to find the optimal policy through the interaction between the agent and the environment, and through policy iteration or value iteration using reward signals. For complex nonlinear systems like aero-engines, reinforcement learning methods do not require linearization or the establishment of precise component-level models, and the resulting acceleration time is shorter than that of traditional PID control. However, the training results may not converge, and the training process may produce solutions outside the safe range, causing engine damage and posing certain risks to bench tests. Therefore, this method is currently mostly studied through simulation.

[0009] In summary, existing technologies still present many challenges in the transient control of aero-engines. There is an urgent need for a new control method that can provide performance optimization control for this high-dimensional nonlinear model of aero-engines under varying operating conditions and complex constraints, and ensure that it operates within a safe envelope. Summary of the Invention

[0010] To address the aforementioned shortcomings in existing technologies, this invention provides a transient control method for aero-engines based on hierarchical safety reinforcement learning. It uses the engine's linear variable parameter model as the prediction model, employs a standard reinforcement learning controller as the outer controller, and combines a proportional-integral (PI) controller with a predictive safety filter capable of online control quantity correction as the inner controller. This approach aims to leverage the advantages of data-driven methods to achieve rapid thrust response of the engine under transient safety hard constraints. It addresses the shortcomings of traditional control methods in high-dimensional nonlinear systems, such as complexity, high computational burden, and difficulty in fully releasing engine performance, as well as the deficiencies of standard reinforcement learning control methods in ensuring engine safety.

[0011] To achieve the aforementioned objectives, the technical solution adopted by this invention is: a transient control method for aero-engines based on hierarchical safety reinforcement learning, comprising the following steps:

[0012] S1: Collect transient time series data of aero-engine, establish a linear variable parameter model of aero-engine, and use the linear variable parameter model as a prediction model;

[0013] S2: Design an inner layer controller, which consists of a PI controller and a predictive safety filter;

[0014] S3: Design an outer controller, which is a reinforcement learning controller based on a deep deterministic policy gradient algorithm;

[0015] S4: The inner controller and the outer controller are built into a closed-loop control architecture. The control architecture is deployed under different transitional acceleration tasks. The control variables are output in real time through the prediction model to realize the transitional control of the aero-engine based on hierarchical safety reinforcement learning.

[0016] Furthermore, in step S1, the transient time-series data of the aero-engine are collected to establish a linear variable parameter model of the aero-engine, including the following steps:

[0017] S11: Collect transient timing data of the aero-engine, the timing data including state vector, input vector and output vector;

[0018] S12: The Jacobi method is used to linearize the nonlinear system of the aero-engine, resulting in multiple linear time-invariant systems;

[0019] S13: By interpolating and fitting the multiple linear time-invariant systems, the discrete state-space equation of the linear variable parameter system is established.

[0020] Furthermore, the discrete state-space equation of the linear variable parameter system in S13 is:

[0021]

[0022]

[0023] in, For system state variables, For system output variables, Sampling time, , , and The coefficient matrix, It is a vector consisting of scheduling parameters. These are the system state variables at the steady-state equilibrium point. This is the system output at the steady-state equilibrium point. For system control input, This is the system input at the steady-state equilibrium point.

[0024] Furthermore, the output expression of the PI controller in S2 is:

[0025]

[0026] in, and For gain, This represents the error at the current sampling time. It is a time constant. For time step.

[0027] Furthermore, the predictive security filter in S2 is:

[0028]

[0029]

[0030] in, Let cost function be The control commands are corrected by the predictive safety filter. These are the raw control commands from the inner controller. To predict the time domain, and These are the lower and upper bounds of the safety limit for the control quantity, respectively. and These are the lower and upper bounds of the safety margin for the rate of change of the control quantity, respectively. and These are the lower and upper bounds of the model output, respectively. For the first Initial values ​​of engine state variables for step 1. For the first The engine condition measurement values ​​of the step.

[0031] Furthermore, in the reinforcement learning controller of S3, the observation space is:

[0032]

[0033]

[0034] in, for The space of observation at any moment This refers to the low-pressure rotor speed of the engine. For low-pressure rotor speed error, It is the matrix transpose operator. The target low-pressure rotor speed;

[0035] The motion space is:

[0036]

[0037] in, for The space of action at any given moment.

[0038] Furthermore, in the reinforcement learning controller in S3, the Actor network adopts a single-path feedforward neural network structure design, and finally maps the output to the actual action range through a scaling layer; the Critic network adopts a dual-path input design containing observation path and action path, and after the two features are added, fused and activated, the Q-value is estimated through a fully connected layer.

[0039] Furthermore, the reward function of the reinforcement learning controller in S3 is:

[0040]

[0041]

[0042] in, For the reward function, and These are the weighting coefficients. The penalty is imposed for the original control input exceeding the safety boundary. For the first The value of each safety protection parameter For the first The upper limit of the safety protection parameter, For the first The lower bound of each safety protection parameter This is the correction amount for the control input by the predictive safety filter.

[0043] The beneficial effects of this invention are as follows: This invention proposes a three-layer collaborative control architecture integrating reinforcement learning (RL), proportional-integral (PI) control, and predictive safety filters for intelligent safety control of the transient process in aero-engines. Compared with standard reinforcement learning methods, this method significantly improves engineering feasibility and safety while preserving the system's adaptability, and has the following innovative advantages:

[0044] (1) Adaptive parameter tuning: Utilizing RL to optimize the proportional gain of the PI controller online With integral gain This enables the control law to be dynamically adjusted according to the current operating conditions of the engine, effectively overcoming the problems of slow response or excessive overshoot in traditional fixed-parameter PI control during large-scale nonlinear transitions.

[0045] (2) Constraint safety assurance: A predictive safety filter is introduced to impose physical constraints on the output of the PI controller (such as the limits of key parameters such as turbine inlet temperature and high-pressure rotor speed) to ensure that all control commands are always within the safe operating envelope of the engine, thus solving the bottleneck of the lack of safety assurance of pure learning controllers.

[0046] (3) Balancing computational efficiency and robustness: The predictive safety filter is only used to make lightweight corrections to the PI output, rather than full-state optimization, which greatly reduces the real-time computational burden; at the same time, RL training can be completed offline in a high-fidelity simulation environment, avoiding the risks of real test driving, and maintaining good robustness under model mismatch or sensor noise.

[0047] Compared with traditional fixed-gain PI, pure model predictive control (MPC) or unconstrained RL control methods, this scheme has significant improvements in key indicators such as acceleration response time, overshoot suppression, constraint satisfaction rate and multi-condition adaptability, providing a technical path that combines academic cutting-edge technology with engineering practicality for the next generation of intelligent, safe and certifiable aero-engine control systems. Attached Figure Description

[0048] Figure 1 This is a flowchart of a transient control method for aero-engines based on hierarchical safety reinforcement learning.

[0049] Figure 2 A control architecture diagram for a hierarchical security reinforcement learning control method.

[0050] Figure 3 This is a diagram of the Actor network and Critic network architecture.

[0051] Figure 4 This is a schematic diagram of the selected turbofan engine.

[0052] Figure 5 A comparison chart showing the simulation results of the LPV model and the component-level model.

[0053] Figure 6 This diagram illustrates the changes in key engine parameters during a typical step command training process when using a traditional soft-constraint architecture.

[0054] Figure 7 This diagram illustrates the changes in key engine parameters during a typical step command training process when using the hard constraint architecture of this invention.

[0055] Figure 8 This is a schematic diagram illustrating the dynamic performance of a hierarchical safety reinforcement learning controller under wide-condition APRBS excitation. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0057] like Figure 1 and Figure 2 As shown, a transient control method for aero-engines based on hierarchical safety reinforcement learning includes the following steps:

[0058] S1: Collect transient time series data of aero-engine, establish a linear variable parameter model of aero-engine, and use the linear variable parameter model as a prediction model;

[0059] The Linear Variable Parameter (LPV) model describes the dynamic behavior of a system by introducing time- or system state-dependent scheduling variables. It is suitable for systems with time-varying or parameter-varying characteristics and can accurately reflect the changing characteristics of the system at different operating points. Through scheduling variables, the LPV model can dynamically adjust the controller parameters under different operating conditions, making it more flexible and adaptable.

[0060] A nonlinear state-space model of the engine is constructed based on the LPV model. This model is deployed through the following process, providing core computational support for the safety reinforcement learning algorithm.

[0061] The step S1 involves collecting transient time-series data of the aero-engine and establishing a linear variable parameter model of the aero-engine, including the following steps:

[0062] S11: Collect transient timing data of the aero-engine, the timing data including state vector, input vector and output vector;

[0063] Collect transient time-series data of aero-engines; key variables include: state vector. , representing the low-pressure rotor speed and the high-pressure rotor speed; input vector ,in Main fuel flow rate; output vector The parameters are the engine low-pressure rotor speed, high-pressure rotor speed, turbine inlet temperature, thrust, low-pressure rotor surge margin, high-pressure rotor surge margin, and air-fuel ratio.

[0064] S12: The Jacobi method is used to linearize the nonlinear system of the aero-engine, resulting in multiple linear time-invariant systems;

[0065] The Jacobi method is used for linearizing nonlinear systems. Its basic theory is a first-order Taylor expansion, which does not impose special formal constraints on nonlinear systems. In a nonlinear system, Jacobi linearization can be performed at different operating state points throughout the entire operating region, resulting in a set of linear time-invariant (LTI) systems. These LTI systems can describe the operating points and the operating states within a small surrounding area. LPV systems can be obtained by interpolating the linearized clusters at these equilibrium points.

[0066] A general nonlinear system can be described as follows:

[0067]

[0068]

[0069] in, For system state variables, For system output variables, For system control input, , It is a nonlinear operator. This represents the system scheduling quantity, and the superscript "." indicates that the vector is differentiated.

[0070] Then, at a certain steady-state operating point of the nonlinear system, the following equation holds:

[0071]

[0072]

[0073] in, , , , These represent the input, output, state variables, and scheduling variables at the steady-state equilibrium point. They are the changes in the state variables of a nonlinear system in steady state. Expanding the first-order Taylor equation at this steady-state point yields:

[0074]

[0075]

[0076] In the formula, , , , The subscript "ss" describes the parameters of the nonlinear system at this steady-state equilibrium point. Here, we can let:

[0077] , , ,

[0078] The linearized model of the nonlinear system at the aforementioned steady-state equilibrium point can be obtained as follows:

[0079]

[0080]

[0081] The system represented by the above equation can describe the steady-state and dynamic characteristics in the region near this steady-state point. Wherein A , B , C , DThe coefficients of a constant matrix are readily obtained by taking its partial derivatives for simple nonlinear systems. For complex nonlinear systems such as aerospace propulsion systems, the coefficients of the aforementioned matrix can usually be obtained using the power extraction method, fitting method, and sequential method.

[0082] S13: By interpolating and fitting the multiple linear time-invariant systems, the discrete state-space equation of the linear variable parameter system is established.

[0083] Thus, the linearized model at a single steady-state point has been obtained, representing an LTI system near that steady-state point. Following this approach, by selecting several steady-state points within a certain operating region of the nonlinear system and performing Jacobi linearization, a cluster of LTI systems near these steady-state points can be obtained. By performing interpolation, fitting, or other operations on these results, the continuous form of the LPV model expression for the nonlinear system can be obtained, as shown in the following equation:

[0084]

[0085]

[0086] The discrete state-space equation of the linear variable parameter system is:

[0087]

[0088]

[0089] in, For system state variables, For system output variables, Sampling time, , , and The coefficient matrix, It is a vector consisting of scheduling parameters. These are the system state variables at the steady-state equilibrium point. This is the system output at the steady-state equilibrium point. For system control input, This is the system input at the steady-state equilibrium point.

[0090] S2: Design an inner controller, which consists of a proportional-integral (PI) controller and a predictive safety filter;

[0091] The PI controller tracks commands by adjusting fuel flow, but it can easily produce overly aggressive control values ​​when the command deviation is large. To address this issue, a predictive safety filter is incorporated into the PI controller output to safely identify and correct potentially aggressive control values ​​in real time. Simultaneously, the correction is fed back as a penalty to the agent, prompting it to learn the true safety boundaries. The PI controller is responsible for achieving fast and accurate speed tracking, while the predictive safety filter performs online safety assessments and real-time corrections on its output commands. Together, they ensure that the system meets stringent transient safety constraints and performance requirements at the lowest level.

[0092] As a classic linear feedback control strategy, the PI controller synthesizes the control quantity by performing proportional and integral operations on the error signal. Theoretically, it can achieve zero steady-state error tracking while ensuring the system response speed.

[0093] The simulation environment is in the discrete-time domain, and the output expression of the PI controller is:

[0094]

[0095] in, and To increase gain, through proper tuning and Gain, while ensuring system stability, can effectively suppress overshoot, shorten settling time, and achieve the control objective of zero steady-state error. This represents the error at the current sampling time. It is a time constant. For time step.

[0096] In the safety reinforcement learning control framework proposed in this invention, the PI controller serves as the underlying tracking controller, and its proportional and integral gains are adaptively tuned online by the standard reinforcement learning decision module, thereby effectively improving the system's generalization ability and achieving optimal tracking performance under various operating conditions.

[0097] Predictive safety filter: The predictive safety filter is the "safety" core of this layered safety reinforcement learning architecture. By constructing explicit system safety boundaries, it corrects the control instructions output by the inner layer reinforcement learning-based PI controller in real time to the optimal feasible instructions that satisfy all predefined safety boundaries, thereby ensuring that the closed-loop system always meets strict safety hard constraints.

[0098] Predictive safety filter at each control sampling time Received raw control commands from the inner controller Based on the current system state This problem solves a finite-time open-loop optimization problem. The objective is to minimize the modification of the original control command so that, when the modified control command is applied to the system, the system satisfies various hard safety constraints, which can be categorized into four types:

[0099] ① Control quantity constraints: that is, the control quantity of the engine must be within a safe range, and the change range cannot exceed the allowable value of the actuator;

[0100] ② Input constraints: The state parameters of the engine must be within a safe range, and their values ​​are usually obtained by sensor measurements or observer estimation;

[0101] ③ Output constraints: The engine's output parameters must be within a safe range;

[0102] ④ State constraints: The dynamics of the engine system must follow the state-space equations described by the LPV prediction model.

[0103] The complete mathematical description of the optimization problem is as follows. By solving this optimization problem, the predictive safety filter can realize the safety judgment of the original control command and the real-time minimum correction of the unsafe control command, so that the engine always operates within the safety boundary.

[0104] The predictive security filter is:

[0105]

[0106]

[0107] in, Let cost function be The control commands are corrected by the predictive safety filter. These are the raw control commands from the inner controller. To predict the time domain, and These are the lower and upper bounds of the safety limit for the control quantity, respectively. and These are the lower and upper bounds of the control variable change rate (the change in the control variable within one sampling period), respectively. and These are the lower and upper bounds of the model output, respectively. For the first Initial values ​​of engine state variables for step 1. For the first The engine state measurement value of the next step, that is, the initial state of the engine in the next step is the measurement value of the previous step.

[0108] S3: Design an outer controller, which is a reinforcement learning controller based on a deep deterministic policy gradient algorithm;

[0109] The outer controller is a reinforcement learning controller, whose purpose is to adaptively tune the underlying PI controller in the inner controller. When the instruction deviation is large, it adaptively changes the control gain to speed up the tracking speed.

[0110] Studies have shown that the Deep Deterministic Policy Gradient (DDPG) algorithm can meet the requirements of fast response and steady-state accuracy in engine transient and steady-state control. Considering its fundamental nature, ease of use, and broad applicability, the outer controller of the architecture adopts a reinforcement learning controller based on the DDPG algorithm. This controller can provide a stable, reproducible, and optimizable reinforcement learning controller benchmark while ensuring control performance.

[0111] The deep reinforcement learning controller based on the DDPG algorithm is designed from three aspects: ① observation space and action space; ② actor network and critic network; ③ reward function.

[0112] In transient control tasks, the core objective of the controller is to achieve rapid and accurate tracking of the target rotational speed. Combining the characteristic of the integral element in classical control theory that it can eliminate steady-state errors, the observation space is ultimately defined as follows, constituting a minimal set of states that is both informative and compact:

[0113]

[0114]

[0115] in, for The space of observation at any moment This refers to the low-pressure rotor speed of the engine. For low-pressure rotor speed error, It is the matrix transpose operator. The target low-pressure rotor speed;

[0116] The purpose of reinforcement learning controllers is to achieve adaptive parameter tuning of PI controllers. Therefore, the action space is defined as:

[0117]

[0118] in, for The space of action at any given moment.

[0119] In the reinforcement learning controller, the Actor network adopts a single-path feedforward neural network structure, and the output is finally mapped to the actual action range through a scaling layer to adapt to the environmental action space requirements. The Critic network adopts a dual-path input design with both observation and action paths. The features from the two paths are added, fused, and activated, and then output a Q-value estimate through a fully connected layer. The specific structures of the two networks are attached. Figure 3 As shown.

[0120] The reward function, as a key feedback signal in the interaction between the agent and the environment, directly influences the direction and effectiveness of policy optimization. To guide the agent in learning a control strategy that balances dynamic performance and safety, this invention designs a two-part reward function: the first part evaluates the speed and steady-state accuracy of engine speed tracking, while the second part corresponds to the safety constraints of key engine parameters. By balancing the tracking target and safety boundary constraints, this function incentivizes the agent to output control commands that are both responsive and meet safety requirements.

[0121] The reward function of the reinforcement learning controller is:

[0122]

[0123]

[0124] in, For the reward function, and These are the weighting coefficients. The penalty is imposed for the original control input exceeding the safety boundary. For the first The value of each safety protection parameter For the first The upper limit of the safety protection parameter, For the first The lower bound of each safety protection parameter This is the correction amount for the control input by the predictive safety filter.

[0125] Under soft constraints, the core function is to achieve safe convergence by penalizing exceeding limits to guide the final policy to meet safety requirements. Under hard constraints, the core function is to optimize the original instructions. Under the premise that the predictive safety filter ensures absolute safety, it incentivizes the agent to actively output original instructions that conform to safety constraints, thereby minimizing the intervention of the filter's correction.

[0126] In this invention, a DDPG reinforcement learning controller is used to adaptively adjust the core parameters of the PI controller under different engine operating conditions, thereby obtaining the control quantity. Based on the design of the reinforcement learning controller described in the previous three parts, the mathematical principle of the algorithm is as follows:

[0127] The joint probability formula for state transition and reward is:

[0128]

[0129] in, Indicates time In state And perform the action At any moment Transition to state And receive a reward The probability of; For a moment The action output by the agent (the PI parameter adjustment amount to be optimized here). For a moment Reward values ​​based on feedback of control effectiveness.

[0130] Cumulative rewards measure the long-term control effect, and the formula is:

[0131]

[0132] in, From time The accumulated rewards, The discount rate (the closer it is to 1, the more emphasis is placed on long-term rewards). This is the time step offset. For a moment Instant rewards.

[0133] The action value function evaluates the long-term value of a state-action combination, and the formula is:

[0134]

[0135] in, For strategy Next state Execute action value, For strategy The expected value of the following mathematical expression For strategy Next state Select Action The probability, For the next state Optional actions below.

[0136] The formula for finding the optimal strategy is:

[0137]

[0138] in, The optimal strategy is... This means finding a strategy that maximizes the value of an action; the agent iteratively optimizes the strategy, gradually approaching the optimal strategy, and adjusts the action by outputting the optimal PI parameters.

[0139] Through reinforcement learning and iterative optimization, the final output is the PI parameter that optimizes control performance, as shown in the formula:

[0140]

[0141] in, For the optimal ratio and integral coefficient, This indicates the search for the average cumulative reward. Maximize the combination of parameters. The average cumulative reward under different PI parameters (obtained by averaging the cumulative rewards from multiple iterations).

[0142] S4: The inner controller and the outer controller are built into a closed-loop control architecture. The control architecture is deployed under different transitional acceleration tasks. The control variables are output in real time through the prediction model to realize the transitional control of the aero-engine based on hierarchical safety reinforcement learning.

[0143] By incorporating the aforementioned predictive security filter into a PI controller as the inner controller, and using the DDPG reinforcement learning controller as the outer controller, a system is constructed as follows: Figure 2 The architecture shown is a safety reinforcement learning controller; a safety reinforcement learning-based controller is deployed under different transitional acceleration tasks, with the engine state vector as input. and reference trajectory Real-time output of control variables That is, main fuel flow The performance of the controller is verified through simulation to ensure that the rotor speed can quickly track the reference value, and that the engine speed, control input and all other key parameters (such as surge margin, turbine inlet temperature, etc.) are always within a safe range.

[0144] In one embodiment of the present invention, the technical solution of the present invention will be described in detail with reference to specific examples.

[0145] Step S1: Establish a linear variable parameter (LPV) model for the aero-engine:

[0146] The engine model selected in this case study is a turbofan engine with three control variables: fuel flow rate and fuel output. Tail nozzle throat area Post-duct ejector area Its overall structure diagram is attached. Figure 4 As shown.

[0147] from Figure 4It can be seen that along the axial direction, starting from the airflow inlet, the main components are: intake duct, fan, low-pressure compressor, high-pressure compressor, bypass nozzle, combustion chamber, high-pressure turbine, low-pressure turbine, mixing chamber, and tail nozzle. The definitions of each section are shown in Table 1.

[0148] Table 1 Definition of Engine Section

[0149]

[0150] The parameters in the LPV model are determined based on the scheduling parameters that characterize the operating state (such as speed and mode), therefore, the scheduling amount should be selected first. Since the low-pressure rotor speed... It is an important parameter characterizing engine operation and implementing engine control, and it is measurable. Therefore, in this case, the low-pressure rotor speed is selected as the continuous scheduling variable, i.e. To further highlight the security and practical advantages of the safety reinforcement learning control architecture in this invention, this case study uses the simplest and most common single-variable fuel control. The LPV model variables are defined in the following table:

[0151] Table 2 Definitions of LPV Model Variables

[0152]

[0153] A closed-loop simulation verification method was adopted, and simulations were conducted based on the LPV model and the engine component-level (CLM) model constructed in this invention. The response characteristics of the two models during typical transient processes were compared under design point conditions to evaluate the accuracy of the LPV model established in this invention. Simulation results show that the identified LPV model has an error of no more than 6% in the low-pressure compressor surge margin, no more than 3% in the air-fuel ratio, and no more than 1% in the errors of other parameters during typical transient processes; under steady-state conditions, the errors of all parameters are 0. The identified LPV model lays a solid model foundation for the online accurate protection of predictive safety filters. Simulation results are attached. Figure 5 As shown.

[0154] Step S2, design a PI controller based on reinforcement learning adaptive parameter tuning:

[0155] Since the DDPG algorithm can be applied to complex systems with continuous state and action spaces, and is less sensitive to hyperparameter selection compared to algorithms such as TD3, it is simpler and easier to implement. Therefore, the reinforcement learning DDPG algorithm is chosen to implement adaptive hyperparameter tuning of the PI controller.

[0156] The observations, outputs, and reward function of the DDPG algorithm in this case study have been explained previously. Its Actor and Critic network architectures are detailed in the appendix. Figure 3 As indicated in the text, the output range of the Actor network is [0,5], which means... and The upper and lower limits of the parameter are [0, 5].

[0157] The main parameters of the DDPG algorithm in this case are shown in Table 3 below:

[0158] Table 3 Reinforcement Learning Training and Algorithm Hyperparameter Settings

[0159]

[0160] The PI controller accepts input from reinforcement learning. , and current speed error This enables adaptive adjustment of parameters, thereby achieving adaptive output of fuel control quantity.

[0161] Step S3, Design a predictive security filter:

[0162] The fuel output of the aforementioned PI controller is generated through reinforcement learning exploration. The exploration strategy in reinforcement learning is guided by maximizing the reward function value. During its trial and error process, it may generate control commands that exceed the fuel safety threshold. Due to the lack of explicit constraints on the engine's physical limits, this unguided exploration can easily lead to instantaneous engine operating conditions exceeding limits, resulting in irreversible mechanical damage or reliability degradation. Therefore, when applying reinforcement learning to actual engine control, a safety layer or constraint mechanism must be introduced to strictly limit the exploration within safety boundaries, achieving a balance between performance optimization and operational safety. Considering the explicit constraint processing characteristics of MPC, this case adopts a similar approach to MPC, designing a "predictive safety filter." If the fuel flow output from the PI controller keeps the engine operating in a safe state, no correction is made; if the output fuel flow causes the engine operating state to exceed the safety boundary, the predictive safety filter will make minimal corrections, turning the corrected fuel flow into a safe control command.

[0163] Predictive safety filters use the LPV model as the predictive model, calculating the optimal control input by solving an optimization problem at each control step. To avoid verbosity and ambiguity in the formula derivation, variables related to scheduling parameters will be omitted below. The theoretical derivation and design of predictive safety filters are similar to those of MPC.

[0164] (1) State prediction

[0165] Predictable arrive The state sequence is shown in the following equation:

[0166]

[0167] In the formula The meaning is Always Prediction of time; where the part after "|" Represented as the current time Therefore, it is possible to... arrive The state is predicted as follows:

[0168]

[0169] Simplifying the equation using matrix form, the predicted state equation can be expressed as follows:

[0170]

[0171] In the formula, , which are the predicted state variables; This is the input quantity at the steady-state point at the current moment; These are the state variables at the current steady-state point; the coefficient matrices are as follows:

[0172]

[0173] (2) Output prediction

[0174] Similarly, the discrete LPV state-space model can predict... arrive The output sequence is shown in the following formula:

[0175]

[0176] In the formula The meaning is Always Prediction of time; where the part after "|" Represented as the current time Therefore, it is possible to... arrive The output is used for prediction. As follows:

[0177]

[0178] Simplifying the equation using matrix form, the prediction output equation can be expressed as follows:

[0179]

[0180] In the formula, , which are the predicted state variables; This is the input quantity at the steady-state point at the current moment; These are the state variables at the current steady-state point; the coefficient matrices are as follows:

[0181]

[0182] (3) Feedback correction

[0183] In model predictive control (MMC), feedback correction is a crucial step in ensuring control accuracy and system robustness, and direct state feedback (DSF) is a commonly used method for achieving this. The core idea of ​​DSF is to feed the actual system state back to the controller in real time via measurement devices, replacing the current state of the predictive model as the initial condition for the next optimization. This method effectively eliminates the accumulation of prediction errors, enabling the controller to accurately track the reference target even under conditions of model uncertainty, external disturbances, or unmodeled dynamics.

[0184] The mathematical form of direct state feedback correction is:

[0185]

[0186] In the formula, This serves as the initial condition for the next optimization. This represents the real-time state of the LPV model.

[0187] (4) Specific design

[0188] The goal of this problem is to make minimal modifications to the original control commands so that the system satisfies all hard safety constraints after the modified control commands are applied. The complete mathematical description of the optimization problem is as follows:

[0189]

[0190] In this architecture, the main parts of the predictive safety filter design are the prediction time domain, objective function, and constraints. The specific parameters are shown in Table 4.

[0191] Table 4 Predictive Security Filter Parameters

[0192]

[0193] Step S4, Real-time Control Implementation and Verification:

[0194] Taking the engine accelerating from a standstill on the ground as an example, let's select the initial value. Final value The performance index was set to control the low-pressure speed, and the normalized baseline value was the value corresponding to the design point. To verify the effectiveness of the safety reinforcement learning architecture of the predictive safety filter, the following simulation verification was conducted:

[0195] (1) Typical step instruction

[0196] In the simulation, the initial value of the low-pressure rotor speed was set to 60%, and the reference value was 100%. A comparative experiment was conducted on the two constraint architectures.

[0197] ① Architecture 1 (Traditional Soft Constraint Architecture): The excess of engine parameters is considered in the reward function and treated as a penalty.

[0198] ② Architecture 2 (Hard Constraint Architecture of this Invention): Based on Architecture 1, a predictive safety filter is added, and the correction amount of the control quantity is used as a penalty.

[0199] Simulation results are attached. Figure 6 , 7 As shown, the changes in key engine parameters during training are illustrated in the two scenarios described above. In each sub-figure, the solid blue line represents the parameter change curve for each training episode (where the bold blue solid line represents the last training episode), the dashed black line represents the parameter reference value, the red dotted line represents the parameter safety boundary, and the light yellow background area represents the parameter safety domain.

[0200] The results show that, in terms of safety performance, the hard-constraint architecture proposed in this invention demonstrates a strict guarantee of engine safety performance. Throughout the entire transition process, all key parameters were strictly limited within the preset safety boundaries, with no exceedances. Furthermore, the maximum correction to fuel during the testing phase was less than 20% of the per-unit value. In contrast, the traditional architecture cannot guarantee the transient safety of the engine, exhibiting significant exceedances of the safety boundaries during the training phase and temporary exceedances of the low-pressure surge margin during the testing phase, with maximum exceedance values ​​of 1.34% and 7.24%, respectively.

[0201] (2) APRBS instruction

[0202] Amplitude-modulated pseudo-random binary signal (APRBS) undergoes a random transition every 20 seconds within the 60%–100% low-voltage speed range, with the absolute value of each transition amplitude greater than 0.1. A comparative experiment is conducted on the two constraint architectures.

[0203] ① Architecture 2 (Hard Constraint Architecture of this Invention): Based on Architecture 1, a predictive safety filter is added, and the correction amount of the control quantity is used as a penalty.

[0204] ② Architecture 3 (Ablation Experiment Architecture): Based on Architecture 2 (Complete Architecture of this Invention), the PI controller is removed, allowing the agent to directly output control quantities.

[0205] The simulation test results are attached. Figure 8As shown, under wide-amplitude, highly random APRBS signal excitation, the complete architecture proposed in this invention enables the engine to quickly track the reference command within 5 seconds with no steady-state error, and all constraint variables remain strictly within the safe operating boundaries throughout the entire dynamic process. In contrast, after removing the PI controller, the minimum steady-state error at low-pressure speed in Architecture 3 is 0.02%, and the maximum is 1.84%.

[0206] This embodiment models the engine as an LPV model and establishes a closed-loop architecture for safe reinforcement learning control through the collaborative design of standard reinforcement learning, a PI controller, and a predictive safety filter. This enables safe adaptive control of the aero-engine's transient acceleration process. The controller ensures that the engine always operates within its safety envelope, providing a reliable and innovative technical solution for the engineering application of intelligent control systems for aero-engines.

[0207] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the invention.

Claims

1. A transient control method for aero-engines based on hierarchical safety reinforcement learning, characterized in that, Includes the following steps: S1: Collect transient time series data of aero-engine, establish a linear variable parameter model of aero-engine, and use the linear variable parameter model as a prediction model; The step S1 involves collecting transient time-series data of the aero-engine and establishing a linear variable parameter model of the aero-engine, including the following steps: S11: Collect transient timing data of the aero-engine, the timing data including state vector, input vector and output vector; S12: The Jacobi method is used to linearize the nonlinear system of the aero-engine, resulting in multiple linear time-invariant systems; S13: By interpolating and fitting the multiple linear time-invariant systems, the discrete state-space equation of the linear variable parameter system is established. The discrete state-space equation of the linear variable parameter system in S13 is: in, For system state variables, For system output variables, Sampling time, , , and The coefficient matrix, It is a vector consisting of scheduling parameters. These are the system state variables at the steady-state equilibrium point. This is the system output at the steady-state equilibrium point. For system control input, This refers to the system input at the steady-state equilibrium point; S2: Design an inner layer controller, which consists of a PI controller and a predictive safety filter; The output expression of the PI controller in S2 is: in, and For gain, This represents the error at the current sampling time. It is a time constant. For time step; The predictive security filter in S2 is: in, Let cost function be The control commands are corrected by the predictive safety filter. These are the raw control commands from the inner controller. To predict the time domain, and These are the lower and upper bounds of the safety limit for the control quantity, respectively. and These are the lower and upper bounds of the safety margin for the rate of change of the control quantity, respectively. and These are the lower and upper bounds of the model output, respectively. For the first Initial values ​​of engine state variables for step 1. For the first Step-by-step engine condition measurement values; S3: Design an outer controller, which is a reinforcement learning controller based on a deep deterministic policy gradient algorithm; S4: The inner controller and the outer controller are built into a closed-loop control architecture. The control architecture is deployed under different transitional acceleration tasks. The control variables are output in real time through the prediction model to realize the transitional control of the aero-engine based on hierarchical safety reinforcement learning.

2. The aero-engine transient state control method based on hierarchical safety reinforcement learning according to claim 1, characterized in that, In the reinforcement learning controller described in S3, the observation space is: in, for The space of observation at any moment This refers to the low-pressure rotor speed of the engine. For low-pressure rotor speed error, It is the matrix transpose operator. The target low-pressure rotor speed; The motion space is: in, for The space of action at any given moment.

3. The aero-engine transient control method based on hierarchical safety reinforcement learning according to claim 2, characterized in that, In the reinforcement learning controller in S3, the Actor network adopts a single-path feedforward neural network structure design, and finally maps the output to the actual action range through a scaling layer; the Critic network adopts a dual-path input design containing observation path and action path, and after the two features are added, fused and activated, the Q-value is estimated through a fully connected layer.

4. The aero-engine transient state control method based on hierarchical safety reinforcement learning according to claim 3, characterized in that, The reward function of the reinforcement learning controller in S3 is: in, For the reward function, and These are the weighting coefficients. The penalty is for exceeding the safety boundary in the original control input. For the first The value of each safety protection parameter For the first The upper limit of the safety protection parameter, For the first The lower bound of each safety protection parameter This is the correction amount for the control input by the predictive safety filter.

Images