Aero-engine performance degradation robust control method based on deep reinforcement learning

Abstract

The application discloses a kind of based on deep reinforcement learning's aero-engine performance degradation robust control method, belong to aero-engine intelligent control field, including the following steps: introduce health parameter, quantification engine component level model rotor component performance degradation degree;Actor-critic framework based aero-engine deep network controller is constructed;Propose performance degradation robust performance index;Combining random environment domain method, randomly initialize component level model health parameter;Introduce confrontation intelligent agent, and form zero-sum game with controller;Deep network controller is trained based on deterministic policy deep reinforcement learning algorithm;Carry out simulation experiment verification.

Description

Technical Field

[0001] This invention relates to a robust control method for performance degradation of aero-engines based on deep reinforcement learning, belonging to the field of intelligent control of aero-engines. Background Technology

[0002] Reinforcement learning suffers from low sample efficiency and involves trial and error during training, making the use of real-world objects as the interaction environment in industrial applications extremely costly. In practice, it's typically necessary to build mathematical models of real-world objects, train agents using these models as the interaction objects, and then migrate from the simulation environment to the real-world environment (sim-to-real). However, this mismatch between the real controlled object and the model can affect control performance after migrating from simulation to real-world conditions. In aero-engines, model mismatch primarily stems from engine performance degradation. Aero-engines operate in extreme and harsh environments for extended periods, and with accumulated service time, their gas path components inevitably undergo slow degradation. It's generally assumed that this degradation is concentrated in four key components: the fan, compressor, high-pressure turbine, and low-pressure turbine. After degradation of these components, the engine's steady-state operating point changes to ensure the completion of its main control tasks, leading to reduced surge margin, increased low-pressure turbine outlet temperature, and altered engine dynamics. This increases the risks of surge, over-temperature, and over-revving. Therefore, addressing the robust limitation and protection control problem for aero-engine performance degradation is crucial.

[0003] This invention primarily addresses the performance degradation problem of aero-engines. Based on the robust reinforcement learning framework, it proposes a robust reinforcement learning approach to combat performance degradation, thereby improving the robustness of the reinforcement learning controller's control strategy against aero-engine performance degradation. Finally, simulation experiments are used to verify the effectiveness and advantages of the robust deep reinforcement learning controller. Summary of the Invention

[0004] The technical problem to be solved by this invention is to improve the robustness of the deep reinforcement learning controller for aero-engines to engine performance degradation. Based on the robust reinforcement learning framework, a robust performance index for performance degradation is constructed, a robust action value function for performance degradation is defined, and the variable range of health parameters at a single step time is reduced by combining the stochastic environment domain method to improve the feasibility of the robust controller. An adversarial agent is introduced to find the lower bound of the health parameters and form a zero-sum game with the controller. The training of the robust deep network controller for aero-engines is realized based on the deterministic deep reinforcement learning algorithm, and simulation verification is performed.

[0005] This invention adopts the following technical solution: a robust control method for performance degradation of aero-engines based on deep reinforcement learning, characterized in that the method includes the following steps:

[0006] Step A: Introduce health parameters to quantify the performance degradation of the rotor components in the engine component-level model;

[0007] Step B: Construct a deep network controller for aero-engines based on the actor-critic framework;

[0008] Step C: Propose a robust performance index for performance degradation, define a robust action value function under engine performance degradation, construct a robust Bellman operator for performance degradation, and realize the iterative solution of the robust action value function.

[0009] Step D: Combining the random environment domain method, randomly initialize the health parameters of the component-level model to reduce the variable range of health parameters within the task loop;

[0010] Step E: Combine adversarial reinforcement learning to introduce adversarial agents, forming a zero-sum game with the controller;

[0011] Step F: Using a component-level model of a certain type of aero-engine as the interaction object, train a deep network controller based on a deterministic policy deep reinforcement learning algorithm;

[0012] Step G: Conduct simulation experiments based on a component-level model of a certain type of aero-engine to verify the robustness of the deep network controller to engine performance degradation.

[0013] Step A includes: using the ratio of the change in flow rate and efficiency of the main engine components relative to the corresponding baseline value of a healthy engine to the baseline value as the engine health parameter to reflect the degree of engine degradation. The vector expression for the health parameter is h = [△SE1, △SW1, △SE2, △SW2, △SE3, △SW3, △SE4, △SW4]. T In the formula Let △SW be the efficiency health parameter of the i-th pneumatic component. i =W i / W i * -1 represents the flow health parameter of the i-th pneumatic component, η and W represent the actual values ​​of the pneumatic component's efficiency and flow rate, respectively, and η * and W * These represent the actual values ​​of efficiency and flow rate of the air circuit components, respectively. The subscripts i = 1, 2, 3, 4 represent the fan, compressor, high-pressure turbine, and low-pressure turbine components, respectively.

[0014] Step B includes the following steps:

[0015] (1) Controller network input / output vector selection

[0016] Select main fuel flow rate W fb The cross-sectional area A8 of the tail nozzle throat is the control vector u. t =[Wfb A8] T , with action vector a t For the controller network output, the action vector and control vector have a momentum relationship: u t =(1-ε)u t-1 +εa t Select the compressor rotor conversion speed N. c,cor Engine pressure ratio EPR is the controlled variable, turbine after-temperature T6, and fan rotor physical speed N. f Compressor rotor physical speed N c Fan surge margin S mf Compressor surge margin S mc To constrain the variables, construct the engine state vector as x. t =[N c,cor ,EPR,N f N c ,T6,S mf ,S mc ] T The controlled variable command signal vector is ref = [N] c,cor,r EPR r ] T The environment vector includes flight altitude and flight Mach number env = [H, Ma]. T Construct the observation vector o t =[x t ,ref,env] T Construct a state vector s. t =[o t-m ,u t-m ,...,o t-1 ,u t-1 ,o t ] T In the formula, m represents the depth of the historical time series, expressed as the state vector s t As the input vector of the controller network.

[0017] (2) Construction of deep network structure

[0018] Construct an actor network as a controller network, with s t For input, a t For the output, a five-layer fully connected neural network is used, with each hidden layer containing 256 nodes. The ReLU function is used as the activation function for the intermediate layers, and the Tanh function is used as the activation function for the output layer. A critic network is constructed as the evaluation network, with [s] as the [evaluation function]. t ,a t ] T As input, with value function Q π (s t ,a tThe output is ), and the network structure is the same as that of the actor network.

[0019] Step C includes the following steps:

[0020] (1) Based on the robust reinforcement learning framework, a robust performance index for performance degradation is proposed:

[0021]

[0022] In the formula This represents the expectation of the initial probability distribution ρ0 of the state vector s. For the set of uncertainties in health parameters, h i The engine's health parameters at time i are... Let γ represent the expectation of the state probability distribution and control policy, r be the reinforcement learning reward function, and γ be the discount factor.

[0023] (2) Define the robust performance degradation action value function based on the robust performance degradation performance index:

[0024]

[0025] (3) Construct a robust Bellman operator to realize iterative computation of the value function:

[0026]

[0027] Step D includes the following steps:

[0028] (1) Determine the range of uncertainty of initial health parameters

[0029] Referring to the degree of change in health parameters within 6000 mission cycles of a real engine, the variable range of uncertainty in the initial health parameters is defined as follows:

[0030]

[0031] In the formula h high,i Let i be the size of the health parameter of the i-th health parameter after 6000 task cycles;

[0032] (2) Determine the variable range of uncertainty of health parameters within a single step time.

[0033] Define the variable range of health parameters within a single step as... In the formula k h The gain coefficient is represented by the circle (⊙). The circle operation represents the element-wise multiplication of two vectors to obtain a new vector.

[0034] (3) Propose robust performance index for random environment domain

[0035] An improved robustness performance index for stochastic environment domains is proposed:

[0036]

[0037] (4) Define a robust action value function for a random environment domain:

[0038]

[0039] and related Bellman operators:

[0040] Step E includes the following steps:

[0041] (1) Selection of input and output vectors for adversarial agents

[0042] To counteract action vector a ad,t The output vector of the adversarial agent network is given by h, where h is the relationship between the adversarial action vector and the health parameters. t+1 =h t +a ad,t Using the state vector s t As the input vector for adversarial agent networks.

[0043] (2) Construction of adversarial agent network structure

[0044] A five-layer fully connected neural network is used, with each hidden layer containing 256 nodes. The ReLU function is used as the activation function for the intermediate layers, and the Tanh function is used as the activation function for the output layer.

[0045] Step F includes the following steps:

[0046] (1) Initialize the actor, critic, and adversary networks;

[0047] (2) The network controller, adversarial agent, and component-level model interact to perform control tasks and collect actor training data (s,a,r,s′) and adversarial training data (s,a). ad (r, s′) and stored in the training data experience pool.

[0048] (3) In the training data experience pool Randomly sample n sets of training data and calculate the loss function of the critic network. Computational performance metrics

[0049] (4) Update the crit network parameter vector using gradient descent method Gradient ascent method for updating actor network parameter vectors Gradient descent method for updating the adaptive network parameter vector

[0050] (5) Repeat the above process until the maximum number of training iterations is reached, and the network training ends.

[0051] In step G: simulation experiments are conducted based on a component-level model of a certain type of aero-engine, and the robust adversarial learning (RA) + stochastic environment domain method (RD), robust adversarial learning (RA), stochastic environment domain method (RD), and benchmark methods are compared under no-fading, medium-fading, and high-fading conditions.

[0052] The present invention has the following beneficial effects:

[0053] (1) This invention combines the random environment domain method to reduce the variable range of health parameters in a single step and improve the feasibility of robust controllers.

[0054] (2) This invention introduces an adversarial agent and uses interactive training to find the lower bound of the performance index with respect to the health parameters, which facilitates the calculation of the robust action value function.

[0055] (3) The present invention can effectively improve the robustness of the deep reinforcement learning controller for aero-engines to engine performance degradation. Attached Figure Description

[0056] Figure 1 It is the interaction process between the controller, the antibody, and the aircraft engine;

[0057] Figure 2 It is an actor network structure;

[0058] Figure 3 It is a critic network structure;

[0059] Figure 4 The following is a comparison of dynamic simulation curves of a robust adversarial reinforcement learning controller (without performance degradation): (a) main fuel flow rate, (b) cross-sectional area of ​​the nozzle throat, (c) engine thrust, (d) turbine after-temperature, (e) fan surge margin, and (f) compressor surge margin.

[0060] Figure 5 The following is a comparison of dynamic simulation curves of a robust adversarial reinforcement learning controller (low performance degradation): (a) main fuel flow rate, (b) exhaust nozzle throat cross-sectional area, (c) engine thrust, (d) turbine after-temperature, (e) fan surge margin, and (f) compressor surge margin.

[0061] Figure 6The following is a comparison of dynamic simulation curves of a robust adversarial reinforcement learning controller (performance degradation in the middle): (a) main fuel flow rate, (b) cross-sectional area of ​​the nozzle throat, (c) engine thrust, (d) turbine after-temperature, (e) fan surge margin, and (f) compressor surge margin.

[0062] Figure 7 The following is a comparison of dynamic simulation curves of a robust adversarial reinforcement learning controller (high-performance degradation): (a) main fuel flow rate, (b) cross-sectional area of ​​the nozzle throat, (c) engine thrust, (d) turbine after-temperature, (e) fan surge margin, and (f) compressor surge margin. Detailed Implementation

[0063] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings:

[0064] In a specific embodiment of this invention, a component-level model of a certain type of turbofan engine is used as the controlled object, and a deep network trained using deep reinforcement learning is used as the controller carrier. The interaction process between the controller, the anti-coronavirus, and the aero-engine is as follows: Figure 1 As shown, before interaction, the engine states s0~ρ0 are randomly initialized, and the health parameters are also randomly initialized. The actor is the carrier of the controller network, with state vector s t As input, with action vector a t For the output, the control vector is calculated using the formula u. t =(1-ε)u t-1 +εa t The adversary agent is responsible for changing the engine's health parameters, outputting a. ad,t The relationship between health parameters and time is h. t =h t-1 +a ad,t The goal of the controller actor is to maximize performance metrics. The goal of adversarial agents is to find performance metrics. The lower bound in This means minimizing the performance metric, creating a zero-sum game between the two. The change in the aircraft engine's health parameter is h. t The control vector u is executed afterward. t The state vector transition is s t+1 Enter the next time step and loop and s t+1 The system serves as the input to the controller and the antibody, cycling until the task is completed.

[0065] The actor serves as the deep network carrier of the controller, and its structure is as follows: Figure 2As shown, this is a fully connected deep network with three hidden layers. The ReLU activation function is used for the intermediate layers, and the Tanh activation function is used for the output layer. The critic network serves as an auxiliary evaluation network, and its structure is as follows: Figure 3 As shown, the network uses a state vector s t With action vector a t As input, the value function Q(s,a|ν) and the MAX value function Q max (s,a|ν) is the output, representing a fully connected deep network with three hidden layers. All layers use the ReLU activation function. Adversary is used as the adversarial agent carrier, except that its output is a. ad,t Except for that, the rest is the same as the actor structure.

[0066] During the controller training process, the actor, critic, and adversary networks are first initialized. Then, following the interaction process described above, the network controller, adversarial agent, and component-level model interact to execute control tasks and collect actor training data (s,a,r,s′) and adversary training data (s,a). ad (r, s′) and store them in the training data experience pool; after completing an interaction, the training data experience pool is... Randomly sample n sets of training data and calculate the loss function of the critic network. Computational performance metrics Update the crit network parameter vector using gradient descent. Update the actor network parameter vector using gradient ascent. Update the parameter vector of the adversary network using gradient descent. Repeat the above process multiple times until the maximum number of training iterations is reached, at which point the network training ends.

[0067] The trained network controller was used for control tasks of an aero-engine component-level model, and the time series of engine state vectors and control vectors were output and compared. To assess the effectiveness of robust reinforcement learning methods for engine performance degradation, controllers trained using four methods were compared: 1) a nominal reinforcement learning method without considering robustness; 2) a robust adversarial reinforcement learning (RA) method, but without randomly initializing engine health parameters, where the variable health parameters of the adversarial network range from the health parameters in the absence of performance degradation. Within a smaller range 3) Randomized Environment Domain (RD): At the beginning of each interactive training round, the engine health parameters are randomly initialized. 4) Simultaneously employ random environment and robust adversarial reinforcement learning (RA+RD).

[0068] Table 1 shows a statistical comparison of the average cumulative reward of each method under different performance degradation conditions. When the engine does not experience performance degradation, nominal reinforcement learning has the highest average cumulative reward of 500.2. This is because the other three methods consider robustness to performance degradation, inevitably sacrificing some control performance in the absence of degradation. When the engine experiences low degradation, the average cumulative reward of nominal reinforcement learning decreases significantly to 43.0. RA and RD methods both exhibit some robustness, but their average cumulative rewards are not significant. The RA+RD method shows the most significant advantage, decreasing only from 396.5 to 370.2, becoming the highest average cumulative reward among the four methods under low degradation. Under medium degradation, the three robust methods still show improvement compared to the nominal reinforcement learning method, demonstrating some robustness. However, under high degradation, the RA method, which only uses robustness against reinforcement learning, fails, and its average cumulative reward is almost the same as that of the nominal reinforcement learning method. Under low, medium, and high degradation conditions, the controller trained by the RA+RD method achieves the highest average cumulative reward, and its performance improvement compared to nominal reinforcement learning is significant, demonstrating a clear advantage in robustness. Table 2 shows a statistical comparison of the average steady-state errors of each method under different performance degradation conditions. The analysis results are basically the same as those in Table 1, so we will not elaborate further.

[0069] To more intuitively compare the differences between the controllers trained by the four methods, at 0 altitude and 0 Mach number, the four controllers were used to execute the thrust command F from engine idle state to thrust F. in The direct thrust control task with a value of 1 was tested, and the simulation curves of the dynamic control process were compared. Figure 4 The simulation curves of the dynamic control process under the condition of no engine performance degradation are shown. Figure 4 (d)(e)(f) show that the limiting variables for each controller did not exceed the limits (in each simulation experiment, the fan speed N...). f and compressor speed N c No limits were exceeded, therefore the simulation curves for the two speeds were not shown in the paper, nor were they analyzed in detail. However, according to... Figure 4 (c) It can be seen that the controller trained by nominal reinforcement learning has the smallest steady-state control error, which is almost negligible. Figure 5 The simulation curves of the dynamic control process under low engine performance degradation conditions are shown. Figure 5 (d)(e)(f) show that the various limiting variables are still within their limits, but Figure 5 (c) shows a significant change in steady-state control error. All four controllers exhibit a certain degree of steady-state control error, with the controller trained by nominal reinforcement learning exhibiting the largest steady-state control error.

[0070] Figure 6The simulation curves of the dynamic control process under the condition of performance degradation in the engine are shown. At this time, there is a conflict between the tracking control command and the limit protection requirements. If the turbine after-temperature T6 is to be kept within the limit, the tracking requirement of the thrust control command cannot be met. Figure 6 (c) The controllers corresponding to the stochastic environments RD and RD+RA exhibited large control errors, while the controllers corresponding to the nominal reinforcement learning and robust adversarial reinforcement learning RA showed relatively smaller steady-state control errors. Figure 6 (d) The turbine after-temperature T6 of the controller corresponding to the nominal reinforcement learning and robust adversarial reinforcement learning RA has exceeded the limit, and the limiting protection mechanism has failed.

[0071] Furthermore, Figure 7 Simulation curves of the dynamic control process under engine performance degradation are presented. Except for the RA+RD method, the controllers trained by the other methods all exhibited the problem of exceeding the turbine after-temperature T6. The controller trained by the random environment RD even showed oscillation in the exhaust nozzle throat cross-sectional area A8, leading to instability in various engine state parameters. In summary, while the RD and RA methods possess certain robustness, they are still insufficient to cope with engine performance degradation. The RD+RA method, however, can significantly improve the controller's robustness to engine performance degradation at the cost of sacrificing some control performance under conditions without performance degradation, ensuring that the limiting protection mechanism is effective under low, medium, and high performance degradation conditions, and guaranteeing that the limiting variables do not exceed limits.

[0072] The present invention provides a robust control method for performance degradation of aero-engines based on deep reinforcement learning, comprising the following steps:

[0073] Step A: Introduce health parameters to quantify the performance degradation of the rotor components in the engine component-level model;

[0074] Step B: Construct a deep network controller for aero-engines based on the actor-critic framework;

[0075] Step C: Propose a robust performance index for performance degradation, define a robust action value function under engine performance degradation, construct a robust Bellman operator for performance degradation, and realize the iterative solution of the robust action value function.

[0076] Step D: Combining the random environment domain method, randomly initialize the health parameters of the component-level model to reduce the variable range of health parameters within the task loop;

[0077] Step E: Combine adversarial reinforcement learning to introduce adversarial agents, forming a zero-sum game with the controller;

[0078] Step F: Using a component-level model of a certain type of aero-engine as the interaction object, train a deep network controller based on a deterministic policy deep reinforcement learning algorithm;

[0079] Step G: Conduct simulation experiments based on a component-level model of a certain type of aero-engine to verify the robustness of the deep network controller to engine performance degradation.

[0080] Step A includes: using the ratio of the change in flow rate and efficiency of the main engine components relative to the corresponding baseline value of a healthy engine to the baseline value as the engine health parameter to reflect the degree of engine degradation. The vector expression for the health parameter is h = [△SE1, △SW1, △SE2, △SW2, △SE3, △SW3, △SE4, △SW4]. T In the formula Let △SW be the efficiency health parameter of the i-th pneumatic component. i =W i / W i * -1 represents the flow health parameter of the i-th pneumatic component, η and W represent the actual values ​​of the pneumatic component's efficiency and flow rate, respectively, and η * and W * These represent the actual values ​​of efficiency and flow rate of the air circuit components, respectively. The subscripts i = 1, 2, 3, 4 represent the fan, compressor, high-pressure turbine, and low-pressure turbine components, respectively.

[0081] Step B includes the following steps:

[0082] (1) Controller network input / output vector selection

[0083] Select main fuel flow rate W fb The cross-sectional area A8 of the tail nozzle throat is the control vector u. t =[W fb A8] T , with action vector a t For the controller network output, the action vector and control vector have a momentum relationship: u t =(1-ε)u t-1 +εa t Select the compressor rotor conversion speed N. c,cor Engine pressure ratio EPR is the controlled variable, turbine after-temperature T6, and fan rotor physical speed N. f Compressor rotor physical speed N c Fan surge margin S mf Compressor surge margin S mc To constrain the variables, construct the engine state vector as x. t =[N c,cor ,EPR,N f N c ,T6,S mf ,S mc ] TThe controlled variable command signal vector is ref = [N] c,cor,r EPR r ] T The environment vector includes flight altitude and flight Mach number env = [H, Ma]. T Construct the observation vector o t =[x t ,ref,env] T Construct a state vector s. t =[o t-m ,u t-m ,...,o t-1 ,u t-1 ,o t ] T In the formula, m represents the depth of the historical time series, expressed as the state vector s t As the input vector of the controller network.

[0084] (2) Construction of deep network structure

[0085] Construct an actor network as a controller network, with s t For input, a t For the output, a five-layer fully connected neural network is used, with each hidden layer containing 256 nodes. The ReLU function is used as the activation function for the intermediate layers, and the Tanh function is used as the activation function for the output layer. A critic network is constructed as the evaluation network, with [s] as the [evaluation function]. t ,a t ] T As input, with value function Q π (s t ,a t The output is ), and the network structure is the same as that of the actor network.

[0086] Step C includes the following steps:

[0087] (1) Based on the robust reinforcement learning framework, a robust performance index for performance degradation is proposed:

[0088]

[0089] In the formula This represents the expectation of the initial probability distribution ρ0 of the state vector s. For the set of uncertainties in health parameters, h i The engine's health parameters at time i are... Let γ represent the expectation of the state probability distribution and control policy, r be the reinforcement learning reward function, and γ be the discount factor.

[0090] (2) Define the robust performance degradation action value function based on the robust performance degradation performance index:

[0091]

[0092] (3) Construct a robust Bellman operator to realize iterative computation of the value function:

[0093]

[0094] Step D includes the following steps:

[0095] (1) Determine the range of uncertainty of initial health parameters

[0096] Referring to the degree of change in health parameters within 6000 mission cycles of a real engine, the variable range of uncertainty in the initial health parameters is defined as follows:

[0097]

[0098] In the formula h high,i Let i be the size of the health parameter of the i-th health parameter after 6000 task cycles;

[0099] (2) Determine the variable range of uncertainty of health parameters within a single step time.

[0100] Define the variable range of health parameters within a single step as... In the formula k h The gain coefficient is represented by the circle (⊙). The circle operation represents the element-wise multiplication of two vectors to obtain a new vector.

[0101] (3) Propose robust performance index for random environment domain

[0102] An improved robustness performance index for stochastic environment domains is proposed:

[0103]

[0104] (4) Define a robust action value function for a random environment domain:

[0105]

[0106] and related Bellman operators:

[0107] Step E includes the following steps:

[0108] (1) Selection of input and output vectors for adversarial agents

[0109] To counteract action vector a ad,t The output vector of the adversarial agent network is given by h, where h is the relationship between the adversarial action vector and the health parameters. t+1 =h t +a ad,tUsing the state vector s t As the input vector for adversarial agent networks.

[0110] (2) Construction of adversarial agent network structure

[0111] A five-layer fully connected neural network is used, with each hidden layer containing 256 nodes. The ReLU function is used as the activation function for the intermediate layers, and the Tanh function is used as the activation function for the output layer.

[0112] Step F includes the following steps:

[0113] (1) Initialize the actor, critic, and adversary networks;

[0114] (2) The network controller, adversarial agent, and component-level model interact to perform control tasks and collect actor training data (s,a,r,s′) and adversarial training data (s,a). ad (r, s′) and stored in the training data experience pool.

[0115] (3) In the training data experience pool Randomly sample n sets of training data and calculate the loss function of the critic network. Computational performance metrics

[0116] (4) Update the crit network parameter vector using gradient descent method Gradient ascent method for updating actor network parameter vectors Gradient descent method for updating the adaptive network parameter vector

[0117] (5) Repeat the above process until the maximum number of training iterations is reached, and the network training ends.

[0118] In step G: simulation experiments are conducted based on a component-level model of a certain type of aero-engine, and the robust adversarial learning (RA) + stochastic environment domain method (RD), robust adversarial learning (RA), stochastic environment domain method (RD), and benchmark methods are compared under no-fading, medium-fading, and high-fading conditions.

[0119] Table 1

[0120]

[0121] Table 2

[0122]

Claims

1. A robust control method for performance degradation of aero-engines based on deep reinforcement learning, characterized in that, The method includes the following steps: The degree of performance degradation of rotor components in engine component-level models is quantified using health parameters; A deep network controller for aero-engines based on a deep reinforcement learning framework is constructed, its performance degradation robustness index is determined, and a robust action value function under engine performance degradation is defined according to the performance degradation robustness index. A performance degradation robust Bellman operator is constructed to realize the iterative solution of the robust action value function. By combining the stochastic environment domain method, the health parameters of the component-level model are randomly initialized to reduce the variable range of health parameters within the task loop, and a stochastic environment domain robust performance index, a stochastic environment domain robust action value function, and related Bellman operators are constructed. Using an aero-engine component-level model as the interaction object, a deep network controller is trained based on a deterministic policy deep reinforcement learning algorithm. An adversarial agent with changeable health parameters is introduced to form a zero-sum game with the controller, which assists the controller training process. The performance degradation robustness index is as follows: ； In the formula This represents the expectation of the initial probability distribution of the state vector. For the set of uncertainties in health parameters, Let j be the engine's health parameter vector at time j. Represented in the health parameter vector State transition probability distribution of the engine component-level model. Represents the state vector exist Under probability distribution, action vector In control strategy Expectation under the distribution, Indicates the state Next action The instant reward value obtained, Represent the discount factor; construct the performance degradation robustness action value function based on the performance degradation robustness performance index: ； And construct a performance degradation robust Bellman operator Implement iterative solutions for value functions: .

2. The robust control method for performance degradation of aero-engines based on deep reinforcement learning as described in claim 1, characterized in that: The ratio of the change in the flow rate and efficiency of engine components relative to the corresponding baseline value of a healthy engine to the baseline value is used as an engine health parameter to reflect the degree of engine degradation. The health parameter vector expression is In the formula Let be the efficiency health parameter of the i-th pneumatic component. Let be the flow health parameter of the i-th gas path component. and These represent the changes in the efficiency η and flow rate W of the gas path components, respectively. and These represent the baseline values ​​for the efficiency and flow rate of the gas path components, respectively, with subscripts indicating the values. These respectively represent the fan, compressor, high-pressure turbine, and low-pressure turbine components.

3. The robust control method for performance degradation of aero-engines based on deep reinforcement learning as described in claim 1, characterized in that: In order to reduce the robustness performance index and the corresponding action value function, the lower bound is... To expand the solution range while preserving the controller's robustness across the entire performance degradation range, a stochastic environment domain method is introduced, which randomly initializes the engine health parameters at the initial moment within each task loop during the interaction process. Reduce the variable range of health parameters at a single moment. The range of the uncertainty set in the formula Based on this, a robust performance index for random environment domains is defined as follows: ； and robust action value function for random environment domain: and the corresponding Bellman operator: .

4. The robust control method for performance degradation of aero-engines based on deep reinforcement learning as described in claim 1, characterized in that: The deep reinforcement learning framework includes an actor (controller network), a critic (evaluation network), and an adversary (adversary agent). The adversary agent adjusts engine health parameters during the interaction process and outputs... The relationship between health parameters and time is as follows: The goal of the controller is to maximize the robustness metric in the stochastic environment domain, while the goal of the adversary is to find a lower bound in the robustness metric in the stochastic environment domain. That is, minimizing the robust performance index of the random environment domain. The controller and the advisor form a zero-sum game in the process of interacting with the engine component-level model, which assists the controller training.

5. The robust control method for performance degradation of aero-engines based on deep reinforcement learning as described in claim 4, characterized in that, Includes the following steps: (1) Initialize the actor, critic, and adversary networks; (2) The network controller, adversarial agent and component-level model interact to perform control tasks and collect actor training data. and adversary training data And stored in the training data experience pool. ; (3) In the training data experience pool Randomly sample n sets of training data and calculate the loss function of the critic network. ; Computational performance indicators ; (4) Update the critic network parameter vector using gradient descent method Gradient ascent method updates actor network parameter vector Gradient descent method updates the adaptive network parameter vector ; (5) Repeat the above process until the maximum number of training iterations is reached, and the network training ends.

Images