Aero-engine sliding mode speed control method based on reinforcement learning optimization

Abstract

This invention discloses a sliding mode speed control method for aero-engines based on reinforcement learning optimization, comprising the following steps: Step S1, acquiring raw data of the aero-engine; Step S2, performing denoising and smoothing processing on the acquired raw data; Step S3, performing system identification based on the processed fuel flow rate and processed rotor speed to obtain a second-order discrete state-space model of the aero-engine; Step S4, designing a sliding mode controller based on the second-order discrete state-space model of the aero-engine; Step S5, building a reinforcement learning environment according to the aero-engine sliding mode control process; Step S6, training using the SAC algorithm in deep reinforcement learning to finally obtain the optimal sliding mode controller; Step S7, using the optimal sliding mode controller to control the speed of the aero-engine to be controlled. This method solves the problem that the selection of control parameters for existing sliding mode controllers relies on engineering experience and trial-and-error methods, making it difficult to achieve global optimization.

Description

[Technical Field]

[0001] This invention belongs to the field of aero-engine control technology, specifically relating to a sliding mode speed control method for aero-engines based on reinforcement learning optimization. [Background Technology]

[0002] With the rapid development of the aviation industry, the performance requirements for aero-engines are increasing, especially in speed control, where high precision, high stability, and fast response have become key indicators. Traditional aero-engine speed control methods, such as proportional-integral-derivative (PID) control, while meeting basic requirements to a certain extent, often fall short when facing complex and ever-changing flight environments and stringent performance requirements. In recent years, sliding mode control (SMC), as a robust nonlinear control method, has attracted widespread attention in the field of aero-engine control. SMC designs a sliding surface, enabling the system state to reach and remain on the sliding surface within a finite time, thereby achieving precise control of the system state. However, the performance of SMC largely depends on the selection of its control parameters, and the optimization of these parameters often relies on engineering experience and trial-and-error methods, making it difficult to achieve global optimum.

[0003] Reinforcement learning, as a machine learning paradigm that learns optimal policies through interaction with the environment, has shown great potential in the control of complex systems. In particular, the Soft Actor-Critic (SAC) algorithm, as an advanced algorithm in the field of reinforcement learning, can maximize cumulative rewards while maintaining policy diversity and exploratory nature by introducing entropy regularization terms and automatically adjusting temperature parameters. It is very suitable for handling the complexity and uncertainty in the speed control of aero-engines.

[0004] The performance of existing sliding mode controllers largely depends on the selection of their control parameters, and the optimization of these parameters often relies on engineering experience and trial and error, making it difficult to achieve global optimum. [Summary of the Invention]

[0005] The purpose of this invention is to provide a sliding mode speed control method for aero-engines based on reinforcement learning optimization, in order to solve the problem that the selection of control parameters for existing sliding mode controllers relies on engineering experience and trial and error, making it difficult to achieve global optimum.

[0006] This invention adopts the following technical solution: a sliding mode speed control method for aero-engines based on reinforcement learning optimization, comprising the following steps:

[0007] Step S1: Conduct an open-loop test on the aero-engine that requires speed control, and collect the raw data of the aero-engine. The raw data includes the input fuel flow rate as the control variable and the output rotor original speed as the controlled variable.

[0008] Step S2: Perform noise reduction and smoothing processing on the collected raw data to obtain the processed fuel flow rate and the processed rotor speed;

[0009] Step S3: Based on the processed fuel flow rate and processed rotor speed, system identification is performed to obtain the second-order discrete state-space model of the aero-engine, specifically as follows:

[0010]

[0011] In the formula, u(k) is the input fuel flow rate at the current time, y(k) is the output rotor speed at the current time, x(k) is the state variable of the second-order discrete state-space model at the current time, x(k+1) is the state variable of the second-order discrete state-space model at the next time, and A, B, C, and D are the system matrix, input matrix, output matrix, and direct transfer matrix of the second-order discrete state-space model, respectively.

[0012] Next, verify the accuracy of the second-order discrete state-space model. If the accuracy meets the requirements, proceed to step S4; if the accuracy does not meet the requirements, increase the amount of raw data collected and return to continue executing steps S1 to S3 until the accuracy meets the requirements.

[0013] Step S4: Design a sliding mode controller based on the second-order discrete state-space model of the aero-engine;

[0014] Step S5: Build a reinforcement learning environment based on the sliding mode control process of the aero-engine, that is, define the action space, state space and reward function in the reinforcement learning environment, and realize the interactive simulation control process between the sliding mode controller and the second-order discrete state space model.

[0015] Step S6: Based on the reinforcement learning environment built in step S5, the SAC algorithm in deep reinforcement learning is used for training. During the training process, the parameters of the sliding mode controller are continuously optimized to finally obtain the optimal sliding mode controller.

[0016] Step S7: Use the optimal sliding mode controller to control the rotational speed of the aero-engine that needs to be controlled.

[0017] Furthermore, the action space, state space, and reward function of the reinforcement learning environment built in step S5 are specifically set as follows:

[0018] Action Space A t= [c, q, ε]; c, q and ε are the parameters of the sliding mode controller; where c is the sliding surface parameter, which affects the approach speed of the system state to the sliding surface and the dynamic performance of the system sliding along the sliding surface; ε and q are positive constants, which are used to adjust the approach speed and the smoothness of the approach process, respectively, and both are greater than 0;

[0019] State space S t = [y_cur, error_cur, error_cumulation], where y_cur is the current rotor speed of the aero-engine, error_cur is the deviation between the current rotor speed of the aero-engine and the target speed, and error_cumulation is the cumulative speed deviation at the current moment;

[0020] Reward function R t =e -0.1|error_cur| Reward R t The reward is inversely proportional to the current speed deviation error_cur, meaning the smaller the speed deviation, the greater the reward.

[0021] Furthermore, the interactive simulation control process between the sliding mode controller and the second-order discrete state-space model in the reinforcement learning environment built in step S5 is as follows:

[0022] Step S51: Assign the values ​​of the system matrix, input matrix, output matrix, direct transfer matrix, and initial state directly to the corresponding variables; then obtain the action output by the strategy, which is the parameter of the sliding mode controller in this control process;

[0023] Step S52: At each moment in the interactive simulation control process, the output at the current moment is obtained according to the sliding mode controller, and then the output rotor speed at the current moment and the state variables of the second-order discrete state space model at the next moment are obtained according to the second-order discrete state space model.

[0024] Step S53 repeats the process of step S52 until a complete interactive simulation control process is completed.

[0025] Furthermore, in step S6, the SAC algorithm in deep reinforcement learning uses five neural networks, which are, in order, Actor networks. Critic1 network Critic2 network Target Critic1 network TargetCritic2 network

[0026] Furthermore, the training process of the SAC algorithm is as follows:

[0027] S61. Initialize the five neural networks of the SAC algorithm and the experience replay pool M;

[0028] S62. Initialize the reinforcement learning environment and initial state s;

[0029] S63. Observe the current state and use... Collect state-action-reward samples, i.e., {s,a,r,s'}, where s is the current state, a is the random action at the current time, r is the reward at the current time, and s' is the state at the next time step; after collection, save this sample to the experience replay pool D;

[0030] S64. Collect a batch of samples {s} from the experience replay pool D. i ,a i ,r i ,s′ i There are N items in this batch, and the actions here are according to... Obtained by random sampling from a probability distribution;

[0031] Where s i ,a i ,r i ,s′ i The subscript i refers to the i-th sample among these N samples;

[0032] S65, Use and The target value is calculated using the following formula:

[0033]

[0034] Where γ is the discount factor and α is the entropy temperature coefficient. In the case of j=1 and 2, For the Target Critic1 network Target Critic2 network

[0035] S66, Calculation and The network's loss function is updated using gradient descent. and The network parameters θ1 and θ2 are shown in the following formula:

[0036]

[0037] S67. Calculate the loss function of the Actor network and update the Actor network using gradient descent. use or As shown in the formula below:

[0038]

[0039] S68. Calculate the loss function for entropy, and update the entropy using gradient descent, as shown in the following formula:

[0040] Where H0 is the target entropy set before training;

[0041] S69, Update target Q network and As shown in the formula below:

[0042] θ′ i =ρθ i +(1-ρ)θ′ i Where ρ is the soft update coefficient;

[0043] S610, return to S63, until training ends.

[0044] The beneficial effects of this invention are as follows: This invention employs the SAC algorithm from deep reinforcement learning for interactive training with the second-order discrete state-space model of an aero-engine. During the interactive training process, the parameters of the sliding mode controller are trained and optimized, ultimately yielding the optimal sliding mode controller. Using this trained optimal sliding mode controller to control the aero-engine's rotational speed improves control accuracy and response speed. This invention optimizes the cumbersome parameter tuning process of sliding mode control. Experimental results show that the aero-engine speed sliding mode controller trained and optimized using the SAC algorithm has a fast response speed and high control accuracy. Utilizing the SAC algorithm enables intelligent optimization and verification of control strategies, ensuring the effective implementation and high robustness of the control method, and promoting the intelligent development of aero-engine control technology. [Attached Image Description]

[0045] Figure 1 This is a flowchart of a sliding mode speed control method for aero-engines based on reinforcement learning optimization according to the present invention;

[0046] Figure 2 A comparison chart of the output of the original JT9D engine and the identified second-order discrete state-space model;

[0047] Figure 3 The flowchart shows the interactive simulation process between the sliding mode controller and the second-order discrete state-space model.

[0048] Figure 4 The diagram shows the Actor neural network structure of the SAC algorithm.

[0049] Figure 5 Here is the structure diagram of the Critic neural network for the SAC algorithm;

[0050] Figure 6 A cumulative reward distribution chart for SAC training rounds;

[0051] Figure 7 The diagram shows the control effect of the optimal sliding mode controller obtained through training on the original JT9D aero-engine.

Detailed Implementation Methods

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0053] A sliding mode speed control method for aero-engines based on reinforcement learning optimization, such as Figure 1 As shown, it includes the following steps:

[0054] Step S1: Conduct an open-loop test on the aero-engine that requires speed control, and collect the raw data of the aero-engine. The raw data includes the input fuel flow rate as the control variable and the output rotor original speed as the controlled variable.

[0055] Step S2: Perform noise reduction and smoothing processing on the collected raw data to obtain the processed fuel flow rate and the processed rotor speed;

[0056] Step S3: Based on the processed fuel flow rate and processed rotor speed, system identification is performed to obtain the second-order discrete state-space model of the aero-engine, specifically as follows:

[0057]

[0058] In the formula, u(k) is the input fuel flow rate at the current time, y(k) is the output rotor speed at the current time, x(k) is the state variable of the second-order discrete state-space model at the current time, x(k+1) is the state variable of the second-order discrete state-space model at the next time, and A, B, C, and D are the system matrix, input matrix, output matrix, and direct transfer matrix of the second-order discrete state-space model, respectively. Matrix A is a 2*2 matrix, matrix B is a 2*1 matrix, matrix C is a 2*2 matrix, and matrix D is a 2*1 matrix.

[0059] like Figure 2 As shown, the accuracy of the second-order discrete state-space model is verified again. If the accuracy meets the requirements, step S4 is performed; if the accuracy does not meet the requirements, the amount of raw data collected is increased, and steps S1 to S3 are returned to continue until the accuracy meets the requirements. That is, if the accuracy does not meet the requirements, the newly collected raw data is denoised and smoothed and then system identification is performed.

[0060] Among them, verifying the accuracy of the second-order discrete state-space model involves inputting the same fuel flow rate into the second-order discrete state-space model of the aero-engine identified by the system and the original aero-engine, and observing the degree of fit between the output rotor speeds of the two. If the degree of fit reaches 80%, the accuracy meets the requirements.

[0061] Step S4: Design a sliding mode controller based on the second-order discrete state-space model of the aero-engine;

[0062] Step S5: Based on steps S3 and S4, a reinforcement learning environment is built according to the sliding mode control process of the aero-engine, that is, the action space A is defined in the reinforcement learning environment. t State space S t Reward function R t And realize the interactive simulation control process between the sliding mode controller and the second-order discrete state-space model, such as Figure 3 As shown;

[0063] Step S6: Based on the reinforcement learning environment built in step S5, the SAC algorithm in deep reinforcement learning is used for training. During the training process, the parameters of the sliding mode controller are continuously optimized to finally obtain the optimal sliding mode controller.

[0064] Step S7: Use the optimal sliding mode controller to control the rotational speed of the aero-engine that needs to be controlled.

[0065] In some embodiments, the action space A of the reinforcement learning environment built in step S5 t State space S t and reward function R t Specifically set as follows:

[0066] Action Space A t = [c, q, ε], representing the action output by the reinforcement learning policy; where c is the sliding surface parameter, which affects the approach speed of the system state to the sliding surface and the dynamic performance of the system sliding along the sliding surface; ε and q are positive constants, used to adjust the approach speed and the smoothness of the approach process, respectively, and both are greater than 0;

[0067] State space S t = [y_cur, error_cur, error_cumulation], where y_cur is the current rotor speed of the aero-engine, error_cur is the deviation between the current rotor speed of the aero-engine and the target speed, and error_cumulation is the cumulative speed deviation at the current moment;

[0068] Reward function R t =e -0.1*|error_cur| Reward R tThe reward is inversely proportional to the current speed deviation error_cur, meaning the smaller the speed deviation, the greater the reward.

[0069] In some embodiments, the interactive simulation control process between the sliding mode controller and the second-order discrete state-space model in the reinforcement learning environment built in step S5 is as follows:

[0070] Step S51: Assign the values ​​of the system matrix, input matrix, output matrix, direct transfer matrix, and initial state directly to the corresponding variables; then obtain the action output by the strategy, which is the parameter c, q, and ε of the sliding mode controller in this control process;

[0071] Step S52: At each moment of the interactive simulation control process, the output u(k) of the current moment k is obtained according to the sliding mode controller, and then the output rotor speed y(k) of the current moment k and the state quantity x(k+1) of the second-order discrete state space model at the next moment are obtained according to the second-order discrete state space model.

[0072] Step S53 repeats the process of step S52 until a complete interactive simulation control process is completed.

[0073] In some embodiments, the sliding mode controller design in step S4 includes the following sub-steps:

[0074] S41. Suppose the target rotational speed signal is r(k), and its rate of change is dr(k). Use linear extrapolation to predict r(k+1) and its rate of change dr(k+1) at time k+1:

[0075]

[0076] Define R(k) = [r(k); dr(k)];

[0077] S42. Let the sliding mode function be s(k), and its formula is as follows:

[0078] s(k)=G*(R(k)-x(k)),

[0079] At this point, s(k+1) can be expressed by the following formula:

[0080]

[0081] S43. Select the exponential reaching law for sliding mode control. Different reaching laws in sliding mode control mainly affect the approach speed of the system state on the sliding surface and the system's ability to suppress chattering and noise. Among them, the exponential reaching law ensures both the speed of the approach process and reduces the speed at which the system reaches the sliding surface, thus helping to suppress chattering. The formula is shown below:

[0082]

[0083] In the formula, s is the sliding surface function. ε is its derivative, representing the rate of change of the sliding surface function; ε and q are positive constants, used to adjust the approach velocity and the smoothness of the approach process, respectively, both of which are greater than 0; sgn1(s) is a function constructed based on the sign function, defined as:

[0084]

[0085] S44: Based on the continuous sliding mode exponential reaching law of S43, the corresponding discrete reaching law is derived, as shown in the following formula:

[0086]

[0087] Where T is the sampling time;

[0088] S45: Combining the two s(k+1) formulas obtained from S42 and S44, the formula for the sliding mode controller is:

[0089] u(k)=(GB) -1 [GR(k+1)-GAx(k)-[T*[-ε*sgn1(s(k))-q*s(k)]+s(k)]],

[0090] Where G is the parameter of the sliding mode function s(k) in S42, and G = [c, 1].

[0091] In some embodiments, the SAC algorithm in deep reinforcement learning used in step S6 involves five neural networks, which are, in order, Actor networks. Critic1 network Critic2 network Target Critic1 network Target Critic2 network The training process of the SAC algorithm is as follows:

[0092] S61. Initialize the five neural networks of the SAC algorithm and the experience replay pool M.

[0093] S62. Initialize the reinforcement learning environment and initial state s.

[0094] S63. Observe the current state and use... Collect state-action-reward samples, i.e., {s, a, r, s'}, where s is the current state, a is the random action at the current time, r is the reward at the current time, and s' is the state at the next time step. After collection, save this sample to the experience replay pool D.

[0095] S64. Collect a batch of samples {s} from the experience replay pool D. i ,a i ,r i ,s′ i There are N items in this batch, and the actions here are according to... It is obtained by random sampling from a probability distribution. Where s i ,a i ,r i ,s′ i The subscript i refers to the i-th sample among these N samples.

[0096] S65, Use and The target value is calculated using the following formula:

[0097]

[0098] Here, γ is the discount factor used to calculate the discount when accumulating rewards, ensuring that the total sum of accumulated rewards is finite. α is the entropy temperature coefficient, which determines the degree to which entropy is maximized. This refers to the Target Critic1 network mentioned above. Target Critic2 network The formula calculates the minimum of these two values, with the index j taking values ​​of 1 and 2.

[0099] S66, Calculation and The network's loss function is updated using gradient descent. and The network parameters θ1 and θ2 are shown in the following formula:

[0100]

[0101] S67. Calculate the loss function of the Actor network and update the Actor network using gradient descent. use or As shown in the formula below:

[0102]

[0103] S68. Calculate the loss function for entropy and update the entropy using gradient descent. The formula is shown below. Where H0 is the target entropy set before training.

[0104]

[0105] S69, Update target Q network and As shown in the formula below:

[0106] θ′ i =ρθ i +(1-ρ)θ′ i , where ρ is the soft update coefficient, used to control the target network weight update speed.

[0107] S610, return to S63, until training ends.

[0108] Example

[0109] This embodiment uses the JT9D aero-engine model based on T-MATS as the control object, and uses the aero-engine sliding mode speed control method proposed in this invention to achieve precise control of the low-pressure rotor speed of the JT9D aero-engine. The specific implementation steps are as follows:

[0110] SS1. An open-loop experiment was conducted on the JT9D aero-engine. The initial low-pressure rotor speed of the JT9D aero-engine was set to 3667 rpm. The input fuel flow rate was set as a step signal with an initial value of 0.015 kg / s, which increased to 0.020 kg / s at 20 s. The simulation time was 50 s in total, with a simulation step size of 0.001 s. The raw data of the JT9D aero-engine, namely the input fuel flow rate and the original rotor speed, were collected. The fuel flow rate was used as the control variable, and the original rotor speed was used as the controlled variable.

[0111] SS2. The collected raw data is denoised and smoothed to obtain the processed fuel flow and rotor speed of the JT9D aero-engine.

[0112] SS3. Based on the processed fuel flow rate and rotor speed, system identification is performed to obtain the second-order discrete state-space model of the JT9D aero-engine:

[0113]

[0114] The system identified the following:

[0115] A=[0.999855555415616,-0.000402443549474426;-0.0000924334128923676,0.99977559432944 9], B=[0.001551116; 0.000918850718750047], C=[42903.7563216226,-4.61690338784296], D=0.

[0116] The initial state is x(0) = [0.0855; 0.0431].

[0117] The step fuel flow signal from SS1 was input into the original JT9D aero-engine model and the identified second-order discrete state-space model of the JT9D aero-engine, respectively. The resulting output rotor speed comparison diagram is shown in the attached figure. Figure 2 As shown. From Figure 2 It can be seen that the second-order discrete state-space model of the JT9D aero-engine identified by the system has a good fit of 81.22% with the original JT9D aero-engine model. This indicates that the model identified by the system can be used for subsequent sliding mode controller design.

[0118] SS4. Based on the second-order discrete state-space model of the JT9D aero-engine, the sliding mode controller is designed as shown in the following formula:

[0119] u(k)=(GB) -1 [GR(k+1)-GAx(k)-[T*[-ε*sgn1(s(k))-q*s(k)]+s(k)]],

[0120] Where G is the parameter of the sliding mode function in S42, and G = [c, 1].

[0121] SS5. Use the gym library to build a reinforcement learning environment for the sliding mode control of the JT9D aero-engine. Set up the action space, state space, and reward function in sequence, and realize the interactive simulation control process between the sliding mode controller and the second-order discrete state space model.

[0122] In this context, the action space represents the action output by the reinforcement learning policy, which consists of three parameters of the sliding mode controller in SS4. In the state space, y_cur is the current low-pressure rotor speed of the JT9D aero-engine, error_cur is the deviation between the current low-pressure rotor speed and the target low-pressure rotor speed, and error_cumulation is the cumulative low-pressure rotor speed deviation at the current moment. The reward function contains a reward R. t The reward is inversely proportional to the current low-pressure rotor speed deviation error_cur, meaning the smaller the speed deviation, the greater the reward.

[0123] In the interactive simulation control process between the sliding mode controller and the second-order discrete state space model in the reinforcement learning environment built in this example, the total time of one interactive simulation is set to 20s, the simulation step size is 0.001s, and the maximum reward for each simulation step is 1. Then the maximum total reward for completing one interactive simulation is 20000.

[0124] SS6, based on the environment built in SS5, uses the SAC algorithm from deep reinforcement learning based on Stable-Baselines3 for training. The Actor neural network structure of the SAC algorithm is shown in the appendix. Figure 4As shown, the Critic neural network structure is as follows (see attached diagram). Figure 5 As shown in the diagram. The learning rate was set to 3e-4, and all other hyperparameters were set to their default values ​​for Stable-Baselines3. The number of training epochs was set to 300, and the optimal model was saved based on the maximum cumulative reward per epoch. During training, the parameters of the sliding mode controller were continuously optimized to obtain the optimal sliding mode controller.

[0125] The cumulative reward distribution chart for SAC training rounds is shown below. Figure 6 As shown, from Figure 6 As can be seen, the model reaches its optimum after 229 training rounds, at which point the maximum cumulative reward for the maximum number of rounds is 18962.28432. This means that the sliding mode controller parameters given by SAC at this point are the optimal parameters. These optimal parameters are:

[0126] c=37178.88671875, q=72.9747314453125, ε=57.26795196533203

[0127] SS7. Using this optimal sliding mode controller to control the original JT9D aero engine, setting the target rotor speed as a step signal, the control effect is as follows: Figure 7 The control effect of the optimal sliding mode controller obtained through training on the original JT9D aero-engine is shown in the attached diagram. Figure 7 As can be seen, when the target low-pressure rotor speed changes, the control method of this invention can quickly control the low-pressure rotor speed of the JT9D aero-engine to reach the target low-pressure rotor speed, i.e., the settling time is very short, there is no overshoot, and the steady-state error is also very small. This demonstrates that the sliding mode controller designed according to the method proposed in this invention has excellent fast tracking capability and control accuracy.

[0128] In traditional sliding mode control, the performance of the control system highly depends on a series of carefully designed control parameters. Adjusting these parameters typically requires engineers with a strong background in control theory and extensive practical experience, finding the optimal or near-optimal parameter combination through repeated trial and error and experimental verification. This process is not only time-consuming and labor-intensive, but also often fails to guarantee the robustness and stability of the final control system, especially when facing complex and dynamic systems. Traditional sliding mode control methods often require complex parameter tuning and trial-and-error processes. In contrast, this invention introduces the SAC algorithm, an advanced reinforcement learning algorithm, to automatically complete the complex parameter tuning process in traditional sliding mode control. The core of this invention lies in constructing a reinforcement learning environment corresponding to the sliding mode control system. This environment can simulate the dynamic behavior of the system and allows the SAC algorithm to optimize the sliding mode controller parameters through trial and error learning, which greatly simplifies the design of the control system. Engineers only need to focus on the training and verification of the SAC algorithm to obtain the optimal control parameters, reducing the development difficulty and cost of the control system.

[0129] This invention utilizes deep learning and optimization of the SAC algorithm to find the optimal SMC control parameters for a specific aero-engine model, thereby achieving high-precision control of engine speed.

[0130] The reinforcement learning SAC algorithm used in this invention incorporates motion noise. Specifically, when the SAC algorithm's strategy provides the motion (i.e., the three parameters of the sliding mode controller), it adds a certain amount of Gaussian noise to the output motion. This noise allows the algorithm to provide a wider range of motions during training. This enables the sliding mode controller trained using the SAC algorithm to undergo extensive exploration during training and learn how to respond appropriately under different states, thus exhibiting stronger robustness. Even when encountering unforeseen disturbances during actual operation, the control system can maintain stable performance, ensuring the safe operation of the aero-engine.

Claims

1. A sliding mode speed control method for aero-engines based on reinforcement learning optimization, characterized in that, Includes the following steps: Step S1: Conduct an open-loop test on the aero-engine that requires speed control, and collect the raw data of the aero-engine. The raw data includes the input fuel flow rate as the control variable and the output rotor original speed as the controlled variable. Step S2: Perform noise reduction and smoothing processing on the collected raw data to obtain the processed fuel flow rate and the processed rotor speed; Step S3: Based on the processed fuel flow rate and processed rotor speed, system identification is performed to obtain the second-order discrete state-space model of the aero-engine, specifically as follows: In the formula, u(k) is the input fuel flow rate at the current time, y(k) is the output rotor speed at the current time, x(k) is the state variable of the second-order discrete state-space model at the current time, x(k+1) is the state variable of the second-order discrete state-space model at the next time, and A, B, C, and D are the system matrix, input matrix, output matrix, and direct transfer matrix of the second-order discrete state-space model, respectively. Next, verify the accuracy of the second-order discrete state-space model. If the accuracy meets the requirements, proceed to step S4; if the accuracy does not meet the requirements, increase the amount of raw data collected and return to continue executing steps S1 to S3 until the accuracy meets the requirements. Step S4: Design a sliding mode controller based on the second-order discrete state-space model of the aero-engine; Step S5: Build a reinforcement learning environment based on the sliding mode control process of the aero-engine, that is, define the action space, state space and reward function in the reinforcement learning environment, and realize the interactive simulation control process between the sliding mode controller and the second-order discrete state space model. Step S6: Based on the reinforcement learning environment built in step S5, the SAC algorithm in deep reinforcement learning is used for training. During the training process, the parameters of the sliding mode controller are continuously optimized to finally obtain the optimal sliding mode controller. Step S7: Use the optimal sliding mode controller to control the rotational speed of the aero-engine that needs to be controlled.

2. The method for sliding mode speed control of aero-engines based on reinforcement learning optimization according to claim 1, characterized in that, The action space, state space, and reward function of the reinforcement learning environment built in step S5 are specifically set as follows: Action Space A t = [c, q, ε]; c, q and ε are the parameters of the sliding mode controller; where c is the sliding surface parameter, which affects the approach speed of the system state to the sliding surface and the dynamic performance of the system sliding along the sliding surface; ε and q are positive constants, which are used to adjust the approach speed and the smoothness of the approach process, respectively, and both are greater than 0; State space S t = [y_cur, error_cur, error_cumulation], where y_cur is the current rotor speed of the aero-engine, error_cur is the deviation between the current rotor speed of the aero-engine and the target speed, and error_cumulation is the cumulative speed deviation at the current moment; Reward function R t =e -0.1*|error_cur| Reward R t The reward is inversely proportional to the current speed deviation error_cur, meaning the smaller the speed deviation, the greater the reward.

3. The method for sliding mode speed control of aero-engines based on reinforcement learning optimization according to claim 2, characterized in that, The interactive simulation control process between the sliding mode controller and the second-order discrete state-space model in the reinforcement learning environment built in step S5 is as follows: Step S51: Assign the values ​​of the system matrix, input matrix, output matrix, direct transfer matrix, and initial state directly to the corresponding variables; then obtain the action output by the strategy, which is the parameter of the sliding mode controller in this control process; Step S52: At each moment in the interactive simulation control process, the output at the current moment is obtained according to the sliding mode controller, and then the output rotor speed at the current moment and the state variables of the second-order discrete state space model at the next moment are obtained according to the second-order discrete state space model. Step S53 repeats the process of step S52 until a complete interactive simulation control process is completed.

4. The method for sliding mode speed control of aero-engines based on reinforcement learning optimization according to claim 3, characterized in that, In step S6, the SAC algorithm in deep reinforcement learning uses five neural networks, which are, in order, Actor networks. Criticl network Critic2 network Target Critic1 network Target Critic2 network 5. The method for sliding mode speed control of aero-engines based on reinforcement learning optimization according to claim 4, characterized in that, The training process of the SAC algorithm is as follows: S61. Initialize the five neural networks of the SAC algorithm and the experience replay pool M; S62. Initialize the reinforcement learning environment and initial state s; S63. Observe the current state and use... Collect state-action-reward samples, i.e., {s, a, r, s′}, where s is the current state, a is the random action at the current time, r is the reward at the current time, and s′ is the state at the next time step; after collection, save this sample to the experience replay pool D; S64. Collect a batch of samples {s} from the experience replay pool D. i a i r i ,s′ i There are N items in this batch, and the actions here are according to... Obtained by random sampling from a probability distribution; Where s i a i r i ,s′ i The subscript i refers to the i-th sample among these N samples; S65, Use and The target value is calculated using the following formula: Where γ is the discount factor and α is the entropy temperature coefficient. In the case of j=1 and 2, For the Target Critic1 network Target Critic2 network S66, Calculation and The network's loss function is updated using gradient descent. and The network parameters θ1 and θ2 are shown in the following formula: S67. Calculate the loss function of the Actor network and update the Actor network using gradient descent. use or As shown in the formula below: S68. Calculate the loss function for entropy, and update the entropy using gradient descent, as shown in the following formula: Where H0 is the target entropy set before training; S69, Update target Q network and As shown in the formula below: θ′ i =ρθ i +(1-ρ)θ′ i Where ρ is the soft update coefficient; S610, return to S63, until training ends.

Images