A Compressor Active Surge Control Method and System Based on Deep Reinforcement Learning

Abstract

The present invention discloses a compressor active surge control method and system based on deep reinforcement learning, which relates to the field of aero-engine active stability control. The steps include: (1) establishing a compressor mathematical model including actuators; (2) establishing a deep reinforcement learning agent simulation training environment for the compressor active surge control task; (3) training the agent using the soft actor-critic algorithm; (4) after the training is completed, fixing the weight parameters of the action network and deploying it to the electronic controller for online application. The present invention takes deep reinforcement learning as a new idea to solve the complex nonlinearity in the design of the compressor active surge control system, improves the adaptability and robustness of the controller, and greatly reduces the controller design difficulty.

Description

Technical Field

[0001] The present invention relates to the technical field of active stability control of aeroengines, and particularly relates to a method and system for active surge control of a compressor based on deep reinforcement learning. Background Art

[0002] Surge, as a typical flow pattern of compressor instability, has a great impact on the performance and safety of aeroengines. In severe cases, it will also cause engine failures, leading to catastrophic consequences. In order to prevent the compressor from crossing the instability boundary and experiencing rotating stall and surge, people often adopted passive anti-surge measures in the early days, that is, leaving enough surge margin when designing the compressor. This is an open-loop control idea, which can reduce the possibility of unstable states to a certain extent, but at the same time greatly limits the working flow rate and pressure ratio range of the compressor, sacrificing performance and operating efficiency. With the in-depth study of the unstable characteristics of the compressor, the idea of active surge control emerged, that is, suppressing the formation and development of pressure or flow disturbances in the flow field at the initial stage of compressor surge through the feedback control of actuators such as swinging blade rows, high-pressure jets, loudspeakers, piston damping mechanisms, controllable regulating valves, tight connection valves, and throttle valves, so that the compressor operates stably in the high-pressure ratio and high-efficiency region within the surge boundary.

[0003] The patent with the publication number CN113279997A proposed an active control system for aeroengine surge based on fuzzy switching of a controller, using a modal control method of Lyapunov stability theory to design multiple basic controllers applicable to different working ranges, and determining the final control quantity by weighted fusion of the control signals of each basic controller according to the principle of fuzzy switching; the patent with the publication number CN109339954A proposed an active control method based on the aerodynamic instability of aeroengine compressor components, using the estimated feedback of the compressor pressure coefficient and flow coefficient combined with bifurcation theory to design a surge active controller. The above methods design controllers from the perspective of compressor model characteristics analysis, making the whole system satisfy Lyapunov stability. Although these methods can control the compressor to operate in a stable state to a certain extent, these control methods require accurate model parameters and set some constraints. In other words, they do not consider external disturbances and model uncertainties, resulting in these control methods not being robust. In addition, these non-linear control algorithms involve multiple matrix differentiations and inverse calculations, with high computational complexity, and have high requirements for the online computing ability of the surge control system. Therefore, there are limitations when deployed to engineering applications. Summary of the Invention

[0004] To solve the above problems, the present invention proposes a compressor active surge control method and system based on deep reinforcement learning to achieve robust adaptive optimal control under the conditions of compressor model uncertainty and external disturbances.

[0005] The technical solution proposed by the present invention is as follows:

[0006] A compressor active surge control method and system based on deep reinforcement learning, comprising the following steps:

[0007] (1) Establish a compressor mathematical model including an actuator;

[0008] (2) Establish a deep reinforcement learning agent simulation training environment for the compressor active surge control task;

[0009] (3) Train the agent using the Soft Actor-Critic algorithm;

[0010] (4) Deploy the action network of the trained agent to the electronic controller for online application.

[0011] For the establishment of the compressor mathematical model including an actuator in step (1), the flow coefficient-pressure ratio coefficient characteristic term of the compressor is identified using the real compressor physical characteristic data; and the final compressor surge dynamic model is established according to the identified compressor characteristic term.

[0012] Furthermore, the real compressor physical characteristic data describes the compressor pressure ratio π when the flow rate is m at different relative percentage speeds n of the compressor;

[0013] Furthermore, the method for identifying the compressor characteristic term is to dimensionlessize the pressure ratio and flow rate in the real compressor physical characteristic data to obtain the pressure coefficient and the flow coefficient:

[0014]

[0015] where φ is the flow coefficient; ψ is the pressure coefficient; ρ is the internal gas density of the compressor; A c is the equivalent cross-sectional area of the internal flow path of the compressor; U is the rim line speed at the mid-diameter of the compressor rotor; p0 is the ambient pressure;

[0016] Subsequently, a cubic surface equation is used for data fitting by the least squares method:

[0017]

[0018] where ψ(φ, n) is the compressor characteristic term; a0, a1, b0, b1, c0, c1 are fitting coefficients;

[0019] Further, the actuator is a close-coupled valve, and the final compressor mathematical model is:

[0020]

[0021] where u is the control quantity input to the close-coupled valve actuator, which is the model input; φ is the flow coefficient of the model output; ψ is the pressure coefficient of the model output; B is the characteristic B parameter; l c is the equivalent length of the compressor; γ T is the throttle valve opening; d φ and d ψ are the disturbance and uncertainty terms of the flow coefficient and pressure coefficient respectively.

[0022] The establishment of the deep reinforcement learning agent simulation training environment for the active surge control task of the compressor in step (2) includes:

[0023] Step (2.1), given the pressure ratio coefficient reference command p ref of the compressor as the goal of the active surge control of the compressor, the pressure ratio coefficient reference command is used to ensure that the pressure ratio of the compressor smoothly transitions to the low-flow region outside the surge boundary, and is given by the following formula:

[0024]

[0025] where τ is the inertia coefficient; C is the final value of the pressure coefficient reference command.

[0026] Step (2.2), select the observation variable O as the pressure ratio coefficient reference command p ref in k historical control cycles and the current control cycle, the flow coefficient φ of the compressor model, the pressure coefficient ψ of the compressor model, the control error e = p ref - ψ and its integral eint and derivative edot, then the observation variable O t at time t is expressed as a multivariate time series matrix with a dimension of (k + 1, 6).

[0027]

[0028] Step (2.3), design the action network and evaluation network of the agent;

[0029] The action network π θ of the agent includes an input layer, a fully connected layer, an LSTM layer, a relu activation function layer, and then outputs through two branch networks respectively; Branch network 1 consists of a fully connected layer, a relu activation function, a fully connected layer, and an output layer 1 in sequence; Branch network 2 consists of a fully connected layer, a relu activation function, a fully connected layer, a softplus activation function layer, and an output layer 2 in sequence; The input parameter of the action network is the observable O, and the output layer 1 is the mean value of the control quantity The standard deviation σ of the 2-bit control quantity of the output layer;

[0030] The agent includes two evaluation networks with exactly the same structure, namely and the evaluation network Each evaluation network is composed of the merger of two branch networks. Branch network 1 consists of an input layer 1, a fully connected layer, a relu activation function layer, and a fully connected layer in sequence; branch network 2 consists of an input layer 2 and a fully connected layer in sequence; after the outputs of branch network 1 and branch network 2 are concatenated into a high-dimensional vector by a concat layer, they pass through an LSTM recurrent neural network layer, a relu activation function, a fully connected neural network layer, and an output layer in sequence; the input of branch network 1 of the evaluation network is the observation quantity O, and the input of branch network 2 is the control quantity u at the previous moment t-1 ; the output of the evaluation network is the expectation of the reward that can be obtained from the current observation quantity and control quantity.

[0031] Step (2.4), according to the observation quantity and the designed reward function r, specifically as follows:

[0032] r = r1 + r2 + r3

[0033] Where:

[0034]

[0035] Based on the deep reinforcement learning agent simulation training environment built in step 2), step (3) uses the soft actor-critic algorithm to train the agent, specifically including the following steps:

[0036] Step (3.1), establish the target evaluation network and the target evaluation network whose structures are respectively the same as those of the evaluation network and ;

[0037] Step (3.2), initialize the weight parameters w1, w2, and θ of the evaluation network evaluation network and the action network π θ with random parameters; then use to initialize the weight of the target evaluation network Use to initialize the weight of the target evaluation network

[0038] Step (3.3), initialize the experience replay pool R, set the number of training episodes E, simulation time T, simulation sampling step size ΔT, number of training loops λ, define the discount factor γ, and exponential moving average coefficient τ;

[0039] Step (3.4), the round robin starts;

[0040] Step (3.5), the simulation starts;

[0041] Step (3.6), at time t of the simulation, input O t+1 into the action network to obtain the control quantity u t ; execute u t , calculate the reward r t , and at this time the environmental state becomes O t+1 ;

[0042] Step (3.7), store (O t+1 , u t , r t , O t+1 ) as a sample into the experience replay pool R;

[0043] Step (3.8), the training loop starts;

[0044] Step (3.9), sample N samples from R and update the weight parameters of all networks:

[0045]

[0046] Step (3.10), execute step (3.9) until the training loop ends;

[0047] Step (3.11), execute steps (3.5) to (3.10) until the simulation ends;

[0048] Step (3.12), execute steps (3.4) to (3.11) until the round robin ends;

[0049] Step (4), after training is completed, fix the weight parameters of the action network and deploy them to the electronic controller. In each control cycle, the electronic controller receives the observation variables from the compressor in real time, inputs them into the trained action network, and finally outputs the control signal to the compressor connection valve to ensure the stable operation of the compressor;

[0050] Furthermore, in the deployment to the electronic controller, the standard deviation σ of the control quantity output by the action network is set to 0, so the mean value of the control quantity output by the action network is directly used as the final control signal u.

[0051] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0052] The compressor active surge control method and system based on deep reinforcement learning proposed by the present invention can ensure that the compressor accurately tracks the pressure ratio coefficient command in the presence of complex non - linearities and external disturbances, effectively expanding the stable operating range of the compressor and ensuring the efficient and reliable operation of the aero - engine. At the same time, with the powerful self - learning ability of deep reinforcement learning, the present invention solves the problem of complex manual design of the compressor active surge controller. Brief Description of the Drawings

[0053] Figure 1 It is the technical principle diagram of the present invention;

[0054] Figure 2(a) shows the actual physical characteristic data of the compressor in the embodiment of the present invention;

[0055] Figure 2(b) shows the dimensionless compressor characteristic data in the embodiment of the present invention;

[0056] Figure 2(c) shows the cubic surface graph fitted from the dimensionless compressor characteristic data in the embodiment of the present invention;

[0057] Figure 3(a) shows the action network structure diagram of the deep reinforcement learning agent in the present invention;

[0058] Figure 3(b) shows the evaluation network structure diagram of the deep reinforcement learning agent in the present invention;

[0059] Figure 4 It is the change graph of the reward function in the training process of the agent in the embodiment of the present invention;

[0060] Figure 5 It is the comparison graph of the control effects of three compressor active surge control methods. Detailed Embodiment

[0061] The technical solution of the present invention will be further explained below through a specific embodiment in combination with the drawings.

[0062] Refer to the attached Figure 1 , a compressor active surge control method and system based on deep reinforcement learning, includes the following steps;

[0063] (1) Establish a compressor mathematical model including an actuator;

[0064] Use the compressor physical characteristic data as shown in Figure 2(a) to identify the flow coefficient - pressure ratio coefficient characteristic term of the compressor. This compressor physical characteristic data describes the distribution of the compressor pressure ratio π when the flow rate is m at the relative percentage speed n.

[0065] Perform the following dimensionless processing on the pressure ratio and flow rate in the actual compressor physical characteristic data to obtain the pressure coefficient and flow coefficient as shown in Figure 2(b);

[0066]

[0067]

[0068] where φ is the flow coefficient; ψ is the pressure coefficient; ρ is the gas density inside the compressor, taking 1.225 kg / m 3 ; A c is the equivalent cross-sectional area of the internal flow passage of the compressor, taking 0.0291 m 2 ; U is the rim line speed at the mean diameter of the compressor rotor, taking 927.63 m / s; p0 is the ambient pressure, taking 100 kPa;

[0069] Finally, the following cubic surface equation is used to perform data fitting by the least squares method.

[0070]

[0071] where ψ(φ, n) is the compressor characteristic term after fitting; a0, a1, b0, b1, c0, c1 are fitting coefficients;

[0072] The cubic surface fitted in this embodiment is shown in Figure 2(c):

[0073]

[0074] The final compressor mathematical model including the tightly connected valve actuator is:

[0075]

[0076] where u is the control quantity input to the tightly connected valve actuator, which is the model input; φ is the flow coefficient output by the model; ψ is the pressure coefficient output by the model; B is the dimensionless B parameter, taking 1.8; l c is the equivalent length of the compressor, taking; γ T is the throttle valve opening, taking 0.6; d φ and d ψ are the disturbance and uncertainty terms of the flow coefficient and the pressure coefficient respectively, which are given by the following formula in this embodiment:

[0077] d ψ = 0.02sin(0.1t) + 0.02cos(0.4t)

[0078] d φ = 0.02sin(0.1t) + 0.02cos(0.4t)

[0079] (2) Establish a deep reinforcement learning agent simulation training environment for the active surge control task of the compressor;

[0080] Step (2.1), given the compressor pressure ratio coefficient reference command p ref As the target of the compressor active surge control, the pressure ratio coefficient reference command is used to ensure that the compressor pressure ratio smoothly transitions to the low flow region outside the surge boundary, and is given by the following formula:

[0081]

[0082] where τ is the inertia coefficient, taking 0.05; C is the final value of the pressure coefficient reference command, taking 0.6; when solving the equation, p ref The initial value is set to 0.6568;

[0083] Step (2.2), select the observation variable O as the pressure ratio coefficient reference command p ref in the three historical control cycles and the current control cycle, the flow coefficient φ of the compressor model, the pressure coefficient ψ of the compressor model, the control error e = p ref -ψ and its integral eint and derivative edot; then the observation variable O at time t t is expressed as a multivariate time series matrix of dimension (4, 6):

[0084]

[0085] Step (2.3), design the action network and evaluation network of the intelligent agent;

[0086] The action network π of the intelligent agent θ As shown in Figure 3(a), includes an input layer, a fully connected layer, an LSTM layer, a relu activation function layer, and then outputs through two branch networks respectively; Branch network 1 consists of a fully connected layer, a relu activation function, a fully connected layer and an output layer 1 in sequence; Branch network 2 consists of a fully connected layer, a relu activation function, a fully connected layer, a softplus activation function layer and an output layer 2 in sequence; the number of neurons in the fully connected layer and the LSTM layer of the action network is 128; the input parameter of the action network is the observation O, and the output layer 1 is the mean value of the control quantity The output layer 2 is the standard deviation σ of the control quantity.

[0087] The intelligent agent includes two evaluation networks with exactly the same structure as shown in Figure 3(b), which are respectively and evaluation network Each evaluation network is composed of the merger of two branch networks. Branch network 1 consists of an input layer 1, a fully connected layer, a relu activation function layer, and a fully connected layer in sequence; branch network 2 consists of an input layer 2 and a fully connected layer in sequence; after the outputs of branch network 1 and branch network 2 are concatenated into a high-dimensional vector by a concat layer, they pass through an LSTM recurrent neural network layer, a relu activation function, a fully connected neural network layer, and an output layer in sequence; the number of neurons in the fully connected layer and the LSTM layer of the evaluation network is 128; the input of branch network 1 of the evaluation network is the observed quantity O, and the input of branch network 2 is the control quantity u at the previous moment. t-1 ; the output of the evaluation network is the expectation of the reward that can be obtained from the current observed quantity and control quantity.

[0088] Step (2.4), according to the observed quantity and design the reward function r, specifically as follows:

[0089] r = r1 + r2 + r3

[0090] Where:

[0091]

[0092] (3) Use the soft actor-critic algorithm to train the agent;

[0093] Step (3.1), establish the target evaluation network and the target evaluation network whose structures are respectively the same as those of the evaluation network and ;

[0094] Step (3.2), initialize the weight parameters w1, w2, and θ of the evaluation network evaluation network and the action network π θ with random parameters; then use 's weights to initialize the target evaluation network Use 's weights to initialize the target evaluation network

[0095] Step (3.3), initialize the experience replay pool R, set the number of training episodes E = 2000, the simulation time T = 300s, the simulation sampling step size ΔT = 0.02s, the training loop coefficient λ = 0.99, define the discount factor γ = 0.99, and the exponential moving average coefficient τ = 0.5;

[0096] Step (3.4), start the episode loop;

[0097] Step (3.5), start the simulation;

[0098] Step (3.6), at the t moment of the simulation, put Ot+1 Input the action network to obtain the control quantity u t ; Execute u t , Calculate the reward r t , At this time, the environmental state changes to O t+1 ;

[0099] Step (3.7), Store (O t+1 , u t , r t , O t+1 ) as a sample into the experience replay pool R;

[0100] Step (3.8), The training loop starts;

[0101] Step (3.9), Sample N samples from R, calculate the loss function and update the weight parameters:

[0102]

[0103] Step (3.10), Execute step (3.9) until the training loop ends;

[0104] Step (3.11), Execute steps (3.5) to (3.10) until the simulation ends;

[0105] Step (3.12), Execute steps (3.4) to (3.11) until the episode loop ends;

[0106] Appendix Figure 4 shows the change of the reward function during the process of training the above agent using the Soft Actor-Critic algorithm. It can be seen that after 2000 episodes of training, the reward value quickly converges to a relatively high level.

[0107] (4) After the training is completed, fix the weight parameters of the action network and deploy it to the electronic controller. Its online application method is that within each control cycle, the electronic controller receives the observed variables from the compressor in real time, inputs them into the trained action network, and sets the standard deviation σ of the control quantity output by the action network to 0. Therefore, the mean value of the control quantity output by the action network is directly used as the final control signal and output to the compressor connection valve.

[0108] To verify the effectiveness of the compressor active surge control method and system proposed by the present invention, based on the compressor mathematical model of this embodiment, compare the control method of the present invention with the sliding mode control method and the fuzzy backstepping control method, and the results are as Figure 5As shown. It can be seen that when there are external disturbances and model uncertainties, the compressor pressure coefficient surges during the process of the sliding mode control method and the fuzzy backstepping control method tracking the given reference command, resulting in the pressure coefficient being unable to track the given command and experiencing obvious fluctuations, with large dynamic and steady-state errors throughout the process, indicating that the compressor is in an unstable operating state; in contrast, the compressor active surge control method based on deep reinforcement learning proposed by the present invention can ensure that the compressor tracks the given pressure coefficient reference command throughout the process, ensuring the stable operation of the compressor.

[0109] As described above, only the preferred specific embodiments of the present invention are provided, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, making equivalent substitutions or changes, should be covered by the protection scope of the present invention.

Claims

1. A compressor active surge control method based on deep reinforcement learning, characterized in that Including the following steps: (1) Identify the flow coefficient - pressure coefficient characteristic terms of the compressor using the real physical characteristic data of the compressor, and then establish a compressor mathematical model with an actuator according to the identified characteristic terms; among them, the characteristic term identification method is to dimensionlessize the pressure ratio π and the flow rate m in the real physical characteristic data of the compressor to obtain the pressure coefficient and the flow coefficient, and then use the cubic surface equation to perform data fitting by the least squares method, and the expression is: where m is the physical flow rate of the compressor; π is the physical pressure ratio of the compressor; φ is the flow coefficient; ψ is the pressure coefficient; n is the relative percentage speed of the compressor; ρ is the gas density inside the compressor; A c is the equivalent cross-sectional area of the internal flow passage of the compressor; U is the rim line speed at the mean diameter of the compressor rotor; p0 is the ambient pressure; ψ(φ, n) is the compressor characteristic term; a0, a1, b0, b1, c0, c1 are fitting coefficients; (2) Establish a deep reinforcement learning agent simulation training environment for the active surge control task of the compressor; (3) Train the agent using the soft actor - critic algorithm; (4) Deploy the action network of the trained agent to the electronic controller for online application.

2. The active surge control method for a compressor based on deep reinforcement learning according to claim 1, characterized in that The establishment of the deep reinforcement learning agent simulation training environment for the active surge control task of the compressor in step (2) includes: Step (2.1), given the pressure coefficient reference command p of the compressor ref As the target of the active surge control of the compressor, the pressure coefficient reference command is used to ensure that the compressor pressure ratio smoothly transitions to the low flow region outside the surge boundary, and is given by the following formula: where τ is the inertia coefficient; C is the final value of the pressure coefficient reference command; Step (2.2), select the observable quantity O as the pressure coefficient reference command p within k historical control cycles and the current control cycle ref , the flow coefficient φ of the compressor model, the pressure coefficient ψ of the compressor model, the control error e = p ref -ψ and its integral eint and derivative edot, the observable quantity O at time t t is expressed as a multivariate time series matrix of dimension (k + 1, 6): Step (2.3), design the action network and the evaluation network of the agent; The action network π of the agent θ includes an input layer, a fully connected layer, an LSTM layer, and a relu activation function layer, and then outputs through branch network 1 and branch network 2 respectively; branch network 1 consists of a fully connected layer, a relu activation function, a fully connected layer, and output layer 1 in sequence; branch network 2 consists of a fully connected layer, a relu activation function, a fully connected layer, a softplus activation function layer, and output layer 2 in sequence; the input parameter of the action network is the observation O, and output layer 1 is the mean value of the control quantity Output layer 2 is the standard deviation σ of the control quantity; The agent includes two evaluation networks with exactly the same structure, namely evaluation network and evaluation network Each evaluation network is formed by merging branch network 3 and branch network 4. Branch network 3 consists of an input layer 1, a fully connected layer, a relu activation function layer, and a fully connected layer in sequence; branch network 4 consists of an input layer 2 and a fully connected layer in sequence; after the outputs of branch network 3 and branch network 4 are concatenated into a high-dimensional vector by a concat layer, they pass through an LSTM recurrent neural network layer, a relu activation function, a fully connected neural network layer, and an output layer in sequence; the input of branch network 3 of the evaluation network is the observation O, and the input of branch network 4 is the control quantity u at the previous moment t-1 ; the output of the evaluation network is the expectation of the reward that can be obtained from the current observation and control quantity. Step (2.4), design the reward function r according to the observed quantity and the control quantity, specifically as follows: r＝r1+r2+r3 。 3. The active surge control method for a compressor based on deep reinforcement learning according to claim 1, wherein After the training in step (4) is completed, fix the weight parameters of the action network and deploy them to the electronic controller. In the online application mode, within each control cycle, the electronic controller receives the observed variables from the compressor in real time and inputs them into the trained action network; given that the standard deviation σ of the control variable output by the action network is 0, the mean value of the control variable output by the action network is directly used as the control variable u and output to the compressor actuator to control the compressor to stably track the pressure coefficient reference command.

4. The active surge control method for a compressor based on deep reinforcement learning according to claim 3, characterized in that The actuator is a throttle valve, and the compressor mathematical model with the actuator is: Among them, u is the control quantity closely connected to the valve actuator and is the input of the compressor mathematical model; φ is the flow coefficient output by the compressor mathematical model; ψ is the pressure coefficient output by the compressor mathematical model; B is the characteristic parameter; l c is the equivalent length of the compressor; γ T is the throttle valve opening; d φ is the disturbance and uncertainty term of the flow coefficient; d ψ is the disturbance and uncertainty term of the pressure coefficient.

Images