Liquid rocket engine starting process control method and system

Abstract

The invention discloses a liquid rocket engine starting process control method and system. The method comprises the following steps: S1, establishing a rocket engine model; s2, defining a state space, an action space and a reward function in the starting process of the rocket engine model; s3, the TD3 algorithm is subjected to updating including 10 times of iteration and improvement using an annealing learning rate, and an improved IM-TD3 algorithm is formed; and S4, designing, training and evaluating an RL controller by using the IM-TD3 algorithm in the step S3, and taking the state space in the step S2 as the input of the RL controller for controlling the starting process of the liquid rocket engine. According to the method, complex control logic does not need to be designed, a complex target can be achieved by designing a proper award function, and compared with a TD3 algorithm, the method has the advantages that the stability and convergence of model training are better in the control problem of the rocket engine.

Description

Technical Field

[0001] The present invention relates to the field of liquid rocket engines and the field of reinforcement learning algorithm technology, and more specifically, to a liquid rocket engine starting process control method and system. Background Art

[0002] On March 14, 2024, SpaceX's heavy-duty reusable launch vehicle Starship successfully entered orbit in its third launch after two failed launches, pushing aerospace technology to a new level of development. The starting process of a liquid rocket engine is an extremely complex and critical stage, involving precise coordination and control of multiple systems and subsystems. Traditional control methods rely on empirically designed control strategies and fixed parameters, which, although they guarantee the starting performance of the engine to a certain extent, are often difficult to cope with the changing external environment and internal state changes due to lack of adaptability.

[0003] With the development of artificial intelligence technology, reinforcement learning, as an algorithm that can learn the optimal strategy through interaction with the environment, provides a new solution for the control of dynamic and complex systems. Reinforcement learning optimizes the control strategy through continuous trial and error, enabling the system to find the most optimized operating mode in an unknown and changing environment. Control methods based on reinforcement learning have been studied and applied in the control of the aerospace field, and have achieved good results. Therefore, reinforcement learning has become a new idea and technical approach for liquid rocket engine control. Summary of the invention

[0004] The purpose of the present invention is to provide a liquid rocket engine starting process control method and system to overcome the defects of the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A liquid rocket engine starting process control method comprises the following steps:

[0007] S1. Build a rocket engine model;

[0008] S2. Define the state space, action space and reward function during the startup of the rocket engine model;

[0009] S3, updating the TD3 algorithm for 10 iterations and improving the annealing learning rate to form an improved IM-TD3 algorithm;

[0010] S4. Use the IM-TD3 algorithm in step S3 to design, train and evaluate the RL controller, and use the state space in step S2 as the input of the RL controller to control the starting process of the liquid rocket engine.

[0011] Furthermore, the step S1 uses simulation software or programming language to establish a rocket engine model, and the rocket engine model can output the variables that need to be analyzed.

[0012] Furthermore, the state space of step S2 includes turbine speed, combustion chamber pressure, mixture ratio and valve opening, the action space includes valves controlled during the starting process, and the reward function includes the target value of steady state achieved after successful starting, factors causing engine damage or starting failure, and factors affecting engine performance.

[0013] Furthermore, the formula of the observation space S is defined as:

[0014] S=[P G ,P C ,F,n t ,n fpp ,MR GG ,Pos VGO ,Pos VGF ,Pos VCF ]

[0015] Where P G ,P C ,F,n t ,n fpp ,MR GG They are gas generator pressure, main combustion chamber pressure, thrust size, main turbine speed, fuel pre-pressure pump speed, gas generator mixing ratio, Pos VGO ,Pos VGF ,Pos VCF is the opening of the controlled valve;

[0016] The formula of the action space A is defined as:

[0017] A=[Pos VGO ,Pos VGF ,Pos VCF ]

[0018] The formula of the reward function is defined as:

[0019] Reward=r1+r2+r3+r4+r5

[0020] In the formula, ε i ∈[P G ,P C ,F,n t ,n fpp ] Reward for being close to the target value;

[0021] r2=1-clip(FFref / F ref |,1);

[0022]

[0023] Act i ∈[Pos VGO ,Pos VGF ,Pos VCF ] represent the opening of the three valves respectively, and S represents the change of the valve position between two time steps before and after the valve;

[0024] AT represents the opening time of the valve.

[0025] Furthermore, the improved IM-TD3 algorithm in step S3 specifically includes the following steps:

[0026] S30, initialize the evaluation network Qθ1, Qθ2 and the actor network πφ, and randomly assign parameters θ1, θ2, φ;

[0027] S31, initialize the target network θ1′←θ1, θ2′←θ2, φ′←φ;

[0028] S32, initializing playback buffer B and learning rate scheduler;

[0029] S33. For t=1 to T, perform 10 iterations of training updates.

[0030] Furthermore, performing 10 iterations of training updates in step S33 specifically includes:

[0031] S330, sampling from the playback buffer to obtain a transfer (s, a, r, s′, d);

[0032] S331. Disable gradient calculation for target update:

[0033] Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c)

[0034] Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c)

[0035] Calculate the target Qtarget = r + (1-d) · γ · Q′

[0036] S332. Use MSE loss to update the evaluation network: MSE(Qθ(s,a),Qtarget);

[0037] S333, if i modpolicy_freq = 0, update the actor network by maximizing the Q value of the evaluation network, and soft-update the target networks θi′ and φ′;

[0038] S334, using a scheduler to adjust the learning rate;

[0039] Among them, Q θ1 ,Q θ2 : The evaluation network parameterized by parameters θ1 and θ2;

[0040] π φ : Actor network parameterized by parameter φ;

[0041] θ1′, θ2′, φ′ represent the target network of the evaluation and actor networks;

[0042] B represents the playback buffer for storing the transfer tuples;

[0043] s,a,r,s′,d represent the state, action, reward, next state and completion flag sampled from the replay buffer;

[0044] a′ represents the target action computed using the target actor network and noise clipping;

[0045] Q′ represents the target Q value calculated using the target evaluation network;

[0046] Qtarget represents the target of Q value update;

[0047] γ represents the discount factor for future rewards;

[0048] σ,c represents the parameters of noise generation and shearing in the action space;

[0049] MSE represents the mean square error loss used to update the evaluation network;

[0050] policy_freq represents the frequency of policy updates relative to evaluation updates.

[0051] Furthermore, in step S4, based on the Matlab-simulink simulation platform, python code is used to implement the RL controller based on the IM-TD3 algorithm.

[0052] The present invention also provides a system for implementing the above-mentioned liquid rocket engine starting process control method, comprising:

[0053] Modeling module, used to build rocket engine models;

[0054] Parameter definition module, used to define the state space, action space and reward function during the startup of the rocket engine model;

[0055] The algorithm improvement module is used to update the TD3 algorithm for 10 iterations and improve the annealing learning rate to form the improved IM-TD3 algorithm;

[0056] An RL controller design module, for designing, training and evaluating an RL controller using the IM-TD3 algorithm in step S3, using the state space in step S2 as an input of the RL controller for controlling the liquid rocket engine starting process;

[0057] The modeling module, parameter definition module, algorithm improvement module and RL controller design module are connected in sequence.

[0058] Compared with the prior art, the advantages of the present invention are: the present invention determines the state space, action space and reward function on the established engine model, uses the IM-TD3 algorithm, designs, trains and evaluates the RL controller for controlling the rocket engine starting process. The present invention realizes the intelligent control of the rocket engine. Compared with the traditional open-loop and closed-loop control methods, the present invention does not require a lot of ground test experience, does not require the design of complex control logic, and can achieve complex goals by designing a suitable reward function. In addition, compared with the TD3 algorithm, the stability and convergence of the model training of this method on the control problem of the rocket engine are better. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 It is a flow chart of the liquid rocket engine starting process control method of the present invention.

[0061] Figure 2 It is a system diagram of a gas generator secondary combustion cycle engine.

[0062] Figure 3 It is an open loop control valve operation diagram based on engineering experience.

[0063] Figure 4 This is a diagram showing the thrust changes during open-loop control starting.

[0064] Figure 5 It is based on the gas generator mixture ratio change diagram during the open loop control starting process.

[0065] Figure 6 This is the valve motion diagram based on RL control.

[0066] Figure 7 This is a diagram showing the thrust changes during the RL control starting process.

[0067] Figure 8 It is a diagram showing the change of the gas generator mixture ratio during the RL control starting process.

[0068] Fig. 9 It is a schematic diagram of the liquid rocket engine starting process control system of the present invention. DETAILED DESCRIPTION

[0069] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0070] See also Figure 1 and Figure 2 As shown, this embodiment discloses a liquid rocket engine starting process control method, comprising the following steps:

[0071] Step S1, establishing a rocket engine model.

[0072] Specifically, commercial simulation software (such as Matlab-Simulink, Amesim and EcosimPro, etc.) or programming languages ​​(such as Python, C or C++, etc.) can be used to establish the engine model to be analyzed, and the variables to be analyzed can be output.

[0073] Step S2: define the state space, action space and reward function during the startup of the rocket engine model.

[0074] Step S3: The TD3 algorithm is updated for 10 iterations and improved using an annealing learning rate to form an improved IM-TD3 algorithm.

[0075] The annealing learning rate of this embodiment is a method for dynamically adjusting the learning rate, which is used to gradually reduce the learning rate during deep learning model training. A larger learning rate is used in the early stages of training so that the model can quickly converge to a better solution, and then the learning rate is gradually reduced in the later stages of training to fine-tune the model parameters, avoid jumping out of the local optimal solution and reduce oscillation.

[0076] Step S4: Use the IM-TD3 algorithm in step S3 to design, train and evaluate the RL controller, and use the state space in step S2 as the input of the RL controller to control the starting process of the liquid rocket engine.

[0077] Specifically, according to the characteristics of the engine, the state space, action space and reward function for RL controller training are selected. The state space mainly includes turbine speed, combustion chamber pressure, mixture ratio and valve opening. The advantage of valve opening as part of the state space is that the RL controller can adjust the valve according to the relationship between the current valve state and the target value, which can effectively suppress the valve from moving back and forth, causing system oscillation and valve damage; the action space mainly includes the valve controlled during the starting process; the reward function mainly includes the target value to reach the steady state after the start is successful, hard constraints: factors that may cause engine damage or start failure, such as deviations in the combustion chamber mixture ratio, factors that lead to overheating, soft constraints: factors that affect engine performance, such as combustion chamber pressure, turbine speed, etc.

[0078] The formula of the observation space S is defined as:

[0079] S=[P G ,P C ,F,n t ,n fpp ,MR GG ,Pos VGO ,Pos VGF ,Pos VCF ](1)

[0080] Where P G ,P C ,F,n t ,n fpp ,MR GG They are gas generator pressure, main combustion chamber pressure, thrust size, main turbine speed, fuel pre-pressure pump speed, gas generator mixing ratio, Pos VGO ,Pos VGF ,Pos VCF The opening of the controlled valve is used to normalize the observation space with the steady-state reference value. Therefore, the proposed observation space and method of the engine are not limited to the simulation environment in this application. In other engine systems or environments, some parameters, such as engine turbine efficiency and other variables, cannot be directly measured. The action space A of the agent consists of the opening of three valves.

[0081] The formula of the action space A is defined as:

[0082] A=[Pos VGO ,Pos VGF ,Pos VCF ](2)

[0083] In each time step, the RLAgent receives environmental observations and sends control signals to the engine's control valves. The interaction frequency between the RLAgent and the environment is 25 Hz.

[0084] The formula of the reward function is defined as:

[0085] Reward=r1+r2+r3+r4+r5(3)

[0086] In the formula

[0087]

[0088] ε i ∈[P G ,P C ,F,n t ,n fpp ]For rewards close to the target value, each reward component in this term is clipped by a maximum value of 0.2 to improve the cumulative reward during training balance startup and steady state.

[0089] r2=1-clip(|FF ref / F ref |,1) (5)

[0090] Target thrust is one of the important indicators of the engine system, which directly reflects the performance and load capacity of the engine system.

[0091]

[0092] If the mixture ratio of the gas generator is lower than the preset mixture ratio, the temperature of the gas generator will increase, causing the high-temperature gas to impact the gas turbine, which may cause damage to the gas turbine.

[0093]

[0094] Act i ∈[Pos VGO ,Pos VGF ,Pos VCF ] represent the opening of the three valves respectively, S represents the change of the valve position between two time steps before and after the valve, which is used to punish the reciprocating action of the valve. Through this item, the oscillation caused by the frequent actuation of the valve by the agent during the system startup process can be suppressed.

[0095]

[0096] AT represents the valve opening time. This reward is designed based on the characteristics of the gas regenerative cycle engine system and the engine test experience to ensure that the oxidizer valve opens earlier than the fuel valve, and the gas generator fuel valve opens earlier than the main combustion chamber fuel valve. This conditional trigger reward is expected to make the agent use the "discontinuous" state of the engine during training to ensure that VGO, VGF, and VCF can be opened in the correct order.

[0097] like Figure 3 , Figure 4 and Figure 5 As shown, the improved IM-TD3 algorithm in step S3 specifically includes the following steps:

[0098] Step S30, initializing the evaluation network Qθ1, Qθ2 and the actor network πφ, and randomly assigning parameters θ1, θ2, φ;

[0099] Step S31, initialize the target network θ1′←θ1, θ2′←θ2, φ′←φ;

[0100] Step S32, initializing the playback buffer B and the learning rate scheduler;

[0101] Step S33: for t=1 to T, perform 10 iterations of training updates.

[0102] Specifically, the 10 iterations of training updates in step S33 specifically include:

[0103] Step S330, sampling from the playback buffer to obtain a transfer (s, a, r, s′, d);

[0104] Step S331, disable the gradient calculation of target update:

[0105] Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c)

[0106] Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c)

[0107] Calculate the target Qtarget = r + (1-d) · γ · Q′

[0108] Step S332, use MSE loss to update the evaluation network: MSE(Qθ(s,a),Qtarget);

[0109] Step S333: If i modpolicy_freq = 0, update the actor network and soft-update the target networks θi′ and φ′ by maximizing the Q value of the evaluation network;

[0110] Step S334, using a scheduler to adjust the learning rate;

[0111] Among them, Q θ1 ,Q θ2 : The evaluation network parameterized by parameters θ1 and θ2;

[0112] π φ : Actor network parameterized by parameter φ;

[0113] θ1′, θ2′, φ′ represent the target network of the evaluation and actor networks;

[0114] B represents the playback buffer for storing the transfer tuples;

[0115] s,a,r,s′,d represent the state, action, reward, next state and completion flag sampled from the replay buffer;

[0116] a′ represents the target action computed using the target actor network and noise clipping;

[0117] Q′ represents the target Q value calculated using the target evaluation network;

[0118] Qtarget represents the target of Q value update;

[0119] γ represents the discount factor for future rewards;

[0120] σ,c represents the parameters of noise generation and shearing in the action space;

[0121] MSE represents the mean square error loss used to update the evaluation network;

[0122] policy_freq represents the frequency of policy updates relative to evaluation updates.

[0123] Specifically, in step S4, based on the Matlab-simulink simulation platform, python code is used to implement the RL controller based on the IM-TD3 algorithm. The specific steps include: (1) Building a Matlab-Simulink simulation environment, using Matlab-Simulink to create and configure a simulation model, including defining the state, input, output and dynamic equations of the system; (2) Establishing a communication interface, using MATLAB EngineAPI forPython to call Matlab functions and Simulink models; (3) Implementing the IM-TD3 algorithm, implementing the IM-TD3 algorithm in Python, including the definition of the Actor-Critic network, loss function and optimization steps. The defined state space is used as the input of the RL controller. The RL agent controls the valve actuation during the engine startup process according to the trained strategy so that the Agent obtains the maximum reward, as shown in the following examples. Figure 6 , Figure 7 and Figure 8 shown.

[0124] See also Fig. 9 As shown, the present invention also provides a system for implementing the above-mentioned liquid rocket engine starting process control method, including: a modeling module 100, used to establish a rocket engine model; a parameter definition module 200, used to define the state space, action space and reward function in the rocket engine model startup process; an algorithm improvement module 300, used to update the TD3 algorithm including 10 iterations and improve it using an annealing learning rate to form an improved IM-TD3 algorithm; an RL controller design module 400, used to design, train and evaluate an RL controller using the IM-TD3 algorithm in step S3, and use the state space in step S2 as the input of the RL controller to control the liquid rocket engine starting process; the modeling module 100, the parameter definition module 200, the algorithm improvement module 300 and the RL controller design module 400 are connected in sequence.

[0125] The present invention determines the state space, action space and reward function on the established engine model, uses the IM-TD3 algorithm, designs, trains and evaluates the RL controller for controlling the rocket engine starting process. The present invention realizes the intelligent control of the rocket engine. Compared with the traditional open-loop and closed-loop control methods, the present invention does not require a lot of ground test experience, does not require the design of complex control logic, and can achieve complex goals by designing a suitable reward function. In addition, compared with the TD3 algorithm, the stability and convergence of the model training of this method on the control problem of the rocket engine are better.

[0126] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various variations or modifications within the scope of the appended claims. As long as they do not exceed the protection scope described in the claims of the present invention, they should be within the protection scope of the present invention.

Claims

1. A liquid rocket engine starting process control method, It is characterized in that The following steps are involved: S1. Build a rocket engine model; S2. Define the state space, action space and reward function during the startup of the rocket engine model; S3, updating the TD3 algorithm for 10 iterations and improving the annealing learning rate to form an improved IM-TD3 algorithm; S4. Use the IM-TD3 algorithm in step S3 to design, train and evaluate the RL controller, and use the state space in step S2 as the input of the RL controller to control the starting process of the liquid rocket engine.

2. The liquid rocket engine starting process control method according to claim 1, It is characterized in that The step S1 uses simulation software or programming language to establish a rocket engine model, and the rocket engine model can output the variables that need to be analyzed.

3. The liquid rocket engine starting process control method according to claim 1, It is characterized in that The state space of step S2 includes turbine speed, combustion chamber pressure, mixture ratio and valve opening, the action space includes valves controlled during the startup process, and the reward function includes the target value of steady state achieved after successful startup, factors causing engine damage or startup failure, and factors affecting engine performance.

4. The liquid rocket engine starting process control method according to claim 3, It is characterized in that The formula of the observation space S is defined as: S=[P G ,P C ,F,n t ,n fpp ,MR GG ,Pos VGO ,Pos VGF ,Pos VCF ] Where P G ,P C ,F,n t ,n fpp ,MR GG They are gas generator pressure, main combustion chamber pressure, thrust size, main turbine speed, fuel pre-pressure pump speed, gas generator mixing ratio, Pos VGO ,Pos VGF ,Pos VCF is the opening of the controlled valve; The formula of the action space A is defined as: A=[Pos VGO ,Pos VGF ,Pos VCF ] The formula of the reward function is defined as: Reward=r 1 +r 2 +r 3 +r 4 +r 5 In the formula, ε i ∈[P G ,P C ,F,n t ,n fpp ] Reward for being close to the target value; r 2 =1-clip(F-F ref / F ref |,1); Act i ∈[Pos VGO ,Pos VGF ,Pos VCF ] represent the opening of the three valves respectively, and S represents the change of the valve position between two time steps before and after the valve; AT represents the opening time of the valve.

5. The liquid rocket engine starting process control method according to claim 1, It is characterized in that The improved IM-TD3 algorithm in step S3 specifically includes the following steps: S30, initialize the evaluation network Qθ1, Qθ2 and the actor network πφ, and randomly assign parameters θ1, θ2, φ; S31, initialize the target network θ1′←θ1, θ2′←θ2, φ′←φ; S32, initializing playback buffer B and learning rate scheduler; S33. For t=1 to T, perform 10 iterations of training updates.

6. The liquid rocket engine starting process control method according to claim 5, It is characterized in that The 10 iterations of training updates in step S33 specifically include: S330, sampling from the playback buffer to obtain a transfer (s, a, r, s′, d); S331. Disable gradient calculation for target update: Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c) Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c) Calculate the target Qtarget = r + (1-d) · γ · Q′ S332. Use MSE loss to update the evaluation network: MSE(Qθ(s,a),Qtarget); S333, if i modpolicy_freq = 0, update the actor network by maximizing the Q value of the evaluation network, and soft-update the target networks θi′ and φ′; S334, using a scheduler to adjust the learning rate; Among them, Q θ1 ,Q θ2 : The evaluation network parameterized by parameters θ1 and θ2; π φ : A network of actors parameterized by parameter φ; θ1′, θ2′, φ′ represent the target network of the evaluation and actor networks; B represents the playback buffer for storing the transfer tuples; s,a,r,s′,d represent the state, action, reward, next state and completion flag sampled from the replay buffer; a′ represents the target action computed using the target actor network and noise clipping; Q′ represents the target Q value calculated using the target evaluation network; Qtarget represents the target of Q value update; γ represents the discount factor for future rewards; σ,c represents the parameters of noise generation and shearing in the action space; MSE represents the mean square error loss used to update the evaluation network; policy_freq represents the frequency of policy updates relative to evaluation updates.

7. The liquid rocket engine starting process control method according to claim 1, It is characterized in that In step S4, based on the Matlab-simulink simulation platform, a RL controller based on the IM-TD3 algorithm is implemented using python code.

8. A system for implementing the liquid rocket engine starting process control method according to any one of claims 1 to 7, It is characterized in that include: Modeling module, used to build rocket engine models; Parameter definition module, used to define the state space, action space and reward function during the startup of the rocket engine model; The algorithm improvement module is used to update the TD3 algorithm for 10 iterations and improve the annealing learning rate to form the improved IM-TD3 algorithm; An RL controller design module, for designing, training and evaluating an RL controller using the IM-TD3 algorithm in step S3, using the state space in step S2 as an input of the RL controller for controlling the liquid rocket engine starting process; The modeling module, parameter definition module, algorithm improvement module and RL controller design module are connected in sequence.

Images