A method and system for controlling the start-up of a liquid rocket engine

Abstract

The application discloses a kind of liquid rocket engine starting process control method and system.Method includes: S1, rocket engine model is established;S2, define state space, action space and reward function in the starting process of rocket engine model;S3, the update including 10 iterations of TD3 algorithm is improved, and using annealing learning rate is improved, and form improved IM-TD3 algorithm;S4, using the IM-TD3 algorithm in step S3 is designed, trained and evaluated RL controller, state space in step S2 is used as the input of RL controller, for liquid rocket engine starting process is controlled.The application does not need to design complex control logic, and through designing suitable reward function, complex target can be realized, and compared with TD3 algorithm, the stability and convergence of model training are better in the control problem of rocket engine.

Description

Technical Field

[0001] This invention relates to the fields of liquid rocket engines and reinforcement learning algorithm technology, and more specifically, to a method and system for controlling the start-up process of a liquid rocket engine. Background Technology

[0002] On March 14, 2024, SpaceX's Starship heavy-lift reusable launch vehicle successfully entered orbit on its third launch after two previous failures, propelling space technology to new heights. The ignition process of a liquid rocket engine is an extremely complex and critical stage, involving the precise coordination and control of multiple systems and subsystems. Traditional control methods rely on experience-designed control strategies and fixed parameters. While this ensures engine ignition performance to a certain extent, it often lacks adaptability and struggles to cope with changing external environments and internal conditions.

[0003] With the development of artificial intelligence technology, reinforcement learning, as an algorithm that learns optimal strategies through interaction with the environment, provides a new solution for the control of dynamic and complex systems. Reinforcement learning optimizes control strategies through continuous trial and error, enabling the system to find the most efficient operating mode in unknown and changing environments. Reinforcement learning-based control methods have already undergone some preliminary research and application in the aerospace field, achieving promising results. Therefore, reinforcement learning has emerged as a new approach and technical method for the control of liquid rocket engines. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for controlling the start-up process of a liquid rocket engine, so as to overcome the defects of the prior art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for controlling the start-up process of a liquid rocket engine includes the following steps:

[0007] S1. Establish a rocket engine model;

[0008] S2. Define the state space, action space, and reward function during the rocket engine model startup process;

[0009] S3. The TD3 algorithm is improved by updating it through 10 iterations and using an annealing learning rate, resulting in the improved IM-TD3 algorithm.

[0010] S4. Use the IM-TD3 algorithm from step S3 to design, train, and evaluate the RL controller, and use the state space from step S2 as the input to the RL controller to control the start-up process of the liquid rocket engine.

[0011] Furthermore, in step S1, a rocket engine model is established using simulation software or a programming language, which allows the variables to be analyzed to be output.

[0012] Furthermore, the state space of step S2 includes turbine speed, combustion chamber pressure, air-fuel ratio and valve opening, the action space includes valves controlled during startup, and the reward function includes the target value for reaching steady state after successful startup, factors that cause engine damage or startup failure, and factors that affect engine performance.

[0013] Furthermore, the observation space S is defined by the following formula:

[0014] S = [P] G ,P C ,F,n t ,n fpp ,MR GG Pos VGO Pos VGF Pos VCF ]

[0015] In the formula, P G ,P C ,F,n t ,n fpp ,MR GG These are the gas generator pressure, main combustion chamber pressure, thrust, main turbine speed, fuel pre-compression pump speed, and gas generator mixture ratio, Pos. VGO Pos VGF Pos VCF The opening degree of the valve being controlled;

[0016] The formula for the action space A is defined as follows:

[0017] A = [Pos] VGO Pos VGF Pos VCF ]

[0018] The formula for the reward function is defined as follows:

[0019] Reward = r1 + r2 + r3 + r4 + r5

[0020] In the formula, ε i ∈[P G ,P C ,F,n t ,n fpp Rewards for values ​​that are close to the target value;

[0021] r2 = 1 - clip(FF)ref / F ref |,1);

[0022]

[0023] Act i ∈[Pos VGO Pos VGF Pos VCF ] represent the opening degree of the three valves respectively, and S represents the change in valve position between two time steps;

[0024] AT indicates the valve opening time.

[0025] Furthermore, the improved IM-TD3 algorithm in step S3 specifically includes the following steps:

[0026] S30. Initialize the evaluation network Qθ1, Qθ2 and the actor network πφ, and randomly assign values ​​to the parameters θ1, θ2 and φ;

[0027] S31. Initialize the target network: θ1′←θ1, θ2′←θ2, φ′←φ;

[0028] S32. Initialize the replay buffer B and the learning rate scheduler;

[0029] S33. For t=1 to T, perform 10 iterations of training and updating.

[0030] Furthermore, the 10 iterations of training and updating performed in step S33 specifically include:

[0031] S330, Sample the transition (s,a,r,s′,d) from the playback buffer;

[0032] S331. Disable gradient calculation for target update:

[0033] Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c)

[0034] Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c)

[0035] Calculate the target Qtarget = r + (1-d)·γ·Q′

[0036] S332. Update the evaluation network using MSE loss: MSE(Qθ(s,a),Qtarget);

[0037] S333. If i modpolicy_freq=0, update the actor network by maximizing the Q value of the evaluation network and softly update the target network θi′ and φ′.

[0038] S334. Use the scheduler to adjust the learning rate;

[0039] Among them, Q θ1 Q θ2 : Evaluation network parameterized by parameters θ1 and θ2;

[0040] π φ Actor network parameterized by parameter φ;

[0041] θ1′,θ2′,φ′ represent the target network of the evaluation and actor networks;

[0042] B represents the playback buffer used to store the transferred tuples;

[0043] s,a,r,s′,d represent the state, action, reward, next state, and completion flag sampled from the replay buffer;

[0044] a′ represents the target action calculated using the target actor network and noise clipping;

[0045] Q′ represents the target Q value calculated using the target evaluation network;

[0046] Qtarget represents the target for updating the Q value;

[0047] γ represents the discount factor for future rewards;

[0048] σ,c represents the parameters for noise generation and shearing in the action space;

[0049] MSE represents the mean squared error loss used to update the evaluation network;

[0050] policy_freq represents the frequency of policy updates relative to evaluation updates.

[0051] Furthermore, in step S4, an RL controller based on the IM-TD3 algorithm is implemented using Python code based on the Matlab-Simulink simulation platform.

[0052] The present invention also provides a system for implementing the above-described liquid rocket engine start-up process control method, comprising:

[0053] The modeling module is used to create rocket engine models;

[0054] The parameter definition module is used to define the state space, action space, and reward function during the rocket engine model startup process;

[0055] The algorithm improvement module is used to update the TD3 algorithm through 10 iterations and improve it using the annealing learning rate, resulting in the improved IM-TD3 algorithm.

[0056] The RL controller design module is used to design, train, and evaluate the RL controller using the IM-TD3 algorithm in step S3. The state space in step S2 is used as the input of the RL controller to control the start-up process of the liquid rocket engine.

[0057] The modeling module, parameter definition module, algorithm improvement module, and RL controller design module are connected in sequence.

[0058] Compared with existing technologies, the advantages of this invention are as follows: This invention determines the state space, action space, and reward function based on an established engine model, and uses the IM-TD3 algorithm to design, train, and evaluate an RL controller for controlling the rocket engine start-up process. This invention achieves intelligent control of the rocket engine. Compared with traditional open-loop and closed-loop control methods, this invention does not require extensive ground testing experience or the design of complex control logic. Complex objectives can be achieved by designing a suitable reward function. Furthermore, compared with the TD3 algorithm, this method exhibits better model training stability and convergence in rocket engine control problems. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart of the liquid rocket engine start-up process control method of the present invention.

[0061] Figure 2 This is a diagram of a gas generator afterburner cycle engine system.

[0062] Figure 3 This is an animation diagram of an open-loop control valve based on engineering experience.

[0063] Figure 4 This is a diagram showing the change in thrust during startup based on open-loop control.

[0064] Figure 5 This is a diagram showing the change in the gas generator mixture ratio during the open-loop control startup process.

[0065] Figure 6 This is a valve animation based on RL control.

[0066] Figure 7 This is a diagram showing the change in thrust during the start-up process based on RL control.

[0067] Figure 8 This is a diagram showing the change in the gas generator mixture ratio during the RL-controlled start-up process.

[0068] Figure 9 This is a schematic diagram of the liquid rocket engine start-up process control system of the present invention. Detailed Implementation

[0069] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0070] See Figure 1 and Figure 2 As shown in the figure, this embodiment discloses a control method for the start-up process of a liquid rocket engine, including the following steps:

[0071] Step S1: Establish a rocket engine model.

[0072] Specifically, commercial simulation software (such as Matlab-Simulink, Amesim, and EcosimPro) or programming languages ​​(such as Python, C, or C++) can be used to build the engine model to be analyzed. Furthermore, the variables to be analyzed can be output.

[0073] Step S2: Define the state space, action space, and reward function during the rocket engine model startup process.

[0074] Step S3: The TD3 algorithm is updated with 10 iterations and improved by using an annealing learning rate to form the improved IM-TD3 algorithm.

[0075] The annealing learning rate in this embodiment is a method for dynamically adjusting the learning rate, used to gradually reduce the learning rate during deep learning model training. A larger learning rate is used in the early stages of training so that the model can quickly converge to a near-optimal solution. Then, the learning rate is gradually reduced in the later stages of training to fine-tune the model parameters, avoid jumping out of local optima, and reduce oscillations.

[0076] Step S4: Design, train and evaluate the RL controller using the IM-TD3 algorithm from step S3. Use the state space from step S2 as the input to the RL controller to control the start-up process of the liquid rocket engine.

[0077] Specifically, based on the engine's characteristics, the state space, action space, and reward function are selected for RL controller training. The state space mainly includes turbine speed, combustion chamber pressure, air-fuel mixture ratio, and valve opening. Including valve opening as part of the state space allows the RL controller to adjust the valves based on the relationship between the current valve state and the target value, effectively suppressing valve back-and-forth movement that could lead to system oscillations and valve damage. The action space mainly includes the valves controlled during startup. The reward function mainly includes the target value reached after successful startup in a steady state. Hard constraints are factors that could cause engine damage or startup failure, such as deviations in the air-fuel mixture ratio leading to overheating. Soft constraints are factors affecting engine performance, such as combustion chamber pressure and turbine speed.

[0078] The formula for the observation space S is defined as follows:

[0079] S = [P] G ,P C ,F,n t ,n fpp ,MR GG Pos VGO Pos VGF Pos VCF (1)

[0080] In the formula, P G ,P C ,F,n t ,n fpp ,MR GG These are the gas generator pressure, main combustion chamber pressure, thrust, main turbine speed, fuel pre-compression pump speed, and gas generator mixture ratio, Pos. VGO Pos VGF Pos VCF The observation space is normalized using steady-state reference values ​​to determine the opening degree of the controlled valves. Therefore, the proposed observation space and method for this engine are not limited to the simulation environment described in this application. In other engine systems or environments, some parameters, such as engine turbine efficiency, cannot be directly measured. The agent's action space A consists of the opening degrees of three valves.

[0081] The formula for the action space A is defined as follows:

[0082] A = [Pos] VGO Pos VGF Pos VCF (2)

[0083] Within each time step, the RLA agent (the agent in RL training, which is the subject that executes decisions and actions in the reinforcement learning framework, continuously optimizing its behavioral strategy through trial and error and feedback) receives environmental observations and sends control signals to the engine's control valves. The RLA agent interacts with the environment at a frequency of 25 Hz.

[0084] The formula for the reward function is defined as follows:

[0085] Reward = r1 + r2 + r3 + r4 + r5(3)

[0086] In the formula

[0087]

[0088] ε i ∈[P G ,P C ,F,n t ,n fpp For rewards that are close to the target value, each reward component in this item has its maximum value reduced by 0.2 to improve the cumulative reward during training balance startup and steady state.

[0089] r2 = 1 - clip(|FF) ref / F ref |,1) (5)

[0090] Target thrust is one of the important indicators of an engine system, directly reflecting the performance and load capacity of the engine system.

[0091]

[0092] If the gas generator's mixing ratio is lower than the preset mixing ratio, it will cause the gas generator temperature to rise, resulting in high-temperature gas impacting the gas turbine, which may damage the gas turbine.

[0093]

[0094] Act i ∈[Pos VGO Pos VGF Pos VCF ] represents the opening degree of the three valves respectively, and S represents the change in valve position between two time steps before and after the valve. It is used to punish the reciprocating action of the valve. Through this item, the oscillation caused by the agent frequently actuating the valve during the system startup process can be suppressed.

[0095]

[0096] AT represents the valve opening time. This reward is designed based on the characteristics of the gas-fired combustion cycle engine system and engine testing experience, ensuring that the oxidizer valve opens earlier than the fuel valve, and the gas generator fuel valve opens earlier than the main combustion chamber fuel valve. This conditionally triggered reward aims to encourage the agent to use the engine's "discontinuous" states during training, ensuring that VGO, VGF, and VCF open in the correct sequence.

[0097] like Figure 3 , Figure 4 and Figure 5 As shown, the improved IM-TD3 algorithm in step S3 specifically includes the following steps:

[0098] Step S30: Initialize the evaluation network Qθ1, Qθ2 and the actor network πφ, and randomly assign values ​​to the parameters θ1, θ2 and φ;

[0099] Step S31: Initialize the target network θ1′←θ1, θ2′←θ2, φ′←φ;

[0100] Step S32: Initialize the replay buffer B and the learning rate scheduler;

[0101] Step S33: For t=1 to T, perform 10 iterations of training and updating.

[0102] Specifically, the 10 iterations of training and updating in step S33 include:

[0103] Step S330: Sample the transition (s,a,r,s′,d) from the playback buffer;

[0104] Step S331: Disable gradient calculation for target update:

[0105] Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c)

[0106] Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c)

[0107] Calculate the target Qtarget = r + (1-d)·γ·Q′

[0108] Step S332: Update the evaluation network using MSE loss: MSE(Qθ(s,a),Qtarget);

[0109] Step S333: If i modpolicy_freq = 0, update the actor network by maximizing the Q value of the evaluation network, and softly update the target network θi′ and φ′.

[0110] Step S334: Adjust the learning rate using the scheduler;

[0111] Among them, Q θ1 Q θ2 : Evaluation network parameterized by parameters θ1 and θ2;

[0112] π φ Actor network parameterized by parameter φ;

[0113] θ1′,θ2′,φ′ represent the target network of the evaluation and actor networks;

[0114] B represents the playback buffer used to store the transferred tuples;

[0115] s,a,r,s′,d represent the state, action, reward, next state, and completion flag sampled from the replay buffer;

[0116] a′ represents the target action calculated using the target actor network and noise clipping;

[0117] Q′ represents the target Q value calculated using the target evaluation network;

[0118] Qtarget represents the target for updating the Q value;

[0119] γ represents the discount factor for future rewards;

[0120] σ,c represents the parameters for noise generation and shearing in the action space;

[0121] MSE represents the mean squared error loss used to update the evaluation network;

[0122] policy_freq represents the frequency of policy updates relative to evaluation updates.

[0123] Specifically, in step S4, an RL controller based on the IM-TD3 algorithm is implemented using Python code on the Matlab-Simulink simulation platform. The specific steps include: (1) setting up a Matlab-Simulink simulation environment, creating and configuring the simulation model using Matlab-Simulink, including defining the system's state, input, output, and dynamic equations; (2) establishing a communication interface, using the MATLAB Engine API for Python to call Matlab functions and the Simulink model; and (3) implementing the IM-TD3 algorithm in Python, including defining the Actor-Critic network, loss function, and optimization steps. The defined state space is used as the input to the RL controller. The RL agent, based on the trained strategy, controls the valve's operation during engine startup to maximize the agent's reward, as shown below. Figure 6 , Figure 7 and Figure 8 As shown.

[0124] See Figure 9 As shown, the present invention also provides a system for implementing the above-described liquid rocket engine start-up process control method, comprising: a modeling module 100 for establishing a rocket engine model; a parameter definition module 200 for defining the state space, action space, and reward function of the rocket engine model during the start-up process; an algorithm improvement module 300 for performing 10 iterations of updates to the TD3 algorithm and improving it using an annealing learning rate to form an improved IM-TD3 algorithm; and an RL controller design module 400 for designing, training, and evaluating an RL controller using the IM-TD3 algorithm in step S3, using the state space in step S2 as the input of the RL controller for controlling the liquid rocket engine start-up process; the modeling module 100, parameter definition module 200, algorithm improvement module 300, and RL controller design module 400 are connected sequentially.

[0125] This invention, based on an established engine model, defines the state space, action space, and reward function. Using the IM-TD3 algorithm, it designs, trains, and evaluates an RL controller for controlling the rocket engine's start-up process. This invention achieves intelligent control of rocket engines. Compared to traditional open-loop and closed-loop control methods, this invention does not require extensive ground testing experience or complex control logic design. By designing a suitable reward function, complex objectives can be achieved. Furthermore, compared to the TD3 algorithm, this method exhibits better model training stability and convergence in rocket engine control problems.

[0126] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, the patent owner may make various modifications or alterations within the scope of the appended claims, as long as they do not exceed the protection scope described in the claims of the present invention, they shall be within the protection scope of the present invention.

Claims

1. A control method for the starting process of a liquid rocket engine, characterized in that, Includes the following steps: S1. Establish a rocket engine model; S2. Define the state space, action space, and reward function during the rocket engine model startup process; S3. The TD3 algorithm is improved by updating it through 10 iterations and using an annealing learning rate, resulting in the improved IM-TD3 algorithm. S4. Use the IM-TD3 algorithm in step S3 to design, train and evaluate the RL controller, and use the state space in step S2 as the input of the RL controller to control the start-up process of the liquid rocket engine. The state space of step S2 includes turbine speed, combustion chamber pressure, air-fuel ratio and valve opening; the action space includes valves controlled during startup; and the reward function includes the target value for reaching steady state after successful startup, factors that cause engine damage or startup failure, and factors that affect engine performance. The formula for the state space S is defined as follows: S=[P G ,P C ,F,n t ,n fpp ,MR GG ,Pos VGO ,Pos VGF ,Pos VCF ] In the formula, P G ,P C ,F,n t ,n fpp ,MR GG These are the gas generator pressure, main combustion chamber pressure, thrust, main turbine speed, fuel pre-compression pump speed, and gas generator mixture ratio, Pos. VGO Pos VGF Pos VCF The opening degree of the valve being controlled; The formula for the action space A is defined as follows: A=[Pos VGO ,Pos VGF ,Pos VCF ] The formula for the reward function is defined as follows: Reward = r1 + r2 + r3 + r4 + r5 In the formula, ε i ∈[P G ,P C ,F,n t ,n fpp Rewards for values ​​that are close to the target value; r2=1-clip(|F-F ref / F ref |,1); Act i ∈[Pos VGO Pos VGF Pos VCF ] represent the opening degree of the three valves respectively, and S represents the change in valve position between two time steps; AT indicates the valve opening time.

2. The liquid rocket engine start-up process control method according to claim 1, characterized in that, Step S1 involves using simulation software or a programming language to establish a rocket engine model, which allows the variables to be analyzed to be output.

3. The liquid rocket engine start-up process control method according to claim 1, characterized in that, The improved IM-TD3 algorithm in step S3 specifically includes the following steps: S30. Initialize the evaluation network Qθ1, Qθ2 and the actor network πφ, and randomly assign values ​​to the parameters θ1, θ2 and φ; S31. Initialize the target network: θ1′←θ1, θ2′←θ2, φ′←φ; S32. Initialize the replay buffer B and the learning rate scheduler; S33. For t=1 to T, perform 10 iterations of training and updating.

4. The liquid rocket engine start-up process control method according to claim 3, characterized in that, The 10 iterations of training and updating performed in step S33 specifically include: S330, Sample the transition (s,a,r,s′,d) from the playback buffer; S331. Disable gradient calculation for target update: Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c) Calculate the target action a′=πφ′(s′)+clip(N(0,σ),-c,c) Calculate the target Qtarget = r + (1-d)·γ·Q′ S332. Update the evaluation network using MSE loss: MSE(Qθ(s,a),Qtarget); S333. If i modpolicy_freq=0, update the actor network by maximizing the Q value of the evaluation network and softly update the target network θi′ and φ′. S334. Use the scheduler to adjust the learning rate; Among them, Q θ1 Q θ2 : Evaluation network parameterized by parameters θ1 and θ2; π φ Actor network parameterized by parameter φ; θ1′,θ2′,φ′ represent the target network of the evaluation and actor networks; B represents the playback buffer used to store the transferred tuples; s,a,r,s′,d represent the state, action, reward, next state, and completion flag sampled from the replay buffer; a′ represents the target action calculated using the target actor network and noise clipping; Q′ represents the target Q value calculated using the target evaluation network; Qtarget represents the target for updating the Q value; γ represents the discount factor for future rewards; σ,c represents the parameters for noise generation and shearing in the action space; MSE represents the mean squared error loss used to update the evaluation network; policy_freq represents the frequency of policy updates relative to evaluation updates.

5. The liquid rocket engine start-up process control method according to claim 1, characterized in that, In step S4, an RL controller based on the IM-TD3 algorithm is implemented using Python code on the Matlab-simulink simulation platform.

6. A system for implementing the liquid rocket engine start-up process control method according to any one of claims 1-5, characterized in that, include: The modeling module is used to create rocket engine models; The parameter definition module is used to define the state space, action space, and reward function during the rocket engine model startup process; The algorithm improvement module is used to update the TD3 algorithm through 10 iterations and improve it using the annealing learning rate, resulting in the improved IM-TD3 algorithm. The RL controller design module is used to design, train, and evaluate the RL controller using the IM-TD3 algorithm in step S3. The state space in step S2 is used as the input of the RL controller to control the start-up process of the liquid rocket engine. The modeling module, parameter definition module, algorithm improvement module, and RL controller design module are connected in sequence.

Images