A rocket landing control method based on multi-agent reinforcement learning

Abstract

The application discloses a rocket landing control method based on multi-agent reinforcement learning in the technical field of rocket landing control methods, and the rocket landing control method comprises the following steps: establishing a rocket engine model, collecting and outputting necessary state parameters in the working process of a liquid rocket engine; establishing a vertical landing simulation model of a carrier rocket, collecting and outputting necessary parameters in the vertical landing process of the carrier rocket; defining a state space, an action space and a reward function in the engine starting process; improving on the basis of the MADDPG algorithm to realize the LCO-MADDPG algorithm; and designing, training and evaluating the MARL model. The rocket landing control method realizes nonlinear control of the vertical landing process control of the carrier rocket by means of intelligent control methods without the need of designing complex control logic.

Description

Technical Field

[0001] This invention relates to the field of rocket landing control methods, specifically to a rocket landing control method based on multi-agent reinforcement learning. Background Technology

[0002] With the development of artificial intelligence technology, multi-agent reinforcement learning (MARL) is becoming a new research hotspot. By decomposing the control task into cooperative agents such as thrust regulation and state stabilization, MARL can process high-dimensional state spaces in parallel, and its distributed decision-making mechanism enables the system to handle nonlinear optimization problems. Therefore, multi-agent reinforcement learning has become a new approach and technical method for the vertical landing process control of launch vehicles.

[0003] Vertical landing technology for launch vehicles is one of the core technologies for building reusable space transportation systems. The high launch costs and resource waste associated with traditional expendable launch vehicles prompted the global aerospace community to explore recovery and reuse technologies starting in the 1990s. Vertical landing of launch vehicles, through precise control of the reentry trajectory of rocket stages, powered deceleration, and landing leg cushioning, allows for the complete recovery of the stage structure. According to SpaceX engineering data, the cost of a single launch can be reduced by 80% after 10 reuses of a reusable medium-lift launch vehicle. However, this technology involves interdisciplinary challenges such as high-speed reentry aerodynamic-thermodynamic coupling, transonic thrust vector control, and high-precision soft landing at the terminal stage. Its dynamic equations exhibit strong nonlinearity, time-varying characteristics, and multi-constraint coupling features, posing a severe challenge to traditional guidance, navigation, and control methods. Currently, vertical landing control methods for launch vehicles employ traditional open-loop and closed-loop control methods, requiring experience in complex environments, extensive ground testing, and the design of complex control logic. Furthermore, the stability and convergence of model training are relatively poor. Summary of the Invention

[0004] The purpose of this invention is to provide a rocket landing control method based on multi-agent reinforcement learning, which solves the technical problems of existing launch vehicle vertical landing control methods that use traditional open-loop and closed-loop control methods, require complex environments, extensive ground test experience, and complex control logic design, and suffer from poor stability and convergence of model training. Through intelligent control methods, nonlinear control of the launch vehicle vertical landing process can be achieved without designing complex control logic.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] This invention provides a rocket landing control method based on multi-agent reinforcement learning, comprising the following steps:

[0007] S1. Establish a rocket engine model, collect and output the state parameters necessary for the operation of the liquid rocket engine;

[0008] S2. Establish a simulation model for the vertical landing of the launch vehicle, and collect and output the parameters necessary for the vertical landing process of the launch vehicle;

[0009] S3. Define the state space, action space, and reward function during the engine start-up process;

[0010] S4. Based on the MADDPG algorithm, improve upon it to implement the LCO-MADDPG algorithm;

[0011] S5. Design, train, and evaluate MARL models.

[0012] Furthermore, S1 includes the following steps:

[0013] S11. Use commercial simulation software or programming languages ​​to build a model of the liquid rocket engine to be analyzed;

[0014] S12. Collect the necessary state parameters during the operation of the liquid rocket engine, so that the variables of the state parameters to be analyzed can be output.

[0015] Furthermore, S2 includes the following steps:

[0016] S21. Using the same commercial simulation software or programming language as used in step S11, build a simulation model of the vertical landing process of the launch vehicle.

[0017] S22. Collect the necessary parameters during the vertical descent of the launch vehicle and output the variables of the necessary parameters.

[0018] Furthermore, S3 includes the following steps:

[0019] S31. Define the state space during engine start-up;

[0020] The state space for each agent needs to be defined, and the state space is set as follows:

[0021] S agent =[MR GG ,P GG ,T GG ,MR CC ,P CC ,T CC ,F,H,V,a,Pos VGO Pos VGF Pos VCF ]

[0022] Among them, MR GG ,PGG ,T GG ,MR CC ,P CC ,T CC F represents the mixture ratio of the engine gas generator, the gas generator chamber pressure, the gas generator temperature, the mixture ratio of the thrust chamber, the thrust chamber pressure, the thrust chamber temperature, and the thrust magnitude, respectively; H, V, and a represent the altitude, velocity, and acceleration of the launch vehicle, respectively; and Pos represents the thrust magnitude. VGO Pos VGF Pos VCF These represent the oxidizer valve, fuel valve, and combustion chamber fuel valve of the gas generator, respectively, and the state space is normalized using steady-state reference values.

[0023] S32. Define the action space during engine starting;

[0024] The agent's action space A consists of the opening degrees of three valves.

[0025] A agent =[Pos VGO Pos VGF Pos VCF ]

[0026] Within each time step, MARLAgent receives environmental observations and sends control signals to the engine's control valves;

[0027] S33. Define the reward function during the engine start-up process.

[0028] Furthermore, S33 includes the following steps:

[0029] S331. The reward used for training MARL and evaluating the startup sequence consists of the following different parts:

[0030] Reward = R engine +R landing

[0031] R engine =r e1 +r e2 +r e3

[0032] The first reward is:

[0033]

[0034] Where, ε i ∈[MR GG ,P GG ,T GG ,MR CC ,P CC ,TCC For rewards that are close to the target value, each reward component in this item has its maximum value reduced by 0.2 to improve the cumulative reward during training balance startup and steady state.

[0035] The second reward is:

[0036]

[0037] This assumes that once the engine starts, it will not shut down. Therefore, once the engine generates thrust, the engine thrust should not be zero later to prevent the engine from shutting down during landing.

[0038] The third reward is:

[0039]

[0040] Among them, Act i ∈[Pos VGO Pos VGF Pos VCF ] represents the opening degree of the three valves of the engine, encouraging the attempt to open the valves before engine ignition; S represents the change in valve position between two steps before and after the valve, used to penalize the reciprocating action of the valve. This item can suppress the oscillation caused by the agent frequently actuating the valve during the vertical descent.

[0041] S332.

[0042] R landing =r l1 +r l2 +r l3

[0043] r l1 =0.2·exp(-abs(V) / 10)+0.2·(step max -step current ) / step max

[0044] Among them, step max It is the maximum number of steps per training round. current This is the current number of training steps, with exponential rewards used to guide the agent to decelerate the launch vehicle.

[0045] In addition, incorporating the step function into the reward function helps the agent complete the launch vehicle landing in a shorter time.

[0046] r l2 = 0.2·exp(H / 50) + 0.2·(step) max -step current) / step max

[0047] Similar to the formula above, its purpose is to enable the agent to land faster and better.

[0048]

[0049] The agent receives a huge reward after the launch vehicle successfully lands, which encourages the successful landing strategy and trains more successful landing strategies.

[0050] Furthermore, S4 includes the following steps:

[0051] S41. An improvement to the LCO-MADDPG algorithm was made, which involves 10 iterations of updates during training;

[0052] S42. An improvement to the LCO-MADDPG algorithm was made by using an annealing learning rate.

[0053] Furthermore, S5 includes the following steps:

[0054] S51. Combining the nonlinear characteristics of the engine system and the vertical landing phase, a MARL controller based on the LCO-MADDPG algorithm was designed, trained, and evaluated.

[0055] S52. Using the defined state space as the input to the controller, the MARL agent, based on the trained policy, controls the valve operation during the engine start-up process to maximize the reward for each agent.

[0056] By adopting the above technical solution, the present invention has the following advantages:

[0057] This invention provides a rocket landing control method based on multi-agent reinforcement learning. By determining the state space, action space, and reward function on the launch vehicle engine and vertical landing model, and using the LCO-MADDPG algorithm, MARL control is designed, trained, and evaluated to control the vertical landing process of the launch vehicle. This multi-agent reinforcement learning-based rocket landing control method achieves intelligent control of the launch vehicle's vertical landing process. Compared with traditional open-loop and closed-loop control methods, this method does not require extensive ground testing experience or complex control logic design. By designing a suitable reward function, complex objectives can be achieved. Furthermore, compared to the MADDPG algorithm, this multi-agent reinforcement learning-based rocket landing control method exhibits better model training stability and convergence in the control problem of the launch vehicle's vertical landing process. Through intelligent control, it achieves nonlinear control of the launch vehicle's vertical landing process without the need for complex control logic design. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the overall process of the rocket landing control method based on multi-agent reinforcement learning according to the present invention.

[0059] Figure 2 This is a diagram of a gas generator afterburner cycle engine system;

[0060] Figure 3 This is a diagram showing the altitude change of a simulation model of a launch vehicle's vertical descent.

[0061] Figure 4 This is a graph showing the velocity changes in a simulation model of a launch vehicle's vertical descent.

[0062] Figure 5 This is a diagram showing the change in gas generator thrust during the open-loop control startup process. Detailed Implementation

[0063] The technical solution of the present invention will be specifically described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0064] Figure 1 A general flowchart of the rocket landing control method based on multi-agent reinforcement learning of the present invention is shown. A specific rocket landing control method based on multi-agent reinforcement learning is as follows: Figure 1 As shown, it includes the following steps:

[0065] S1. Establish a rocket engine model, collect and output the state parameters necessary for the operation of the liquid rocket engine;

[0066] S1 includes the following specific steps:

[0067] S11. Use commercial simulation software or programming languages ​​to build a model of the liquid rocket engine to be analyzed; commercial simulation software (such as Matlab-Simulink, Amesim and EcosimPro) or programming languages ​​(such as Python, C or C++) can be used to build the engine model to be analyzed.

[0068] S12. Collect the necessary state parameters during the operation of the liquid rocket engine, so that the variables of the state parameters to be analyzed can be output.

[0069] In one specific embodiment, a model of a certain type of gas generator-assisted combustion circulating liquid engine was built based on the Matlab-Simulink platform, as detailed below. Figure 2 As shown in the diagram. RP1 represents fuel, LOX represents oxidizer, VCF represents combustion chamber fuel valve, VRF represents flow control valve, VGF represents gas generator fuel valve, VGO represents gas generator oxidizer valve, VGP represents gas bypass valve, IGN represents ignition device, GG represents gas generator, CC represents combustion chamber, and NE represents nozzle.

[0070] S2. Establish a simulation model for the vertical landing of the launch vehicle, and collect and output the parameters necessary for the vertical landing process of the launch vehicle;

[0071] S2 includes the following specific steps:

[0072] S21. Similar to the commercial simulation software or programming language used in step S11, commercial simulation software (such as Matlab-Simulink, Amesim and EcosimPro) or programming languages ​​(such as Python, C or C++) can be used to build a simulation model of the vertical landing process of the launch vehicle.

[0073] S22. Collect the necessary parameters during the vertical descent of the launch vehicle and output the variables of the necessary parameters.

[0074] In one specific embodiment, a simulation model of the vertical descent process of a launch vehicle was built based on a Python script. The simulation model includes: randomly setting the initial velocity and initial altitude of the launch vehicle from the range of initial altitude and velocity required for a safe landing; assuming the diameter of the launch vehicle hull as a reference for calculating the windward area to account for atmospheric drag, considering the thrust of the launch vehicle; taking into account the changes in gravitational acceleration and air density with the launch vehicle's altitude; and considering the changes in the launch vehicle's mass with propellant consumption. The resulting vertical descent simulation model shows the specific changes in the launch vehicle's altitude and velocity as follows: Figure 3 and Figure 4 As shown.

[0075] S3. Define the state space, action space, and reward function during the engine start-up process;

[0076] Based on the characteristics of the vertical descent process of the engine and the launch vehicle, the state space, action space, and reward function are selected for MARL training. The state space mainly includes engine state parameters, launch vehicle altitude, velocity, and acceleration. Including engine state parameters as part of the state space has the advantage that, since the vertical descent process is non-convex, MARL can better output the required thrust based on the engine and launch vehicle states, while preventing the engine state from exceeding safety thresholds. The action space mainly includes the valves of different engines during the vertical descent. The reward function mainly includes the safety threshold of the engine state during landing and the hard constraints for successful launch vehicle landing: factors that could lead to engine damage or start-up failure, such as deviations in the combustion chamber mixture ratio leading to overheating, and the speed and altitude that could cause the launch vehicle to crash. Soft constraints include factors affecting engine performance, such as thrust magnitude and combustion chamber pressure.

[0077] S31. Define the state space during engine start-up;

[0078] To train and use MARLagent, you need to define the state space and action space for each agent. The state space, which consists of the variables the agent receives from the environment at each time step, must contain enough information to explicitly define the current state of the system. The state space is set as follows:

[0079] S agent =[MR GG ,P GG ,T GG ,MR CC ,P CC ,T CC ,F,H,V,a,Pos VGO Pos VGF Pos VCF (1)

[0080] It contains 13 state parameters, among which MR GG ,P GG ,T GG ,MR CC ,P CC ,T CC F represents the mixture ratio of the engine gas generator, the gas generator chamber pressure, the gas generator temperature, the mixture ratio of the thrust chamber, the thrust chamber pressure, the thrust chamber temperature, and the thrust magnitude, respectively; H, V, and a represent the altitude, velocity, and acceleration of the launch vehicle, respectively; and Pos represents the thrust magnitude. VGO Pos VGF Pos VCFThese represent the oxidizer valve, fuel valve, and combustion chamber fuel valve of the gas generator, respectively. The observation space is normalized using steady-state reference values. Therefore, the proposed state space and method for this engine are not limited to the simulation environment described in this invention. In other engine systems or environments, some parameters, such as engine turbine efficiency, cannot be directly measured.

[0081] S32. Define the action space during engine starting;

[0082] The agent's action space A consists of the opening degrees of three valves.

[0083] A agent =[Pos VGO Pos VGF Pos VCF (2)

[0084] Within each time step, MARLAgent receives environmental observations and sends control signals to the engine's control valves. The interaction frequency between MARLAgent and the environment is 25 Hz.

[0085] S33. Define the reward function during the engine start-up process.

[0086] S33 includes the following steps:

[0087] S331. The reward used for training MARL and evaluating the startup sequence consists of the following different parts:

[0088] Reward = R engine +R landing (3)

[0089] R engine =r e1 +r e2 +r e3 (4)

[0090] The first reward is:

[0091]

[0092] Where, ε i ∈[MR GG ,P GG ,T GG ,MR CC ,P CC ,T CC Rewards are applied to values ​​that are close to the target value. Each reward component in this category has its maximum value reduced by 0.2 to improve cumulative rewards during training balance startup and steady-state periods.

[0093] The second reward is:

[0094]

[0095] This assumes that once the engine starts, it will not shut down. Therefore, after the engine generates thrust, the engine thrust should not be zero later to prevent the engine from shutting down during landing.

[0096] The third reward is:

[0097]

[0098] Among them, Act i ∈[Pos VGO Pos VGF Pos VCF ] represents the opening degree of the three valves of the engine, encouraging the attempt to open the valves before engine ignition; S represents the change in valve position between two steps before and after the valve, used to penalize the reciprocating action of the valve. This term can suppress the oscillation caused by the agent frequently actuating the valve during the vertical descent.

[0099] S332.

[0100] R landing =r l1 +r l2 +r l3 (8)

[0101] r l1 =0.2·exp(-abs(V) / 10)+0.2·(step max -step current ) / step max (9)

[0102] Among them, step max It is the maximum number of steps per training round. current This represents the current number of training steps. An exponential reward is used, meaning the smaller the speed, the larger the reward, guiding the agent to decelerate the launch vehicle. Furthermore, incorporating "step" into the reward function can encourage the agent to complete the launch vehicle landing in a shorter time.

[0103] r l2 = 0.2·exp(H / 50) + 0.2·(step) max -step current ) / step max (10)

[0104] Similar to the above formula, its purpose is to enable the agent to complete a faster and better landing.

[0105]

[0106] The agent receives a huge reward after the launch vehicle successfully lands, which encourages the successful landing strategy and trains more successful landing strategies.

[0107] S4. Based on the MADDPG algorithm, improvements are made, including 10 iterations of updates and the introduction of an annealing algorithm, to implement the LCO-MADDPG (Lossless Convex Optimized-MADDPG) algorithm.

[0108] To better apply the MADDPG algorithm to the vertical landing control process of launch vehicles, the LCO-MADDPG algorithm was proposed with two main improvements: updating the algorithm 10 times during training and using an annealing learning rate.

[0109] S4 includes the following steps:

[0110] S41. An improvement to the LCO-MADDPG algorithm was made, which involves 10 iterations of updates during training;

[0111] S42. An improvement to the LCO-MADDPG algorithm was made by using an annealing learning rate.

[0112] This improvement can further enhance the stability and efficiency of the algorithm to address the nonlinear and nonconvex characteristics of the vertical landing process of launch vehicles, and is particularly suitable for handling environments with complex or multimodal reward structures designed for the vertical landing process of launch vehicles.

[0113] The pseudocode for the LCO-MADDPG algorithm is shown in Table 1 below:

[0114] Table 1: Pseudocode of LCO-MADDPG Algorithm

[0115]

[0116]

[0117] In one specific embodiment, the code is implemented in Python, and real-time interaction between MATLAB and Python is achieved via an API at a frequency of 25Hz. The thrust variation diagram and the gas generator mixture ratio variation diagram during the open-loop control process are obtained, as shown below. Figure 5 As shown, where, Figure 5 This indicates the change in engine thrust under open-loop control;

[0118] S5. Design, train, and evaluate MARL models.

[0119] S5 includes the following steps:

[0120] S51. After transforming the vertical landing problem of a launch vehicle into a MARL problem, and considering the nonlinear characteristics of the engine system and the vertical landing phase, a MARL controller based on the LCO-MADDPG algorithm was designed, trained, and evaluated.

[0121] S52. Using the defined state space as the input to the controller, the MARL agent, based on the trained policy, controls the valve operation during the engine start-up process to maximize the reward for each agent.

[0122] In one specific embodiment, a MARL controller based on the LCO-MADDPG algorithm is implemented using Python code on the Matlab-Simulink simulation platform to control VGO, VGF and VCF respectively.

[0123] Finally, it should be noted that although the present invention has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, any changes or modifications to the above embodiments within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A rocket landing control method based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1. Establish a rocket engine model, collect and output the state parameters necessary for the operation of the liquid rocket engine; S2. Establish a simulation model for the vertical landing of the launch vehicle, and collect and output the parameters necessary for the vertical landing process of the launch vehicle; S3. Define the state space, action space, and reward function during the engine start-up process; S4. Based on the MADDPG algorithm, an improved version of the MADDPG algorithm with lossless convex optimization is implemented. S5. Design, train, and evaluate MARL models; S3 includes the following steps: S31. Define the state space during engine start-up; The state space for each agent needs to be defined, and the state space is set as follows: S agent =[MR GG ,P GG ,T GG ,MR CC ,P CC ,T CC ,F,H,V,a,Pos VGO ,Pos VGF ,Pos VCF ] Among them, MR GG ,P GG ,T GG ,MR CC ,P CC ,T CC F represents the mixture ratio of the engine gas generator, the gas generator chamber pressure, the gas generator temperature, the mixture ratio of the thrust chamber, the thrust chamber pressure, the thrust chamber temperature, and the thrust magnitude, respectively; H, V, and a represent the altitude, velocity, and acceleration of the launch vehicle, respectively; and Pos represents the thrust magnitude. VGO Pos VGF Pos VCF These represent the oxidizer valve, fuel valve, and combustion chamber fuel valve of the gas generator, respectively, and the state space is normalized using steady-state reference values. S32. Define the action space during engine starting; The agent's action space A consists of the opening degrees of three valves. TO agent =[Pos VGO ,Pos VGF ,Pos VCF ] Within each time step, MARLAgent receives environmental observations and sends control signals to the engine's control valves; S33. Define the reward function during engine starting; S33 includes the following steps: S331. The reward used for training MARL and evaluating the startup sequence consists of the following different parts: Reward=R engine +R landing R engine ＝r e1 +r e2 +r e3 The first reward is: Where, ε i ∈[MR GG ,P GG ,T GG ,MR CC ,P CC ,T CC For rewards that are close to the target value, each reward component in this item has its maximum value reduced by 0.2 to improve the cumulative reward during training balance startup and steady state. The second reward is: This assumes that once the engine starts, it will not shut down. Therefore, once the engine generates thrust, the engine thrust should not be zero later to prevent the engine from shutting down during landing. The third reward is: Among them, Act i ∈[Pos VGO Pos VGF Pos VCF ] represents the opening degree of the three valves of the engine, encouraging the attempt to open the valves before engine ignition; S represents the change in valve position between two steps before and after the valve, used to penalize the reciprocating action of the valve. This item can suppress the oscillation caused by the agent frequently actuating the valve during the vertical descent. S332. R landing ＝r l1 +r l2 +r l3 r l1 =0.2·exp(-abs(V) / 10)+0.2·(step max -step current ) / step max Among them, step max It is the maximum number of steps per training round. current This is the current number of training steps, with exponential rewards used to guide the agent to decelerate the launch vehicle. In addition, incorporating the step function into the reward function helps the agent complete the launch vehicle landing in a shorter time. r l2 =0.2·exp(H / 50)+0.2·(step max -step current ) / step max Similar to the formula above, its purpose is to enable the agent to land faster and better. The agent receives a huge reward after the launch vehicle successfully lands, which encourages the successful landing strategy and trains more successful landing strategies.

2. The rocket landing control method based on multi-agent reinforcement learning according to claim 1, characterized in that, S1 includes the following steps: S11. Use commercial simulation software or programming languages ​​to build a model of the liquid rocket engine to be analyzed; S12. Collect the necessary state parameters during the operation of the liquid rocket engine, so that the variables of the state parameters to be analyzed can be output.

3. The rocket landing control method based on multi-agent reinforcement learning according to claim 2, characterized in that, S2 includes the following steps: S21. Using the same commercial simulation software or programming language as used in step S11, build a simulation model of the vertical landing process of the launch vehicle. S22. Collect the necessary parameters during the vertical descent of the launch vehicle and output the variables of the necessary parameters.

4. The rocket landing control method based on multi-agent reinforcement learning according to claim 1, characterized in that, S4 includes the following steps: S41. An improvement to the lossless convex optimization MADDPG algorithm was made, which performs 10 iterations of updates during training; S42. An improvement to the lossless convex optimization MADDPG algorithm was made by using an annealing learning rate.

5. The rocket landing control method based on multi-agent reinforcement learning according to claim 1, characterized in that, S5 includes the following steps: S51. Combining the nonlinear characteristics of the engine system and the vertical landing phase, a MARL controller based on the lossless convex optimization algorithm MADDPG was designed, trained, and evaluated. S52. Using the defined state space as the input to the controller, the MARL agent, based on the trained policy, controls the valve operation during the engine start-up process to maximize the reward for each agent.

Images