Rocket powered landing segment guidance control cooperation method and system based on reinforcement learning

Abstract

The application discloses a rocket powered landing section guidance control cooperation method and system based on reinforcement learning, and belongs to the technical field of guidance control.The application proposes a guidance control parameter adjustment method based on reinforcement learning, models a parameter adjustment process as a Markov decision process, trains a parameter adjustment strategy network of the parameter adjustment process by using a reinforcement learning method, realizes expected optimization of a guidance control comprehensive target under model uncertainty and disturbance, improves the precision of rocket powered landing section guidance control, and reduces landing loss.

Description

Technical Field

[0001] This invention relates to the field of guidance and control technology, and in particular to a cooperative method and system for guidance and control of rocket-powered landing phase based on reinforcement learning. Background Technology

[0002] Reusable rockets are rocket vehicles whose entire rocket body or some of its stages can be reused. Compared to traditional rockets, reusable rockets can reuse key components such as engines, significantly reducing the cost per launch and becoming an important part of the next generation of space transportation systems. The powered landing phase is a critical stage in the recovery process of reusable rockets. This flight phase is characterized by significant model uncertainties and disturbances, severe attitude-trajectory coupling, and stringent requirements for terminal constraints and fuel optimality.

[0003] Existing design methods employ a separate design approach for guidance and control. In guidance design, the tracking dynamics of attitude control in response to guidance commands are ignored; in attitude control design, the impact of engine sway angles on the center of mass motion is neglected. When model uncertainties and disturbances are significant, the guidance law may generate drastically changing attitude commands and increase the tracking error of the attitude control law. Simultaneously, the attitude control law may generate large engine sway angles and create additional disturbances to the rocket's center of mass motion, thus compromising the accuracy of terminal constraint satisfaction, fuel consumption optimization, and attitude control quality during the powered landing phase. Summary of the Invention

[0004] The purpose of this invention is to provide a reinforcement learning-based collaborative method and system for guidance and control during the rocket's powered landing phase, so as to improve the accuracy of guidance and control during the rocket's powered landing phase.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] This invention provides a cooperative guidance and control method for rocket-powered landing phase based on reinforcement learning, the method comprising the following steps:

[0007] Establish the equations of motion for the rocket's powered landing phase;

[0008] Determine the integrated guidance and control objectives for the rocket's powered landing phase;

[0009] Based on the motion equations and the comprehensive guidance and control objective, a guidance and control cooperative Markov decision process model is established. This model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, an attitude control module simulation model, and motion equations. The parameter adjustment strategy network, parameter adjustment value network, guidance module simulation model, attitude control module simulation model, and motion equations form a closed-loop system. The reward function of the guidance and control cooperative Markov decision process model is established based on the comprehensive guidance and control objective.

[0010] Based on the guidance and control cooperative Markov decision process model, the parameters of the parameter adjustment strategy network are trained using the nearest neighbor strategy optimization algorithm to obtain the parameter-optimized parameter adjustment strategy network.

[0011] The guidance and control of the rocket's powered landing phase are based on the behavioral quantities generated by the parameter adjustment strategy network after parameter optimization, where the behavioral quantities are the control parameters.

[0012] Optionally, the equation of motion is:

[0013]

[0014] Where r is the rocket's position; v is the rocket's velocity; g is the acceleration due to gravity; m is the rocket's mass; T is the magnitude of the rocket engine's thrust; V ex Refers to the exhaust velocity of a rocket engine; F aero and F tvc These represent rocket aerodynamic force and rocket engine thrust, respectively; ω b Refers to the angular velocity of the three axes; ψ and γ are the rocket's pitch angle, yaw angle, and roll angle, respectively; R x (γ) is the elementary rotation matrix of the rocket roll angle about the x-axis, R y (ψ) is the elementary rotation matrix of the rocket's yaw angle about the y-axis; J is the rocket's moment of inertia matrix; M aero and M tvc These are the rocket's aerodynamic torque and the engine's thrust torque, respectively.

[0015] Optionally, the reward function is:

[0016]

[0017] Where r(s) t ,a t ) is the reward function, s t Let a be the state variable at time t. t Let t be the action quantity at time t, where t is the time within the rocket's powered landing phase. fFor terminal landing time, k1, k2, k3, k4, k5, and k6 are weighting coefficients for different targets in the guidance and control integrated target, J r For terminal position error, J v For speed error, J m The difference in mass between the initial and final moments of the powered landing phase of flight. J represents the horizontal component of the unit vector pointing along the rocket's longitudinal axis at the moment of final landing. δ J is the average attitude tracking error. error This represents the average engine sway angle.

[0018] Optionally, the step of training the parameters of the parameter adjustment strategy network using the nearest neighbor policy optimization algorithm based on the guidance and control cooperative Markov decision process model to obtain the parameter-optimized parameter adjustment strategy network specifically includes:

[0019] Based on the guidance and control cooperative Markov decision process model, N closed-loop simulations are performed to obtain a sampling sequence; the sampling sequence includes N sets of simulation sequences, and the simulation sequence includes the state and behavior quantities at different times during the rocket's powered landing phase obtained from the simulation.

[0020] The parameter update gradient of the parameter-adjusted value network is calculated based on the sampling sequence, and the parameters of the parameter-adjusted value network are updated to obtain the updated parameter-adjusted value network.

[0021] Based on the sampling sequence and the updated parameter adjustment value network, the parameter update gradient of the parameter adjustment strategy network is calculated, and the parameters of the parameter adjustment strategy network are updated to obtain the updated parameter adjustment strategy network.

[0022] Determine whether the maximum number of sampling steps has been reached, and obtain the determination result;

[0023] If the judgment result indicates no, then return to the step of performing N closed-loop simulations based on the guidance and control cooperative Markov decision process model to obtain the sampling sequence;

[0024] If the judgment result indicates yes, then the updated parameter adjustment strategy network is output as the parameter-optimized parameter adjustment strategy network.

[0025] Optionally, the step of performing N closed-loop simulations based on the guidance and control cooperative Markov decision process model to obtain the sampling sequence specifically includes:

[0026] Based on the uncertainty of state transition, the initial state variables are set as s0~p0(·); where s0 is the initial state variable and p0(·) is the state variable initialization function considering the uncertainty of state transition.

[0027] The state variables at the current iteration time are input into the parameter adjustment policy network to obtain the behavior variables at the current iteration time;

[0028] The behavior parameters at the current iteration moment are input into the simulation models of the guidance module and the attitude control module to perform simulation, thereby obtaining the rocket engine thrust and rocket engine yaw angle at the current iteration moment.

[0029] The rocket engine thrust and rocket engine sway angle at the current iteration time are input into the equation of motion to obtain the state variables at the next iteration time. The process of inputting the state variables at the current iteration time into the parameter adjustment strategy network to obtain the behavior variables at the current iteration time is repeated until the terminal landing time is reached to obtain the simulation sequence of the current closed-loop simulation.

[0030] Optionally, the formula for calculating the parameter update gradient of the parameter-adjusted value network is:

[0031]

[0032] Among them, g td To adjust the gradient of the parameter update of the value network, s t and s t+1 Let θ be the state variables at time t and time t+1, respectively. v Adjusting the parameters of the value network to provide parameters. To approximate the expectation using the arithmetic mean, s t Let a be the state variable at time t. t Let t be the behavioral quantity. For the sampled sequence, r(s) t ,a t ) is the reward function.

[0033] Optionally, the formula for calculating the parameter update gradient of the parameter tuning policy network is:

[0034]

[0035]

[0036] Among them, g ppo To update the gradients for the parameters of the parameter tuning policy network, To approximate the expectation using the arithmetic mean, s t Let a be the state variable at time t. t Let t be the behavioral quantity. Here, θ represents the sampling sequence, and θ represents the parameters of the parameter adjustment strategy network. Let ρ be the gradient with respect to the parameter θ. θ (s t ,a t ) represents the probability ratio of the policy output behavior, πθ(a)t |s t ( ) represents the parameter tuning policy network to be trained. This is the parameter adjustment policy network after the last gradient update. The advantage function, δ(s), is estimated using generalized advantage estimation. τ ,a τ ) represents the temporal difference increment of the value network, r(s) τ ,a τ Let V be the reward function, λ be the weighting coefficient, and V be the weighting coefficient. θv (s τ+1 ) to adjust the value network for the updated parameters, s τ and s τ+1 The state variables at time τ and time τ+1 are respectively, θ v The parameters of the value network are adjusted by clip(x,∈), where clip(x,∈) is the clipping function, x is the referential function, and x=ρ θ (s t ,a t ), where ∈ is the radius of the trust region for policy gradient updates.

[0037] A reinforcement learning-based rocket-powered landing phase guidance and control cooperative system, wherein the system is applied to the above-described method, and the system includes:

[0038] The equations of motion module is used to establish the equations of motion for the rocket's powered landing phase.

[0039] The guidance and control integrated target determination module is used to determine the integrated guidance and control targets for the rocket's powered landing phase.

[0040] A guidance and control cooperative Markov decision process model establishment module is used to establish a guidance and control cooperative Markov decision process model based on the motion equations and the guidance and control integrated objective. The guidance and control cooperative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, an attitude control module simulation model, and motion equations. The parameter adjustment strategy network, parameter adjustment value network, guidance module simulation model, attitude control module simulation model, and motion equations form a closed-loop system. The reward function of the guidance and control cooperative Markov decision process model is established based on the guidance and control integrated objective.

[0041] The optimization solution module is used to train the parameters of the parameter adjustment strategy network based on the guidance and control cooperative Markov decision process model and the nearest policy optimization algorithm to obtain the parameter adjustment strategy network with optimized parameters.

[0042] The control module is used to perform guidance control of the rocket's powered landing phase based on the behavioral quantities generated by the parameter adjustment strategy network after parameter optimization, wherein the behavioral quantities are control parameters.

[0043] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0044] A computer-readable storage medium storing a computer program that, when executed, implements the above-described method.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This invention provides a reinforcement learning-based guidance and control cooperation method and system for rocket-powered landing phase. The method includes: establishing the motion equations for the rocket-powered landing phase; determining the comprehensive guidance and control objective for the rocket-powered landing phase; establishing a guidance and control cooperation Markov decision process model based on the motion equations and the comprehensive guidance and control objective; training the parameters of a parameter adjustment strategy network using a proximity policy optimization algorithm based on the guidance and control cooperation Markov decision process model to obtain a parameter-optimized parameter adjustment strategy network; and performing guidance and control for the rocket-powered landing phase based on the behavioral quantities generated by the parameter-optimized parameter adjustment strategy network. In summary, this invention proposes a reinforcement learning-based guidance and control parameter adjustment method, modeling the parameter adjustment process as a Markov decision process, and using reinforcement learning to train the parameter adjustment strategy network. This achieves the expected optimality of the comprehensive guidance and control objective under model uncertainty and interference, improving the accuracy of guidance and control during the rocket-powered landing phase and reducing landing losses. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a reinforcement learning-based guidance and control cooperation method for rocket-powered landing phase, provided as an embodiment of the present invention;

[0049] Figure 2 A schematic diagram illustrating the principle of network parameter optimization for parameter adjustment strategies provided in embodiments of the present invention;

[0050] Figure 3 A comparison diagram showing the distribution of landing points between the guidance control law designed for the control cooperation method provided in this embodiment of the invention and the guidance control law designed separately for the nominal model. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] The purpose of this invention is to provide a reinforcement learning-based collaborative method and system for guidance and control during the rocket's powered landing phase, so as to improve the accuracy of guidance and control during the rocket's powered landing phase.

[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] This invention provides a reinforcement learning-based guidance and control cooperation method for rocket-powered landing phase, the method comprising the following steps:

[0055] Establish the equations of motion for the rocket-powered landing phase.

[0056] Determine the integrated guidance and control objectives for the rocket's powered landing phase.

[0057] Based on the equations of motion and the comprehensive guidance and control objective, a collaborative Markov decision process model for guidance and control is established. This model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, an attitude control module simulation model, and the equations of motion. The parameter adjustment strategy network, parameter adjustment value network, guidance module simulation model, attitude control module simulation model, and equations of motion form a closed-loop system. The reward function of this collaborative Markov decision process model is established based on the comprehensive guidance and control objective.

[0058] Based on the guidance and control cooperative Markov decision process model, the parameters of the parameter adjustment strategy network are trained using the proximity policy optimization algorithm to obtain the parameter-optimized parameter adjustment strategy network.

[0059] The guidance and control of the rocket's powered landing phase are based on the behavioral quantities generated by the parameter adjustment strategy network after parameter optimization, where the behavioral quantities are the control parameters.

[0060] like Figure 1As shown, in this embodiment of the invention, establishing the equations of motion for the rocket-powered landing segment and determining the integrated guidance and control objective for the rocket-powered landing segment are steps in establishing the integrated guidance and control objective optimization problem. In this embodiment of the invention, establishing a guidance and control cooperative Markov decision process model based on the equations of motion and the integrated guidance and control objective is a step in establishing the guidance and control cooperative Markov decision process. In this embodiment of the invention, training the parameters of the parameter adjustment strategy network using the nearest neighbor strategy optimization algorithm based on the guidance and control cooperative Markov decision process model to obtain the parameter-optimized parameter adjustment strategy network is a step in modular cooperative framework design. In this embodiment of the invention, performing guidance and control for the rocket-powered landing segment based on the behavioral quantities generated by the parameter-optimized parameter adjustment strategy network, where the behavioral quantities are control parameters, is a step in parameter adjustment strategy migration.

[0061] Step 1: Establishing the comprehensive target optimization problem for guidance and control;

[0062] Based on the overall parameters of the rocket, guidance and control performance indicators, model uncertainties, and interference models, this invention establishes a comprehensive target optimization problem model for guidance and control during the powered landing phase.

[0063] First, the equations of motion for the rocket's powered landing phase are established, including the equations of motion around its center of mass and the equations of motion around the center of mass as follows:

[0064]

[0065] Where r is the rocket's position; v is the rocket's velocity; g is the acceleration due to gravity; m is the rocket's mass; T is the magnitude of the rocket engine's thrust; V ex Refers to the exhaust velocity of a rocket engine; F aero and F tvc These are rocket aerodynamic force and rocket engine thrust, respectively, where F tvc It is the rocket engine thrust T and the rocket engine swashplate angle. δ ψ The function; ω b Refers to the angular velocity of the three axes; ψ and γ are the rocket's pitch angle, yaw angle, and roll angle, respectively; R a (b) represents the element rotation matrix; J is the rocket's moment of inertia matrix; M aero and M tvc These are the rocket aerodynamic torque and the engine thrust torque, respectively, where M tvc It is the rocket engine thrust T and the rocket engine swashplate angle. δ ψ The function.

[0066] At the same time, a mathematical model for the integrated guidance and control objectives is established.

[0067] The accuracy of the terminal centroid state constraint is characterized by the terminal position and velocity errors, i.e.:

[0068]

[0069] Where t f For terminal landing time;

[0070] The difference in mass at the beginning and end of the powered landing phase is used to characterize the optimality of fuel consumption.

[0071] J m =m(0)-m(t) f (3)

[0072] The horizontal component of the unit vector pointing to the rocket's longitudinal axis at the terminal moment is used to characterize the accuracy of the constraint satisfaction of the terminal state quantity around the center of mass, i.e.:

[0073]

[0074] The attitude control quality is characterized by the average attitude tracking error and the average engine yaw angle.

[0075]

[0076] Where dt c For attitude control cycle; and ψ c The attitude commands generated by the guidance law. In summary, the integrated objective of guidance and control during the powered landing phase can be represented in the following vector form:

[0077]

[0078] Step 2: Establishment of the guidance and control collaborative Markov decision process;

[0079] This invention establishes a closed-loop system based on the guidance and control integrated objective optimization problem model. This system consists of rocket motion equations, a parameterized guidance module simulation model, a parameterized attitude control module simulation model, and a parameter adjustment strategy. The system is then modeled as a Markov decision process. Specifically, the model is established as follows: the rocket motion equations are discretized, and the parameter update time intervals for the guidance module simulation model and the attitude control module simulation model are selected. Based on the closed-loop system formed by the rocket motion equations, guidance module simulation model, and attitude control module simulation model, a state transition equation for the Markov decision process is established. The state variables and behavioral variables in this state transition are selected, and the reward function is obtained by weighting the components of the guidance and control integrated objective. Specifically:

[0080] Adopting such Figure 2The reinforcement learning training framework shown is used for parameter cooperation strategy design. The closed-loop system consisting of the parameter tuning strategy (including the parameter tuning strategy network and the parameter tuning value network), the guidance module simulation model, the attitude control module simulation model, and the dynamic landing motion equations is modeled as a guidance and control cooperative Markov Decision Process (MDP) model. Under this model, the guidance module simulation model and the attitude control module simulation model can use any existing method with adjustable parameters to convert the state variables s of the guidance and control cooperative Markov Decision Process model. t Selected as a measurable state variable in the dynamic landing motion equations, the behavior variable a t The adjustable parameters selected for the guidance and attitude control module simulation model are the mapping strategies from MDP state variables to behavior variables.

[0081] Given the current state variable s t and behavioral quantity a t The state transitions in the guidance and control cooperative Markov decision process model are determined by a closed-loop system consisting of the guidance module simulation model, the attitude control module simulation model, and the motion equations. Specifically, the parameter update period dt of the given parameter adjustment strategy... gnc Guidance period dt g and attitude control period dt c The MDP state s at the next parameter update time t+1 It can be generated by simultaneously integrating the simulation model expressions of the guidance module, the attitude control module, and the dynamic landing motion equations. A complete episode of this guidance and control cooperative Markov decision process model is from t=0 to t=t. f The complete landing process at any given moment. Furthermore, uncertainties in the state transition include the wind field, the proportion of parameter perturbations, and the initial state dispersion of the powered landing. Therefore, in each round of the MDP, the wind field, parameter perturbations, and the initial rocket state are randomly generated within a given range. For ease of description, the dispersion of the initial state of the guidance and control cooperative Markov decision process model is denoted as s0~p0(·), and the dispersion of the new state after the state transition is denoted as s t+1 ~p(·|s t ,a t ).

[0082] Finally, the reward function in the guidance and control cooperative Markov decision process model is generated by weighting the components of the integrated objective J of the powered landing phase guidance and control.

[0083]

[0084] Where k1-k6 are weighting coefficients.

[0085] Based on the aforementioned guidance and control cooperative Markov decision process model, the parameter adjustment strategy to be designed is a mapping from MDP state variables to MDP behavior variables, denoted as a. t ~π(·|s t Furthermore, the optimization objective of this strategy can be described as maximizing the expected cumulative reward of the guided control cooperative Markov decision process model over the specified number of rounds, i.e.

[0086]

[0087] Step 3: Modular collaboration framework design;

[0088] According to the guidance and control cooperative Markov decision process model, the parameters of the parameter adjustment strategy network are trained by the proximity strategy optimization algorithm to obtain the parameter adjustment strategy network after parameter optimization.

[0089] The optimal strategy π shown in equation (8) (i.e., the parameter adjustment strategy network) is fitted using a parameter adjustment strategy in the form of a neural network. * And denote it as π θ Where θ is the parameter of the neural network to be optimized, then equation (8) can be rewritten as:

[0090]

[0091] To address this neural network parameter optimization problem, a nearest neighbor strategy optimization algorithm is employed to update the neural network parameters using gradients. During each gradient calculation, the strategy updated in the previous gradient update is first used. The simulation model parameters for the guidance and attitude control modules are generated, and N closed-loop simulations are performed with the simulation models of the guidance and attitude control modules and the motion equations of the powered landing to obtain N rounds of state and behavior sampling sequences, which are denoted as... The update gradient of the network parameters for the parameter tuning strategy can be given by the following estimation:

[0092]

[0093] in This indicates that the arithmetic mean is used to approximate the expectation; ∈ represents the radius of the trust region for policy gradient updates; The advantage function is estimated using generalized advantage estimation. It is represented as:

[0094]

[0095] Where λ is the weighting coefficient; For θ v For a neural network with a value function as its parameters, its gradient update is also determined by... The estimated value can be expressed as:

[0096]

[0097] Among them, g td To adjust the gradient of the parameter update of the value network, s t and s t+1 Let θ be the state variables at time t and time t+1, respectively. v Adjusting the parameters of the value network to provide parameters. To approximate the expectation using the arithmetic mean, s t Let a be the state variable at time t. t Let t be the behavioral quantity. For the sampled sequence, r(s) t ,a t ) is the reward function.

[0098] In summary, the complete parameter tuning strategy optimization algorithm can be summarized into the following steps:

[0099] Input: Number of sampling rounds N, maximum number of sampling steps N max The step size α for updating policy network parameters θ Value network parameter update step size The weighting coefficient λ and the trust region radius are ∈.

[0100] Step 1: Initialize the parameter adjustment strategy network π θ Sampling strategy network and parameter adjustment value network Among them, the sampling strategy network is the parameter adjustment strategy network after the previous gradient update;

[0101] Step 2: From the sampling strategy network The process generates behavioral variables, i.e., control parameters, which interact with the guidance and control collaborative Markov decision process model for N rounds, generating sampled sequences of state and behavioral variables.

[0102] Step 3: Based on the sampling sequence Calculate the gradient g of parameter update for parameter-adjusted value networks. v Update parameters to adjust the value network:

[0103] Step 4: Based on the sampling sequence and current parameter adjustment value network The parameter update gradient g of the parameter tuning strategy network is calculated. ppo Update the policy network: θ←θ-α θ g ppo ;

[0104] Step 5: Update the parameters of the sampling policy network: θold ←θ;

[0105] Step 6: If the maximum number of sampling steps N is reached... max If the algorithm terminates, the process ends; otherwise, return to step 2.

[0106] Output: Parameter tuning strategy network π θ

[0107] Step 4: Parameter adjustment strategy migration;

[0108] In this embodiment of the invention, the trained and optimized parameter adjustment strategy network is solidified and a closed loop is formed with the input-output interfaces of the rocket's motion equations, the parameterized guidance module simulation model, and the parameterized attitude control module simulation model. This forms a complete guidance and control law for the powered landing phase. Specifically, the network parameters of the parameter adjustment neural network strategy obtained through reinforcement learning are fixed. The state variables output from the rocket's motion equations are used as the input to the parameter adjustment neural network strategy. After forward propagation of the neural network, the output of the parameter adjustment neural network strategy is used as the parameter values ​​for the guidance control module. The guidance control module generates rocket control variables under the current parameter values, forming a control closed loop. Specifically:

[0109] The parameters of the parameter adjustment strategy network are taken as the optimal values ​​obtained from the convergence of reinforcement learning training. The state variable s is obtained by numerical integration of the rocket motion equation. t , the state variable s t As input to the parameter adjustment strategy network, the behavior quantity 'a' is generated through forward propagation of the neural network. t These parameters are used as the parameters for the guidance module simulation model and the attitude control module simulation model. The engine thrust T and rocket engine pitch angle are generated from the guidance module simulation model and the attitude control module simulation model under the current parameter values. δ ψ Substituting these values ​​into the rocket's equations of motion and integrating them yields new rocket state variables, thus achieving a closed-loop control system.

[0110] To illustrate the effectiveness of the method provided in the embodiments of the present invention, the embodiments of the present invention provide the following simulation examples.

[0111] This case study uses a numerical simulation to demonstrate the methodology and is not an actual flight mission. The nominal values ​​for the rocket model parameters are: reference cross-sectional area S. ref 1m 2 Engine exhaust speed V ex 2700 m·s -1 The engine's maximum thrust is 40 kN; minimum thrust is 20 kN; and the X, Y, and Z axis moments of inertia are 300 kg·m. 2 6000 kg·m 2 and 6000 kg·m 2The atmospheric model used was the US 1976 standard atmospheric model. Multiple Monte Carlo simulations were conducted, taking into account the random distribution of the rocket's initial position, velocity, and attitude during powered landing, model errors in aerodynamic forces and moments, and typical random wind field disturbances from the literature.

[0112] Based on the implementation process, the landing point distribution of the guidance control law designed by the method provided in this embodiment of the invention and the discrete design of the nominal guidance control law without parameter adjustment strategy cooperation are compared, for example... Figure 3 Table 1 shows the comparison of the maximum values ​​of each component of the guidance and control integrated objective. It can be seen that the modular cooperative guidance and control law improves the performance of each component of the integrated objective compared to the discrete design, proving that this method can optimize the guidance and control integrated objective under model uncertainty and disturbances compared to the discrete design.

[0113] Table 1 Maximum values ​​of the integrated target components in guidance and control

[0114]

[0115] Example 2

[0116] Embodiment 2 of the present invention provides a reinforcement learning-based rocket-powered landing phase guidance and control cooperative system, wherein the system is applied to the above-described method, and the system includes:

[0117] The equations of motion module is used to establish the equations of motion for the rocket's powered landing phase.

[0118] The guidance and control integrated target determination module is used to determine the integrated guidance and control targets for the rocket's powered landing phase.

[0119] A guidance and control cooperative Markov decision process model establishment module is used to establish a guidance and control cooperative Markov decision process model based on the motion equations and the guidance and control integrated objective. The guidance and control cooperative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, an attitude control module simulation model, and motion equations. The parameter adjustment strategy network, parameter adjustment value network, guidance module simulation model, attitude control module simulation model, and motion equations form a closed-loop system. The reward function of the guidance and control cooperative Markov decision process model is established based on the guidance and control integrated objective.

[0120] The optimization solution module is used to train the parameters of the parameter adjustment strategy network based on the guidance and control cooperative Markov decision process model and the nearest policy optimization algorithm to obtain the parameter adjustment strategy network with optimized parameters.

[0121] The control module is used to perform guidance control of the rocket's powered landing phase based on the behavioral quantities generated by the parameter adjustment strategy network after parameter optimization, wherein the behavioral quantities are control parameters.

[0122] Example 3

[0123] Embodiment 3 of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0124] Example 4

[0125] Embodiment 4 of the present invention provides a computer-readable storage medium storing a computer program, which, when executed, implements the above-described method.

[0126] In summary, implementing the embodiments of the present invention has the following beneficial effects:

[0127] (1) Based on the reinforcement learning method, the embodiment of the present invention obtains the guidance and control parameter adjustment strategy for rocket-powered landing, which solves the existing problems of severe attitude-orbit coupling, low accuracy of terminal constraint satisfaction, large fuel consumption and poor attitude control quality under model uncertainty and interference. It can be applied to any rocket-powered landing motion model.

[0128] (2) The embodiments of the present invention use a near-end strategy optimization algorithm to optimize the parameter adjustment strategy and perform real-time parameter adjustment on the guidance and control module with a fixed structure. This avoids the problems of poor adaptability to model uncertainty and interference faced by manually designing parameter adjustment laws and can achieve good optimization effect of the comprehensive guidance and control objectives.

[0129] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0130] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A cooperative guidance and control method for rocket-powered landing phase based on reinforcement learning, characterized in that, The method includes the following steps: Establish the equations of motion for the rocket's powered landing phase; Determine the integrated guidance and control objectives for the rocket's powered landing phase; Based on the motion equations and the comprehensive guidance and control objective, a guidance and control cooperative Markov decision process model is established. This model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, an attitude control module simulation model, and motion equations. The parameter adjustment strategy network, parameter adjustment value network, guidance module simulation model, attitude control module simulation model, and motion equations form a closed-loop system. The reward function of the guidance and control cooperative Markov decision process model is established based on the comprehensive guidance and control objective. Based on the guidance and control cooperative Markov decision process model, the parameters of the parameter adjustment strategy network are trained using the nearest neighbor strategy optimization algorithm to obtain the parameter-optimized parameter adjustment strategy network. The rocket's powered landing phase guidance control is based on the behavioral quantities generated by the parameter adjustment strategy network after parameter optimization, where the behavioral quantities are control parameters. Determine the integrated guidance and control objectives for the rocket's powered landing phase, specifically including: The accuracy of the terminal centroid state constraint is characterized by the terminal position and velocity errors, i.e.: ； in, This refers to the terminal's landing time; The difference in mass at the beginning and end of the powered landing phase is used to characterize the optimality of fuel consumption. ； The horizontal component of the unit vector pointing to the rocket's longitudinal axis at the terminal moment is used to characterize the accuracy of the constraint satisfaction of the terminal state quantity around the center of mass, i.e.: ； The attitude control quality is characterized by the average attitude tracking error and the average engine yaw angle. ； in, For attitude control cycle; and The attitude commands generated by the guidance law; and All are rocket engine tilt angles; in summary, the integrated objective of guidance and control during the powered landing phase can be represented in the following vector form: ； The reward function is: ； in, For the reward function, Let be the state variable at time t. Let be the behavioral quantity at time t, where t is the time within the rocket's powered landing phase. For terminal landing time, , , , , , These are the weighting coefficients for different objectives in the overall guidance and control objective. For terminal position error, For speed error, The difference in mass between the initial and final moments of the powered landing phase of flight. This represents the horizontal component of the unit vector pointing along the rocket's longitudinal axis at the moment of final landing. The average attitude tracking error, This represents the average engine sway angle; The equation of motion is: ； in, Position of the rocket; For rocket speed; It is the acceleration due to gravity; For rocket mass; This refers to the amplitude of the rocket engine thrust; Refers to the exhaust velocity of a rocket engine; and These are rocket aerodynamic force and rocket engine thrust, respectively. Refers to the angular velocity of the three axes; , and These are the rocket's pitch angle, yaw angle, and roll angle, respectively. The elementary rotation matrix of the rocket's roll angle about the x-axis, Let be the elementary rotation matrix of the rocket's y-axis; Here is the rocket's moment of inertia matrix; and These are the rocket's aerodynamic torque and the engine's thrust torque, respectively. The step of training the parameters of the parameter adjustment strategy network using the proximity strategy optimization algorithm based on the guidance and control cooperative Markov decision process model to obtain the parameter-optimized parameter adjustment strategy network specifically includes: Based on the guidance and control cooperative Markov decision process model, N closed-loop simulations are performed to obtain a sampling sequence; the sampling sequence includes N sets of simulation sequences, and the simulation sequence includes the state and behavior quantities at different times during the rocket's powered landing phase obtained from the simulation. The parameter update gradient of the parameter-adjusted value network is calculated based on the sampling sequence, and the parameters of the parameter-adjusted value network are updated to obtain the updated parameter-adjusted value network. Based on the sampling sequence and the updated parameter adjustment value network, the parameter update gradient of the parameter adjustment strategy network is calculated, and the parameters of the parameter adjustment strategy network are updated to obtain the updated parameter adjustment strategy network. Determine whether the maximum number of sampling steps has been reached, and obtain the determination result; If the judgment result indicates no, then return to the step of performing N closed-loop simulations based on the guidance and control cooperative Markov decision process model to obtain the sampling sequence; If the judgment result indicates yes, then the updated parameter adjustment strategy network is output as the parameter-optimized parameter adjustment strategy network; The sampling sequence is obtained by performing N closed-loop simulations based on the guidance and control cooperative Markov decision process model, specifically including: Based on the uncertainty of state transition, the initial state variables are set to... ;in, These are the initial state variables. State initialization functions to account for uncertainties in state transitions; The state variables at the current iteration time are input into the parameter adjustment policy network to obtain the behavior variables at the current iteration time; The behavior parameters at the current iteration moment are input into the simulation models of the guidance module and the attitude control module to perform simulation, thereby obtaining the rocket engine thrust and rocket engine yaw angle at the current iteration moment. The rocket engine thrust and rocket engine sway angle at the current iteration time are input into the equation of motion to obtain the state quantity at the next iteration time. The process of inputting the state quantity at the current iteration time into the parameter adjustment strategy network to obtain the behavior quantity at the current iteration time is repeated until the terminal landing time is reached to obtain the simulation sequence of the current closed-loop simulation. The formula for calculating the parameter update gradient of the parameter-adjusted value network is: ； in, Adjust the gradient of the parameter update for the value network. and They are respectively Time and State quantity at any given time. Adjusting the parameters of the value network to provide parameters. To approximate the expectation using the arithmetic mean, Let be the state variable at time t. Let t be the behavioral quantity. For sampling sequences, For the reward function, For The value function of a neural network is a parameter; The formula for calculating the parameter update gradient of the parameter tuning strategy network is: ； ； in, To update the gradients for the parameters of the parameter tuning policy network, To approximate the expectation using the arithmetic mean, Let be the state variable at time t. Let t be the behavioral quantity. For sampling sequences, To adjust the parameters of the strategy network, For parameters gradient, The output behavior probability ratio is used to define the policy. For the parameter tuning policy network to be trained, This is the parameter adjustment policy network after the last gradient update. The advantage function is estimated using generalized advantage estimation. For the temporal difference increment of the value network, For the reward function, These are weighting coefficients. Adjust the value network to the updated parameters. and They are respectively Time and State quantity at any given time. Adjusting the parameters of the value network to provide parameters. For the amplitude limiting function, For the purpose of referring to functions, , Let be the radius of the trust region for policy gradient updates.

2. A rocket-powered landing phase guidance and control cooperative system based on reinforcement learning, characterized in that, The system is applied to the method of claim 1, and the system comprises: The equations of motion module is used to establish the equations of motion for the rocket's powered landing phase. The guidance and control integrated target determination module is used to determine the integrated guidance and control targets for the rocket's powered landing phase. A guidance and control cooperative Markov decision process model establishment module is used to establish a guidance and control cooperative Markov decision process model based on the motion equations and the guidance and control integrated objective. The guidance and control cooperative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, an attitude control module simulation model, and motion equations. The parameter adjustment strategy network, parameter adjustment value network, guidance module simulation model, attitude control module simulation model, and motion equations form a closed-loop system. The reward function of the guidance and control cooperative Markov decision process model is established based on the guidance and control integrated objective. The optimization solution module is used to train the parameters of the parameter adjustment strategy network based on the guidance and control cooperative Markov decision process model and the nearest policy optimization algorithm to obtain the parameter adjustment strategy network with optimized parameters. The control module is used to perform guidance control of the rocket's powered landing phase based on the behavioral quantities generated by the parameter adjustment strategy network after parameter optimization, wherein the behavioral quantities are control parameters.

3. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in claim 1.

4. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed, implements the method as described in claim 1.

Images