Rocket power landing section guidance control cooperation method and system based on reinforcement learning

Abstract

The invention discloses a rocket power landing section guidance control cooperation method and system based on reinforcement learning, and belongs to the technical field of guidance control. According to the guidance control parameter adjustment method based on reinforcement learning, a parameter adjustment process is modeled as a Markov decision process, a parameter adjustment strategy network is trained by using a reinforcement learning method, and the expectation of a guidance control comprehensive target is optimal under the action of model uncertainty and interference. The precision of rocket power landing section guidance control is improved, and landing loss is reduced.

Description

Technical Field

[0001] The present invention relates to the field of guidance control technology, and in particular to a guidance control collaboration method and system for a rocket-powered landing phase based on reinforcement learning. Background Art

[0002] A reusable rocket is a rocket vehicle whose entire body or part of its sub-stage can be reused. Compared with traditional rockets, reusable rockets can reuse key components such as engines, greatly reducing the cost of a single launch, and become an important part of the future new generation of space transportation system. The powered landing phase is a key stage in the recovery process of reusable rockets. This flight phase has the characteristics of significant model uncertainty and interference, severe attitude trajectory coupling, and strict terminal constraints and fuel optimality requirements.

[0003] The existing design method adopts the idea of ​​separate design of guidance and control, ignoring the tracking dynamics of attitude control on guidance instructions in guidance design and ignoring the influence of engine swing angle on center of mass motion in attitude control design. When the model uncertainty and interference are significant, the guidance law may produce drastically changing attitude instructions and increase the tracking error of the attitude control law. At the same time, the attitude control law may produce a large engine swing angle and cause additional interference force on the rocket center of mass motion, thus causing losses in the terminal constraint satisfaction accuracy, fuel consumption optimality and attitude control quality of the powered landing phase. Summary of the invention

[0004] The purpose of the present invention is to provide a rocket-powered landing phase guidance control collaboration method and system based on reinforcement learning to improve the accuracy of rocket-powered landing phase guidance control.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] The present invention provides a rocket-powered landing phase guidance control collaboration method based on reinforcement learning, the method comprising the following steps:

[0007] Establish the equations of motion for the rocket-powered landing phase;

[0008] Determine the guidance and control integrated objectives for the rocket powered landing phase;

[0009] Based on the motion equation and the guidance and control comprehensive objective, a guidance and control collaborative Markov decision process model is established, wherein the guidance and control collaborative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, a posture control module simulation model and a motion equation; the parameter adjustment strategy network, the parameter adjustment value network, the guidance module simulation model, the posture control module simulation model and the motion equation form a closed-loop system, and a reward function of the guidance and control collaborative Markov decision process model is established based on the guidance and control comprehensive objective;

[0010] According to the guidance control collaborative Markov decision process model, the neighboring strategy optimization algorithm is used to train the parameters of the parameter adjustment strategy network to obtain the parameter-optimized parameter adjustment strategy network.

[0011] The guidance control of the rocket-powered landing phase is performed based on the behavior quantity generated by the parameter adjustment strategy network after parameter optimization, and the behavior quantity is the control parameter.

[0012] Optionally, the motion equation is:

[0013]

[0014] Where r is the rocket position; v is the rocket velocity; g is the acceleration due to gravity; m is the rocket mass; T is the magnitude of the rocket engine thrust; V ex Refers to the rocket engine exhaust velocity; F aero and F tvc are the rocket aerodynamic force and rocket engine thrust respectively; ω b Refers to the three-axis angular velocity; ψ and γ are the rocket pitch angle, yaw angle and roll angle respectively; R x (γ) is the primitive rotation matrix of the rocket roll angle around the x-axis, R y (ψ) is the primitive rotation matrix of the rocket yaw angle around the y-axis; J is the rocket moment of inertia matrix; M aero and M tvc They are the rocket aerodynamic torque and the engine thrust torque respectively.

[0015] Optionally, the reward function is:

[0016]

[0017] Among them, r(s t ,a t ) is the reward function, s t is the state quantity at time t, a t is the behavior at time t, t is the time in the rocket-powered landing phase, t f is the terminal landing time, k 1 , k 2 , k3 , k 4 , k 5 , k 6 is the weighting coefficient of different objectives in the guidance and control comprehensive objective, J r is the terminal position error, J v is the speed error, J m The difference between the mass at the beginning and end of the powered landing phase flight, is the horizontal component of the unit vector pointing to the rocket's longitudinal axis at the moment of terminal landing, J δ is the average posture tracking error, J error is the average engine swing angle.

[0018] Optionally, the method of training the parameters of the parameter adjustment strategy network using a neighboring strategy optimization algorithm according to the guidance control collaborative Markov decision process model to obtain a parameter-optimized parameter adjustment strategy network specifically includes:

[0019] Based on the guidance and control collaborative Markov decision process model, N closed-loop simulations are performed to obtain a sampling sequence; the sampling sequence includes N groups of simulation sequences, and the simulation sequence includes state quantities and behavior quantities at different times in the rocket-powered landing phase obtained by simulation;

[0020] Calculating the parameter update gradient of the parameter adjustment value network based on the sampling sequence, and updating the parameters of the parameter adjustment value network to obtain an updated parameter adjustment value network;

[0021] Based on the sampling sequence and the updated parameter adjustment value network, the parameter update gradient of the parameter adjustment strategy network is calculated, and the parameters of the parameter adjustment strategy network are updated to obtain an updated parameter adjustment strategy network;

[0022] Determine whether the maximum number of sampling steps has been reached and obtain the determination result;

[0023] If the judgment result indicates no, returning to the step of performing N closed-loop simulations based on the guidance control collaborative Markov decision process model to obtain a sampling sequence;

[0024] If the judgment result indicates yes, then the updated parameter adjustment strategy network is output as the parameter adjustment strategy network after parameter optimization.

[0025] Optionally, the step of performing N closed-loop simulations based on the guidance and control collaborative Markov decision process model to obtain a sampling sequence specifically includes:

[0026] Based on the uncertainty of state transfer, the initial state quantity is set to s 0 ~p 0 (·); where s0 is the initial state, p 0(·) Initialization function for the state quantity considering the uncertainty of state transition;

[0027] Input the state quantity at the current iteration time into the parameter adjustment strategy network to obtain the behavior quantity at the current iteration time;

[0028] Inputting the behavior quantity at the current iteration moment into the guidance module simulation model and the attitude control module simulation model for simulation, and obtaining the rocket engine thrust and rocket engine swing angle at the current iteration moment;

[0029] The thrust and swing angle of the rocket engine at the current iteration moment are input into the motion equation to obtain the state quantity at the next iteration moment, and the step of inputting the state quantity at the current iteration moment into the parameter adjustment strategy network to obtain the behavior quantity at the current iteration moment is returned until the terminal landing time is reached to obtain the simulation sequence of the current closed-loop simulation.

[0030] Optionally, the formula for calculating the parameter update gradient of the parameter adjustment value network is:

[0031]

[0032] Among them, g td The parameter update gradient of the parameter adjustment value network, s t and t+1 are the state quantities at time t and time t+1 respectively, θ v Tune the parameters of the value network for the parameters, To approximate the expectation using the arithmetic mean, s t is the state quantity at time t, a t is the amount of behavior at time t, is the sampling sequence, r(s t ,a t ) is the reward function.

[0033] Optionally, the formula for calculating the parameter update gradient of the parameter adjustment strategy network is:

[0034]

[0035]

[0036] Among them, g ppo is the parameter update gradient of the parameter adjustment policy network, To approximate the expectation using the arithmetic mean, s t is the state quantity at time t, a t is the amount of behavior at time t, is the sampling sequence, θ is the parameter of the parameter adjustment strategy network, is the gradient with respect to the parameter θ, ρ θ (st ,a t ) is the probability ratio of strategy output behavior, πθ(a t |s t ) is the parameter adjustment strategy network to be trained, is the parameter adjustment strategy network after the last gradient update, is the advantage function estimated using the generalized advantage estimator, δ(s τ ,a τ ) is the time difference increment of the value network, r(s τ ,a τ ) is the reward function, λ is the weighting coefficient, V θv (s τ+1 ) is the updated parameter adjustment value network, s τ and τ+1 are the state quantities at time τ and time τ+1, θ v is the parameter of the parameter adjustment value network, clip(x,∈) is the clipping function, x is the reference function, x=ρ θ (s t ,a t ), ∈ is the trust region radius of the policy gradient update.

[0037] A rocket-powered landing phase guidance and control collaboration system based on reinforcement learning, the system is applied to the above method, and the system includes:

[0038] The motion equation building module is used to build the motion equations for the rocket-powered landing phase;

[0039] The guidance and control integrated target determination module is used to determine the guidance and control integrated target of the rocket-powered landing phase;

[0040] A guidance control collaborative Markov decision process model establishment module is used to establish a guidance control collaborative Markov decision process model based on the motion equation and the guidance control comprehensive target, wherein the guidance control collaborative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, a posture control module simulation model and a motion equation; the parameter adjustment strategy network, the parameter adjustment value network, the guidance module simulation model, the posture control module simulation model and the motion equation form a closed-loop system, and the reward function of the guidance control collaborative Markov decision process model is established based on the guidance control comprehensive target;

[0041] An optimization solution module is used to train the parameters of the parameter adjustment strategy network using a neighboring strategy optimization algorithm according to the guidance control collaborative Markov decision process model to obtain a parameter adjustment strategy network with optimized parameters;

[0042] The control module is used to perform guidance control of the rocket-powered landing phase based on the behavior quantity generated by the parameter adjustment strategy network after parameter optimization, and the behavior quantity is the control parameter.

[0043] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0044] A computer-readable storage medium stores a computer program, and the computer program implements the above method when executed.

[0045] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] The embodiment of the present invention provides a guidance control collaboration method and system for a rocket-powered landing stage based on reinforcement learning, the method comprising: establishing a motion equation for a rocket-powered landing stage; determining a comprehensive guidance control target for the rocket-powered landing stage; establishing a guidance control collaboration Markov decision process model based on the motion equation and the comprehensive guidance control target; according to the guidance control collaboration Markov decision process model, using a neighboring strategy optimization algorithm to train the parameters of a parameter adjustment strategy network to obtain a parameter-optimized parameter adjustment strategy network; and performing guidance control for the rocket-powered landing stage based on the behavior quantity generated by the parameter adjustment strategy network after parameter optimization. In summary, the present invention proposes a guidance control parameter adjustment method based on reinforcement learning, which models the parameter adjustment process as a Markov decision process, uses a reinforcement learning method to train a parameter adjustment parameter adjustment strategy network, and achieves the expected optimality of the comprehensive guidance control target under model uncertainty and interference, thereby improving the accuracy of guidance control for the rocket-powered landing stage and reducing landing losses. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A flowchart of a rocket-powered landing phase guidance and control collaboration method based on reinforcement learning provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of the network parameter optimization of the parameter adjustment strategy provided by the embodiment of the present invention;

[0050] Figure 3A comparison diagram of the landing point distribution of the guidance control law designed by the control cooperation method provided in an embodiment of the present invention and the guidance control law designed by the nominal model discretely. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The purpose of the present invention is to provide a rocket-powered landing phase guidance control collaboration method and system based on reinforcement learning to improve the accuracy of rocket-powered landing phase guidance control.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] An embodiment of the present invention provides a guidance and control collaboration method for a rocket-powered landing phase based on reinforcement learning, the method comprising the following steps:

[0055] Establish the equations of motion for the rocket-powered landing phase.

[0056] Determine the integrated guidance and control objectives for the rocket powered landing phase.

[0057] Based on the motion equation and the guidance and control comprehensive objective, a guidance and control collaborative Markov decision process model is established, and the guidance and control collaborative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, a posture control module simulation model and a motion equation; the parameter adjustment strategy network, the parameter adjustment value network, the guidance module simulation model, the posture control module simulation model and the motion equation form a closed-loop system, and the reward function of the guidance and control collaborative Markov decision process model is established based on the guidance and control comprehensive objective.

[0058] According to the guidance control collaborative Markov decision process model, the neighboring strategy optimization algorithm is used to train the parameters of the parameter adjustment strategy network to obtain the parameter-optimized parameter adjustment strategy network.

[0059] The guidance control of the rocket-powered landing phase is performed based on the behavior quantity generated by the parameter adjustment strategy network after parameter optimization, and the behavior quantity is the control parameter.

[0060] like Figure 1As shown, the establishment of the motion equation of the rocket-powered landing stage and the determination of the guidance control comprehensive target of the rocket-powered landing stage in the embodiment of the present invention belong to the step of establishing the guidance control comprehensive target optimization problem. The establishment of the guidance control collaborative Markov decision process model based on the motion equation and the guidance control comprehensive target in the embodiment of the present invention belongs to the step of establishing the guidance control collaborative Markov decision process; the use of the neighboring strategy optimization algorithm to train the parameters of the parameter adjustment strategy network according to the guidance control collaborative Markov decision process model in the embodiment of the present invention to obtain the parameter adjustment strategy network with optimized parameters belongs to the step of modular collaborative framework design; the behavior quantity generated by the parameter adjustment strategy network after parameter optimization in the embodiment of the present invention is used to perform guidance control of the rocket-powered landing stage, and the behavior quantity is the control parameter, which belongs to the step of parameter adjustment strategy migration.

[0061] Step 1: Establish the comprehensive objective optimization problem of guidance and control;

[0062] The embodiment of the present invention establishes a comprehensive target optimization problem model for guidance and control of the powered landing phase according to the overall parameters of the rocket, guidance and control performance indicators, model uncertainty and interference model.

[0063] First, the motion equations of the rocket-powered landing phase are established. The motion equations of the center of mass and the motion equations around the center of mass are as follows:

[0064]

[0065] Where r is the rocket position; v is the rocket velocity; g is the acceleration due to gravity; m is the rocket mass; T is the magnitude of the rocket engine thrust; V ex Refers to the rocket engine exhaust velocity; F aero and F tvc are the rocket aerodynamic force and the rocket engine thrust, respectively, where F tvc is the rocket engine thrust T and the rocket engine swing angle δ ψ Function of ω b Refers to the three-axis angular velocity; ψ and γ are the rocket pitch angle, yaw angle and roll angle respectively; R a (b) represents the primitive rotation matrix; J is the rocket moment of inertia matrix; M aero and M tvc are the rocket aerodynamic torque and the engine thrust torque respectively, where M tvc is the rocket engine thrust T and the rocket engine swing angle δ ψ function.

[0066] At the same time, a mathematical model of the comprehensive guidance and control objectives is established.

[0067] The terminal position and velocity errors are used to characterize the accuracy of the terminal center of mass state constraint satisfaction, that is:

[0068]

[0069] where t f is the terminal landing time;

[0070] The optimality of fuel consumption is characterized by the difference in mass between the beginning and end of the powered landing phase flight, that is,

[0071] J m =m(0)-m(t f ) (3)

[0072] The horizontal component of the unit vector pointing to the rocket's longitudinal axis at the terminal moment is used to represent the accuracy of the constraint satisfaction of the terminal state quantity around the center of mass, that is:

[0073]

[0074] The average attitude tracking error and the average engine swing angle are used to characterize the attitude control quality, that is,

[0075]

[0076] where dt c is the attitude control cycle; and ψ c is the attitude command generated by the guidance law. In summary, the comprehensive guidance control objective of the powered landing phase can be expressed in the following vector form:

[0077]

[0078] Step 2: Establishment of guidance and control collaborative Markov decision process;

[0079] According to the guidance and control comprehensive target optimization problem model, the embodiment of the present invention establishes a closed-loop system consisting of a rocket motion equation, a parameterized guidance module simulation model, a parameterized attitude control module simulation model and a parameter adjustment strategy, and models the system as a Markov decision process model, that is, the specific method of establishing the model is as follows: discretize the rocket motion equation, and select the parameter update time interval of the guidance module simulation model and the attitude control module simulation model, establish the state transfer equation of the Markov decision process according to the closed-loop system consisting of the rocket motion equation, the guidance module simulation model and the attitude control module simulation model, select the state quantity and behavior quantity in the state transfer, and obtain the reward function by weighting the components of the guidance and control comprehensive target, which is specifically:

[0080] Use Figure 2The reinforcement learning training framework shown in the figure is used to design the parameter cooperation strategy, and the closed-loop system consisting of the parameter adjustment strategy (including the parameter adjustment strategy network and the parameter adjustment value network), the guidance module simulation model, the attitude control module simulation model and the powered landing motion equation is modeled as a guidance control cooperation Markov decision process (MDP) model. Under this model, the guidance module simulation model and the attitude control module simulation model can use any existing method with adjustable parameters to convert the state quantity s of the guidance control cooperation Markov decision process model t Select as the measurable state quantity in the powered landing motion equation, the behavior quantity a t Selected as the adjustable parameters of the guidance and attitude control module simulation model, the parameter adjustment strategy is the mapping strategy from MDP state quantity to behavior quantity.

[0081] Given the current state s t and behavior amount a t , the state transition of the guidance and control collaborative Markov decision process model is determined by the closed-loop system composed of the guidance module simulation model, the attitude control module simulation model and the motion equation. Specifically, the parameter update period dt of the given parameter adjustment strategy gnc , guidance period dt g and attitude control period dt c , the MDP state quantity s at the next parameter update time t+1 It can be generated by the simultaneous integration of the guidance module simulation model expression, the attitude control module simulation model expression and the powered landing motion equation. A complete round (episode) of the guidance control collaborative Markov decision process model is from t = 0 to t = t f In addition, the uncertainty of state transition includes the wind field, parameter perturbation ratio and the initial state dispersion of powered landing. Therefore, in each round of MDP, the wind field, parameter perturbation and rocket initial state are randomly generated within a given range. For the convenience of description, the dispersion of the initial state of the guidance and control cooperative Markov decision process model is denoted as s 0 ~p 0 (·), the new state distribution after state transfer is recorded as s t+1 ~p(·|s t ,a t ).

[0082] Finally, the reward function in the guidance and control collaborative Markov decision process model is generated by weighting each component in the integrated objective J of the powered landing phase, that is,

[0083]

[0084] where k 1 -k 6 is the weighting coefficient.

[0085] Based on the above guidance control cooperative Markov decision process model, the parameter adjustment strategy to be designed is the mapping from MDP state quantity to MDP behavior quantity, which is denoted as a t ~π(·|s t ). Furthermore, the optimization goal of this strategy can be described as maximizing the expected cumulative reward of the round obtained by the guidance control cooperative Markov decision process model, that is,

[0086]

[0087] Step 3: Modular collaboration framework design;

[0088] According to the guidance control collaborative Markov decision process model, the embodiment of the present invention adopts a neighboring strategy optimization algorithm to train the parameters of the parameter adjustment strategy network to obtain the parameter adjustment strategy network after parameter optimization.

[0089] The optimal strategy π shown in equation (8) (i.e., parameter adjustment strategy network) is fitted using a neural network parameter adjustment strategy * , and record it as π θ , where θ is the neural network parameter to be optimized, then equation (8) can be rewritten as:

[0090]

[0091] In view of this neural network parameter optimization problem, the neighboring strategy optimization algorithm is used to update the neural network parameters by gradient. In each gradient calculation, the strategy after the last gradient update is first used. Generate the guidance and attitude control module simulation model parameters, perform N closed-loop simulations with the guidance module simulation model, attitude control module simulation model and the motion equation of powered landing, and obtain N rounds of state and behavior sampling sequences, which are recorded as Then the update gradient of the parameter adjustment strategy network parameters can be estimated as follows:

[0092]

[0093] in Indicates the use of arithmetic mean to approximate expectations; ∈ is the trust region radius of the policy gradient update; is the advantage function estimated using the generalized advantage estimator It is expressed as:

[0094]

[0095] Where λ is the weighting coefficient; θ v The neural network with the value function of the parameter is also updated by It is estimated that it can be expressed as:

[0096]

[0097] Among them, g td The parameter update gradient of the parameter adjustment value network, s t and t+1 are the state quantities at time t and time t+1 respectively, θ v Tune the parameters of the value network for the parameters, To approximate the expectation using the arithmetic mean, s t is the state quantity at time t, a t is the amount of behavior at time t, is the sampling sequence, r(s t ,a t ) is the reward function.

[0098] In summary, the complete parameter adjustment strategy optimization algorithm can be summarized into the following steps:

[0099] Input: single sampling round number N, maximum sampling step number N max , the policy network parameter update step α θ , the value network parameter update step Weighting coefficient λ and trust region radius ∈.

[0100] Step 1: Initialize the parameter adjustment strategy network π θ , sampling strategy network and parameter-adjusted value network Among them, the sampling strategy network is the parameter adjustment strategy network after the last gradient update;

[0101] Step 2: Sampling strategy network Generate behavior quantity, that is, control parameter, and interact with the guidance control collaborative Markov decision process model for N rounds to generate sampling sequence of state quantity and behavior quantity

[0102] Step 3: According to the sampling sequence Calculate the parameter update gradient g of the parameter adjustment value network v , update the parameters to adjust the value network:

[0103] Step 4: According to the sampling sequence and the current parameter adjustment value network Calculate the parameter update gradient g of the parameter adjustment strategy network ppo , update the policy network: θ←θ-α θ g ppo ;

[0104] Step 5: Update the parameters of the sampling strategy network: θ old ←θ;

[0105] Step 6: If the maximum number of sampling steps N is reached max , the algorithm ends; otherwise, return to step 2.

[0106] Output: parameter adjustment strategy network π θ

[0107] Step 4: Migrate parameter adjustment strategy;

[0108] The embodiment of the present invention solidifies the parameter adjustment strategy network after the training and parameter optimization, and realizes a closed loop with the input and output interfaces of the rocket's motion equation, the parameterized guidance module simulation model, and the parameterized attitude control module simulation model, as a complete power landing phase guidance control law, that is, the network parameters of the parameter adjustment neural network strategy obtained by reinforcement learning training are fixed, the state quantity output by the rocket motion equation is used as the input of the parameter adjustment neural network strategy, and after the forward propagation of the neural network, the output quantity of the parameter adjustment neural network strategy is used as the parameter value of the guidance control module, and the guidance control module under the current parameter value generates the rocket control quantity, forming a control closed loop, specifically:

[0109] The network parameters of the parameter adjustment strategy are taken as the optimal values ​​obtained by the reinforcement learning training convergence, and the state quantity s is obtained by numerically integrating the rocket motion equation. t , the state quantity s t As the input of the parameter adjustment strategy network, the behavior a is generated through the forward propagation of the neural network t , which are used as the guidance module simulation model and attitude control module simulation model parameters, and the guidance module simulation model and attitude control module simulation model under the current parameter values ​​generate the engine thrust T and rocket engine swing angle δ ψ , substitute it into the rocket motion equation and integrate it to get the new rocket state quantity, thus realizing the control closed loop.

[0110] In order to illustrate the effect of the method provided by the embodiment of the present invention, the embodiment of the present invention provides the following simulation case.

[0111] This case uses a numerical simulation case as a method demonstration, not an actual flight mission. The nominal values ​​of the rocket model parameters are: Reference cross-sectional area S ref 1m 2 ; Engine exhaust speed V ex 2700m·s -1 ; Maximum engine thrust is 40kN; Minimum thrust is 20kN; X, Y, Z axis inertia is 300kg·m respectively 2 、6000kg·m2 and 6000kg·m 2 The atmospheric model uses the American 1976 standard atmospheric model. Considering the random dispersion of the initial position, velocity and attitude of rocket-powered landing, the model errors of aerodynamic forces and aerodynamic moments, and the typical random wind field interference in the literature, multiple Monte Carlo simulations are performed.

[0112] According to the implementation process, the comparison of the drop point distribution of the guidance control law designed by the method provided in the embodiment of the present invention and the discrete design nominal guidance control law without parameter adjustment strategy cooperation is obtained. Figure 3 The comparison of the maximum values ​​of each component of the guidance control comprehensive objective is shown in Table 1. It can be seen that the modular collaborative guidance control law achieves the performance improvement of each component of the comprehensive objective compared with the discrete design, which can prove that this method can optimize the guidance control comprehensive objective compared with the discrete design under model uncertainty and interference.

[0113] Table 1 Maximum values ​​of the guidance control integrated target components

[0114]

[0115] Example 2

[0116] Embodiment 2 of the present invention provides a rocket-powered landing phase guidance and control collaboration system based on reinforcement learning, the system is applied to the above method, and the system includes:

[0117] The motion equation building module is used to build the motion equations for the rocket-powered landing phase;

[0118] The guidance and control integrated target determination module is used to determine the guidance and control integrated target of the rocket-powered landing phase;

[0119] A guidance control collaborative Markov decision process model establishment module is used to establish a guidance control collaborative Markov decision process model based on the motion equation and the guidance control comprehensive target, wherein the guidance control collaborative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, a posture control module simulation model and a motion equation; the parameter adjustment strategy network, the parameter adjustment value network, the guidance module simulation model, the posture control module simulation model and the motion equation form a closed-loop system, and the reward function of the guidance control collaborative Markov decision process model is established based on the guidance control comprehensive target;

[0120] An optimization solution module is used to train the parameters of the parameter adjustment strategy network using a neighboring strategy optimization algorithm according to the guidance control collaborative Markov decision process model to obtain a parameter adjustment strategy network with optimized parameters;

[0121] The control module is used to perform guidance control of the rocket-powered landing phase based on the behavior quantity generated by the parameter adjustment strategy network after parameter optimization, and the behavior quantity is the control parameter.

[0122] Example 3

[0123] Embodiment 3 of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0124] Example 4

[0125] Embodiment 4 of the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the above method when executed.

[0126] In summary, the implementation of the embodiments of the present invention has the following beneficial effects:

[0127] (1) The embodiment of the present invention obtains a guidance and control parameter adjustment strategy for the rocket-powered landing phase based on a reinforcement learning method, which solves the existing problems of severe attitude-orbit coupling, low accuracy in satisfying terminal constraints, high fuel consumption, and poor attitude control quality under the influence of model uncertainty and interference, and is applicable to any rocket-powered landing motion model.

[0128] (2) The embodiment of the present invention adopts a proximal strategy optimization algorithm to optimize the parameter adjustment strategy and perform real-time parameter adjustment on the guidance and control module with a fixed structure, thereby avoiding the problem of poor adaptability to model uncertainty and interference faced by manually designed parameter adjustment laws, and achieving good guidance and control comprehensive target optimization effect.

[0129] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0130] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A rocket powered landing phase guidance control collaboration method based on reinforcement learning, characterized in that: The method comprises the following steps: Establish the equations of motion for the rocket-powered landing phase; Determine the guidance and control integrated objectives for the rocket powered landing phase; Based on the motion equation and the guidance and control comprehensive objective, a guidance and control collaborative Markov decision process model is established, wherein the guidance and control collaborative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, a posture control module simulation model and a motion equation; the parameter adjustment strategy network, the parameter adjustment value network, the guidance module simulation model, the posture control module simulation model and the motion equation form a closed-loop system, and a reward function of the guidance and control collaborative Markov decision process model is established based on the guidance and control comprehensive objective; According to the guidance control collaborative Markov decision process model, the neighboring strategy optimization algorithm is used to train the parameters of the parameter adjustment strategy network to obtain the parameter-optimized parameter adjustment strategy network. The guidance control of the rocket-powered landing phase is performed based on the behavior quantity generated by the parameter adjustment strategy network after parameter optimization, and the behavior quantity is the control parameter.

2. The rocket powered landing phase guidance control collaboration method based on reinforcement learning according to claim 1 is characterized in that: The equation of motion is: Where r is the rocket position; v is the rocket velocity; g is the acceleration due to gravity; m is the rocket mass; T is the magnitude of the rocket engine thrust; V ex Refers to the rocket engine exhaust velocity; F aero and F tvc are the rocket aerodynamic force and rocket engine thrust respectively; ω b Refers to the three-axis angular velocity; ψ and γ are the rocket pitch angle, yaw angle and roll angle respectively; R x (γ) is the primitive rotation matrix of the rocket roll angle around the x-axis, R y (ψ) is the primitive rotation matrix of the rocket yaw angle around the y-axis; J is the rocket moment of inertia matrix; M aero and M tvc They are the rocket aerodynamic torque and the engine thrust torque respectively.

3. The rocket powered landing phase guidance control collaboration method based on reinforcement learning according to claim 1 is characterized in that: The reward function is: Among them, r(s t , a t ) is the reward function, s t is the state quantity at time t, a t is the behavior at time t, t is the time in the rocket-powered landing phase, t f is the terminal landing time, k1, k2, k3, k4, k5, k6 are the weighting coefficients of different targets in the guidance and control integrated target, J r is the terminal position error, J v is the speed error, J m The difference between the mass at the beginning and end of the powered landing phase flight, is the horizontal component of the unit vector pointing to the rocket's longitudinal axis at the moment of terminal landing, J δ is the average posture tracking error, J error is the average engine swing angle.

4. According to the reinforcement learning-based rocket powered landing phase guidance control collaboration method of claim 1, the parameters of the parameter adjustment strategy network are trained using a neighboring strategy optimization algorithm according to the guidance control collaboration Markov decision process model to obtain a parameter-optimized parameter adjustment strategy network, specifically comprising: Based on the guidance and control collaborative Markov decision process model, N closed-loop simulations are performed to obtain a sampling sequence; The sampling sequence includes N groups of simulation sequences, and the simulation sequence includes state quantities and behavior quantities at different moments in the rocket-powered landing phase obtained by simulation; Calculating the parameter update gradient of the parameter adjustment value network based on the sampling sequence, and updating the parameters of the parameter adjustment value network to obtain an updated parameter adjustment value network; Based on the sampling sequence and the updated parameter adjustment value network, the parameter update gradient of the parameter adjustment strategy network is calculated, and the parameters of the parameter adjustment strategy network are updated to obtain an updated parameter adjustment strategy network; Determine whether the maximum number of sampling steps has been reached and obtain the determination result; If the judgment result indicates no, returning to the step of performing N closed-loop simulations based on the guidance control collaborative Markov decision process model to obtain a sampling sequence; If the judgment result indicates yes, then the updated parameter adjustment strategy network is output as the parameter adjustment strategy network after parameter optimization.

5. The rocket powered landing phase guidance control collaboration method based on reinforcement learning according to claim 4 is characterized in that: The closed-loop simulation is performed N times based on the guidance and control collaborative Markov decision process model to obtain a sampling sequence, specifically including: Based on the uncertainty of state transition, the initial state quantity is set to s0~p0(·); where s0 is the initial state quantity, and p0(·) is the state quantity initialization function considering the uncertainty of state transition; Input the state quantity at the current iteration time into the parameter adjustment strategy network to obtain the behavior quantity at the current iteration time; Inputting the behavior quantity at the current iteration moment into the guidance module simulation model and the attitude control module simulation model for simulation, and obtaining the rocket engine thrust and rocket engine swing angle at the current iteration moment; The thrust and swing angle of the rocket engine at the current iteration moment are input into the motion equation to obtain the state quantity at the next iteration moment, and the step of inputting the state quantity at the current iteration moment into the parameter adjustment strategy network to obtain the behavior quantity at the current iteration moment is returned until the terminal landing time is reached to obtain the simulation sequence of the current closed-loop simulation.

6. The rocket powered landing phase guidance control collaboration method based on reinforcement learning according to claim 4 is characterized in that: The formula for calculating the parameter update gradient of the parameter adjustment value network is: Among them, g td The parameter update gradient of the parameter adjustment value network, s t and t+1 are the state quantities at time t and time t+1 respectively, θ v Tune the parameters of the value network for the parameters, To approximate the expectation using the arithmetic mean, s t is the state quantity at time t, a t is the amount of behavior at time t, is the sampling sequence, r(s τ ,a τ ) is the reward function.

7. The rocket powered landing phase guidance control collaboration method based on reinforcement learning according to claim 4 is characterized in that: The formula for calculating the parameter update gradient of the parameter adjustment strategy network is: Among them, g ppo is the parameter update gradient of the parameter adjustment policy network, To approximate the expectation using the arithmetic mean, s t is the state quantity at time t, a t is the amount of behavior at time t, is the sampling sequence, θ is the parameter of the parameter adjustment strategy network, is the gradient with respect to the parameter θ, ρ θ (s t ,a t ) is the probability ratio of strategy output behavior, π θ (a t |s t ) is the parameter adjustment strategy network to be trained, is the parameter adjustment strategy network after the last gradient update, is the advantage function estimated using the generalized advantage estimator, δ(s τ ,a τ ) is the time difference increment of the value network, r(s τ ,a τ ) is the reward function, λ is the weighting coefficient, Adjust the value network for the updated parameters, s τ and τ+1 are the state quantities at time τ and time τ+1, θ v is the parameter of the parameter adjustment value network, clip(x,∈) is the clipping function, x is the reference function, x=ρ θ (s t ,a t ), ∈ is the trust region radius of the policy gradient update.

8. A rocket-powered landing phase guidance and control collaboration system based on reinforcement learning, characterized in that: The system is applied to the method described in any one of claims 1 to 7, and the system comprises: The motion equation building module is used to build the motion equations for the rocket-powered landing phase; The guidance and control integrated target determination module is used to determine the guidance and control integrated target of the rocket-powered landing phase; A guidance control collaborative Markov decision process model establishment module is used to establish a guidance control collaborative Markov decision process model based on the motion equation and the guidance control comprehensive target, wherein the guidance control collaborative Markov decision process model includes a parameter adjustment strategy network, a parameter adjustment value network, a guidance module simulation model, a posture control module simulation model and a motion equation; the parameter adjustment strategy network, the parameter adjustment value network, the guidance module simulation model, the posture control module simulation model and the motion equation form a closed-loop system, and the reward function of the guidance control collaborative Markov decision process model is established based on the guidance control comprehensive target; An optimization solution module is used to train the parameters of the parameter adjustment strategy network using a neighboring strategy optimization algorithm according to the guidance control collaborative Markov decision process model to obtain a parameter adjustment strategy network with optimized parameters; The control module is used to perform guidance control of the rocket-powered landing phase based on the behavior quantity generated by the parameter adjustment strategy network after parameter optimization, and the behavior quantity is the control parameter.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which implements the method according to any one of claims 1 to 7 when executed.

Images