A rocket projectile attitude self-adaptive control method and system based on reinforcement learning

Abstract

The application discloses a rocket attitude self-adaptive control method and system based on reinforcement learning, and belongs to the technical field of rocket attitude control. The method constructs an adaptive controller for controlling the rocket attitude through a backstepping method, and the adaptive controller is used for realizing the self-adaptation of the rocket to unknown disturbances based on an Actor-Critic (AC) of reinforcement learning, so that the system can rapidly converge to a desired state, unknown nonlinear observation is realized without depending on an internal model, and the tracking control of the rocket attitude is realized.

Description

Technical Field

[0001] This invention belongs to the field of rocket attitude control technology, and relates to a rocket attitude adaptive control method and system based on reinforcement learning. Background Technology

[0002] The guidance and precision of rockets have become a research hotspot for long-range precision strikes. However, during the flight of a rocket, the mass, velocity, and aerodynamic coefficient of the rocket body are constantly changing, resulting in rapid changes in the rocket body parameters, highly nonlinear dynamic characteristics of the rocket body, and the presence of uncertainties. Furthermore, the various flight parameters are mutually coupled and change drastically over time, making it a time-varying system with strong coupling of multiple variables, which makes it difficult to adaptively control the attitude of the rocket. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a rocket attitude adaptive control method and system based on reinforcement learning, so as to solve the technical problem that the rocket attitude is difficult to adaptively control in the prior art.

[0004] To achieve the above objectives, the present invention employs the following technical solution:

[0005] A rocket attitude adaptive control method based on reinforcement learning includes the following steps:

[0006] S1. Establish the rocket's attitude kinematics model and dynamics model, and transform both the kinematics model and dynamics model into nonlinear models using the inverse step method. The nonlinear model includes a first-order subsystem and a second-order subsystem, and contains unknown disturbances and control inputs.

[0007] S2, construct the virtual control law of the first-order subsystem and the tracking error derivative of the second-order subsystem, wherein the tracking error contains an unknown disturbance;

[0008] S3, an adaptive controller based on reinforcement learning is obtained by fitting an unknown disturbance to an execution neural network, and the result of the execution neural network is evaluated by an evaluation neural network, which approximates the evaluation using a penalty function; the stability of the adaptive controller is proven by Lyapunov.

[0009] Both the execution neural network and the evaluation neural network obtain the update rate through the gradient descent method. The update objective of the execution neural network is to minimize the benefit error, and the update objective of the evaluation neural network is to minimize the mean square error of the residual.

[0010] S4, the adaptive controller rocket outputs a control signal, which is used to control the yaw angle and pitch angle to achieve rocket attitude adaptation.

[0011] A further improvement of the present invention is that:

[0012] Preferably, in S1, the first-order subsystem and the second-order subsystem are respectively:

[0013]

[0014] Where, f1(x1)=(0 0) T f2(x1,x2), g1(x1), and g2(x1,x2) are nonlinear matrices, u is the control signal, d is the unknown disturbance, x1 = (ψ θ), where ψ and θ represent the rocket's yaw and pitch angles, respectively, and x2 = (ω y4 ω z4 ), ω y4 ,ω z4 Let be the angular velocity of the rocket orbiting the rocket body in the projectile coordinate system.

[0015] Preferably, in S3, the virtual control of the first subsystem is as follows:

[0016] x 2c =g1(x1) -1 (-k1e1-f1(x1)+x 1d ),k1>0 (13)

[0017] Where, x 2c Let g1(x1) be the expected value, g1(x1) be the abbreviation for the nonlinear matrix, e1 be the tracking error of the rocket's yaw and pitch angles, and x be the expected value. 1d This represents the expected tracking values ​​for the rocket's yaw and pitch angles.

[0018] Preferably, in S2, the derivative of the tracking error of the second-order subsystem is:

[0019]

[0020] in, The derivative of the state variable. Let d be the derivative of the virtual control quantity, and d be the unknown disturbance. This is an estimate of the unknown interference.

[0021] Preferably, in S3, the execution neural network fits the unknown disturbance as follows:

[0022]

[0023] in,

[0024] Preferably, in S3, the update rate of the execution neural network is:

[0025]

[0026] Where Γ is the learning rate of the neural network, and Ω is the gain coefficient.

[0027] Preferably, in S3, the penalty function is approximately:

[0028]

[0029] Among them, the basis functions are bounded and satisfy ||Φ c ||≤Φ cM .

[0030] Preferably, the update rate of the evaluation neural network is:

[0031]

[0032] in,

[0033] Preferably, in S2, the control law for controlling the rocket is:

[0034]

[0035] Among them, g2 -1 The (g1e1) term is the compensation for the angle tracking error e1.

[0036] A rocket attitude adaptive control system based on reinforcement learning, comprising:

[0037] The nonlinear module is used to establish the kinematic and dynamic models of the rocket's attitude, transforming both the kinematic and dynamic models into nonlinear models using the inverse step method. The nonlinear model includes a first-order subsystem and a second-order subsystem, and contains unknown disturbances and control inputs.

[0038] The subsystem module is used to construct the virtual control law of the first-order subsystem and the tracking error derivative of the second-order subsystem, wherein the tracking error contains unknown disturbances;

[0039] The perturbation fitting module is used to fit an unknown perturbation by executing a neural network to obtain an adaptive controller based on reinforcement learning. The result of the executing neural network is evaluated by an evaluation neural network, which approximates the evaluation using a penalty function. The stability of the adaptive controller is proven by Lyapunov.

[0040] Both the execution neural network and the evaluation neural network obtain the update rate through the gradient descent method. The update objective of the execution neural network is to minimize the benefit error, and the update objective of the evaluation neural network is to minimize the mean square error of the residual.

[0041] The control module is used to output control signals to the rocket through the adaptive controller. The control signals are used to control the yaw angle and pitch angle to achieve adaptive attitude of the rocket.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] This invention relates to a reinforcement learning-based adaptive control algorithm for rocket attitude, belonging to the field of rocket attitude control research. It utilizes an Actor-Critic (AC) structure based on reinforcement learning, consisting of an action network and an evaluation network. The evaluation network outputs an evaluation value for the rocket's state based on its current state, while the action network generates the corresponding elevator deflection angle based on the evaluation value output by the evaluation network. This achieves stable attitude control of the rocket without relying on an internal model. This method can achieve fitting estimation of time-varying disturbances based on reinforcement learning and control the system state, which converges rapidly to the desired state. This invention, through the reinforcement learning Actor-Critic structure, achieves tracking control of rocket attitude by handling unknown nonlinear observations without relying on an internal model. Attached Figure Description

[0044] Figure 1 This is a flowchart of the present invention;

[0045] Figure 2 This is the curve showing the change in state x according to the present invention;

[0046] Figure 3 To control the curve of the input u;

[0047] Figure 4 The graph shows the comparison between the reinforcement learning estimate d1 and the actual value.

[0048] Figure 5 The graph shows the comparison between the estimated value d2 and the actual value for reinforcement learning. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings:

[0050] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. The terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, unless otherwise explicitly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection or a detachable connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0051] The first aspect of this invention discloses a rocket attitude adaptive control method based on reinforcement learning, the method comprising the following steps:

[0052] Step 1: Establish and define the rocket's attitude dynamics model.

[0053] The equations of motion for a rolling rocket include: the dynamic and kinematic equations of the center of mass, and the dynamic and kinematic equations of the rocket body rotating around its center of mass. The following assumptions are made regarding the attitude changes of the rolling rocket in this invention:

[0054] 1. Assume the Earth is flat and stationary during the rocket's motion.

[0055] 2. Rockets are considered as rigid bodies, and their deformation is not taken into account.

[0056] Choose state variable x2=(ω y4 ω z4 ), ω y4 ,ω z4 The angular velocity of the rocket around its body in the projectile coordinate system; control input u = (δ y δ z ) represents the control torque, δ y δ z Let y be the rudder angle; the system output is y = x1 = (ψ θ), where ψ and θ represent the rocket's yaw and pitch angles, respectively. The rocket's attitude motion and dynamic equations are established as follows:

[0057]

[0058] Among them, J x4 J y4 J z4 M is the moment of inertia of the projectile axis.y4 M z4 ΔM represents the sum of the projectile's static moment and damping moment along the y-axis and z-axis, respectively; y ,ΔM z These represent the external disturbance torques such as gusts along the y and z axes, respectively; γ, ω x4 These are the projectile's longitudinal axis rotation angle and angular velocity, respectively. The control coefficient is used. Simplifying the above model, we choose the state variable x2 = (ω) y4 ω z4 And abstract the following nonlinear models, corresponding to the first-order subsystem and the second-order subsystem respectively:

[0059]

[0060] in,

[0061]

[0062] Where: state x1=(ψ θ) represents the rocket's yaw angle and pitch angle, x2=(ω y4 ω z4 Let f1(x1) be the angular velocity of the rocket around the projectile in the projectile's coordinate system, and f1(x1) = (0 0). T f2(x1,x2), g1(x1), g2(x1,x2) are abbreviations for the nonlinear matrix in formula (4); d represents the combined unknown interference such as aerodynamic parameter error, model interference and external interference.

[0063] The nonlinear model (3) is the anti-interference system for the rocket's rolling attitude. The control signal u controls the output state x through state feedback. In this invention, d is considered as an external unknown interference. For external slow-varying interference, the unknown interference d can be overcome by adaptively adjusting the input control signal u. However, for complex high-frequency interference d, a reinforcement learning adaptive controller needs to be designed. Therefore, a neural network model (execution network) is designed to fit the unknown interference d in order to compensate for the external interference.

[0064] The task of the control system of this invention is to achieve the target angle command y of the guidance system by adaptively adjusting the input control signal u, without exceeding the control quantity of the control system, i.e., the maximum torque provided by the rudder deflection angle. d Fast and stable tracking.

[0065] Step 2: Define the tracking errors of the rocket's pitch and yaw angles, and define the performance indicators that characterize the tracking errors.

[0066] According to the rocket roll attitude control system model (3), the system output is y = x1 = (ψ θ), where ψ and θ represent the rocket's yaw angle and pitch angle, respectively. Therefore, the tracking errors of the rocket's yaw and pitch angles can be defined as:

[0067] e1 = x1 - x 1d (5)

[0068] Where, x 1d x1 represents the expected tracking values ​​of the rocket's yaw and pitch angles, and x1 represents the yaw and pitch angles output by the system.

[0069] Since the purpose of this study is to track and control posture, performance metrics for reinforcement learning can be defined as follows:

[0070]

[0071] in, Q is a positive definite matrix, representing the cumulative error of the control system.

[0072] Step 3, Design of the backstepping controller

[0073] Consider the attitude motion model with uncertainty constructed by equation (3), and make the following assumptions.

[0074] Assumption 1, trace instruction y d Its second derivative exists and is bounded.

[0075] Assumption 2, Uncertainty Bounded, that is It is a positive constant with an unknown upper bound.

[0076] The basic idea of ​​backstepping design is to decompose a complex nonlinear system into subsystems whose order does not exceed the system's order. Then, Lyapunov functions and intermediate virtual control variables are designed for each subsystem, and this process is repeated until the entire system is designed. By comprehensively considering the control law and the adaptive law, the entire closed-loop system can meet the desired dynamic and static performance indicators.

[0077] Define e1 = x1 - x 1d e² = x² - x 2c y d =x d =(ψθ) T It is the system's desired angle command.

[0078] Taking the derivative with respect to the error e1, we get:

[0079]

[0080] Design virtual control variables:

[0081] x 2c =g1 -1 (y d -f1-k1e1) (8)

[0082] when To avoid the singularity of g1, we take θ = arcsin2(q1q2 + q0q3).

[0083] Substituting (8) into (7) yields:

[0084]

[0085] Taking the derivative with respect to e², we get:

[0086]

[0087] Reinforcement learning executes network output An estimate of the unknown disturbance d can be obtained. If we design a controller-type control input u, then we have:

[0088]

[0089] The control law is designed as follows:

[0090]

[0091] Among them g2 -1 The (g1e1) term is the compensation for the angle tracking error e1.

[0092] Step 4: Based on the backstepping method, design the evaluation network and action network structure.

[0093] For the first-order system Design a virtual control law, as shown in the following equation:

[0094] x 2c =g1(x1) -1 (-k1e1-f1(x1)+x 1d ),k1>0 (13)

[0095] Input virtual control x 2c Substituting into the first-order subsystem, we get:

[0096]

[0097] Define the second-order subsystem The tracking error e2 = x2 - x 2c Then, taking the derivative of the tracking error of the second-order system, we can obtain...

[0098] Control input Substituting the values ​​into the second-order subsystem calculation, the derivative of the tracking error is obtained as follows:

[0099]

[0100] Due to the tracking error of the second-order subsystem Equation (15) can be regarded as

[0101]

[0102] in, This represents the fitting error for the unknown disturbance d. Since d is an unknown function in the control dynamic process, a neural network model (execution network) is designed to fit the unknown d, and the optimal control of the execution network is designed. And the neural network structure of the actual interference d.

[0103] According to neural network fitting theory, there exists a parameter W. a Make

[0104]

[0105] in, The basis functions are bounded, i.e., they satisfy ||Φ a ||≤Φ aM .definition For W a The expected error of the weights, therefore, Define H a To calculate the fitting error of the neural network, we have:

[0106]

[0107] To achieve Design the following reinforcement learning adaptive controller:

[0108]

[0109] Where k2>0 is the control gain.

[0110] Differentiating e² and substituting it into formula (10) yields:

[0111]

[0112] The integral penalty function of a control system is defined as:

[0113]

[0114] in, Q is a positive definite matrix, Q∈{Q|Q∈R,Q>0}.

[0115] Design and evaluate neural network architectures for approximating penalty functions:

[0116]

[0117] Among them, the basis functions are bounded, that is, they satisfy ||Φ c ||≤Φ cM .

[0118] In the above process, the evaluation network is used to evaluate the output of the execution network. The execution network should obtain the maximum evaluation through continuous attempts, and thus continuously interact to find the optimal solution.

[0119] Step 5: Design an evaluation of the neural network update rate and the network update rate of the execution neural network.

[0120] Construct the objective function of the effect error of the execution network module based on the influence of the control action:

[0121]

[0122] Where Ω is the gain coefficient. E a To address the effect error of the network modules, the goal of updating the neural network weights is to make E... a Therefore, the update rate of the network weights is minimized (gradient descent, derivative):

[0123]

[0124] Where Γ is the learning rate of the neural network.

[0125] Construct the residual mean square error function for evaluating neural networks:

[0126]

[0127] Among them, E c To evaluate the network residual function, the objective of updating the neural network weights is to make E... c Therefore, the update rate of the evaluation network weights is minimized, and thus obtained using the gradient descent method:

[0128]

[0129] in,

[0130] Approximation error of reinforcement learning AC network and Defined as Therefore, from (24) and (26), we can obtain:

[0131]

[0132] Where the approximation error ε c satisfy ε cM Let be a certain positive number.

[0133] Step 6: Proof of the stability of the control system based on Lyapunov.

[0134] Constructing Lyapunov functions:

[0135]

[0136] In the formula, To enhance the learning of AC neural networks to approximate the error.

[0137] Differentiating V and substituting it into equations (14) and (20), we get:

[0138]

[0139] Apply 2A to the fifth term of formula (29) T B≤||A|| 2 +||B|| 2 ,A∈R n ,B∈R n And by dealing with the inequality ||AB||≤||A||·||B||, we have:

[0140]

[0141] Substituting formula (27) into the sixth and seventh terms of formula (29), we get:

[0142]

[0143]

[0144] Substituting formulas (32), (31), and (30) into formula (29) yields:

[0145]

[0146] When k1 > 0, k2 > 1, When, satisfy According to Lyapunov's stability theorem, the target controlled object x asymptotically converges to 0, and the weight errors of the execution network and evaluation network are... Uniformity is ultimately bounded, where:

[0147]

[0148] The following simulation examples further verify this:

[0149] In the rocket simulation, the simulation parameters are set as follows: principal inertia J y4 =J z4 =4.59kg, the sum of the projectile's static moment and damping moment M y4 =M z4=4.59kg.

[0150]

[0151] Among them, J x4 J y4 J z4 M is the moment of inertia of the projectile axis. y4 M z4 ΔM represents the sum of the projectile's static moment and damping moment along the y-axis and z-axis, respectively; y ,ΔM z These represent the external disturbance torques such as gusts along the y and z axes, respectively; γ, ω x4 These are the projectile's longitudinal axis rotation angle and angular velocity, respectively. Let these be the control coefficients. Simplifying the above model, we obtain the following nonlinear system:

[0152] System matrix: f1 = (0 0) T ,

[0153]

[0154] Unknown error d = 0.1sin(x² + 5)

[0155] Controller gain: k1 = diag(2,2), k2 = diag(2.5,2.5)

[0156] Initial system values: (x1,x2,ξ1,ξ2)=(8,10,5,9)

[0157] The simulation results are as follows Figures 2-5 As shown:

[0158] Based on the simulation results:

[0159] Depend on Figure 2 Simulation results show that by reasonably adjusting the parameters, the system state stabilizes within 3 seconds at the expected value of [2, 2], proving the stability of the system.

[0160] Depend on Figures 4-5 Simulation results show that the reinforcement learning fitter converges to the actual value in about 4 seconds, and its fit converges to 0 within 4 seconds, proving the effectiveness of the reinforcement learning neural network design.

[0161] This method considers the case of time-varying disturbances in the system and presents a control method for system state feedback that includes reinforcement learning neural network design. This method can quickly eliminate time-varying disturbances and control the system state, which can quickly converge to the desired state.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A rocket attitude adaptive control method based on reinforcement learning, characterized in that, Includes the following steps: S1. Establish the rocket's attitude kinematics model and dynamics model, and transform both the kinematics model and dynamics model into nonlinear models using the inverse step method. The nonlinear model includes a first-order subsystem and a second-order subsystem, and contains unknown disturbances and control inputs. S2, construct the virtual control law of the first-order subsystem and the tracking error derivative of the second-order subsystem, wherein the tracking error derivative contains an unknown disturbance; The virtual control law of the first-order subsystem is: （13） in, x 2c As the expected value, The tracking error is due to the rocket's yaw and pitch angles. This represents the expected tracking values ​​for the rocket's yaw and pitch angles; The first-order subsystem and the second-order subsystem are respectively: （3） in, , , , Here, are nonlinear matrices, u is the control signal, and d is the unknown disturbance. , These represent the rocket's yaw and pitch angles, respectively. , Let be the angular velocity of the rocket around its body in the projectile coordinate system; The derivative of the tracking error of the second-order subsystem is: （15） in, The derivative of the state variable. Let d be the derivative of the virtual control quantity, and d be the unknown disturbance. This is an estimate of the unknown interference. S3, an adaptive controller based on reinforcement learning is obtained by fitting an unknown disturbance to an execution neural network, and the result of the execution neural network is evaluated by an evaluation neural network, which approximates the evaluation using a penalty function; the stability of the adaptive controller is proven by Lyapunov. Both the execution neural network and the evaluation neural network obtain the update rate through the gradient descent method. The update objective of the execution neural network is to minimize the benefit error, and the update objective of the evaluation neural network is to minimize the mean square error of the residual. In S3, the execution neural network fits the unknown disturbance as follows: （17） in, ; S4, the adaptive controller rocket outputs a control signal, which is used to control the yaw angle and pitch angle to achieve rocket attitude adaptation; The control law for controlling the rocket is: （12） in, The term refers to the angle tracking error. Compensation.

2. The rocket attitude adaptive control method based on reinforcement learning according to claim 1, characterized in that, In S3, the update rate of the execution neural network is: （24） in, The learning rate of the neural network. This is the gain coefficient.

3. The rocket attitude adaptive control method based on reinforcement learning according to claim 1, characterized in that, In S3, the penalty function is approximately: （22） Among them, the basis functions are bounded and satisfy the following conditions: .

4. The rocket attitude adaptive control method based on reinforcement learning according to claim 3, characterized in that, The update rate of the neural network is evaluated as follows: （26） in, .

5. A reinforcement learning-based adaptive control system for rocket attitude to implement the control method of claim 1, characterized in that, include: The nonlinear module is used to establish the kinematic and dynamic models of the rocket's attitude, transforming both the kinematic and dynamic models into nonlinear models using the inverse step method. The nonlinear model includes a first-order subsystem and a second-order subsystem, and contains unknown disturbances and control inputs. The subsystem module is used to construct the virtual control law of the first-order subsystem and the tracking error derivative of the second-order subsystem, wherein the tracking error contains unknown disturbances; The perturbation fitting module is used to fit an unknown perturbation by executing a neural network to obtain an adaptive controller based on reinforcement learning. The result of the executing neural network is evaluated by an evaluation neural network, which approximates the evaluation using a penalty function. The stability of the adaptive controller is proven by Lyapunov. Both the execution neural network and the evaluation neural network obtain the update rate through the gradient descent method. The update objective of the execution neural network is to minimize the benefit error, and the update objective of the evaluation neural network is to minimize the mean square error of the residual. The control module is used to output control signals to the rocket through the adaptive controller. The control signals are used to control the yaw angle and pitch angle to achieve adaptive attitude of the rocket.

Images