Rocket projectile attitude adaptive control method and system based on reinforcement learning

Abstract

The invention discloses a rocket projectile attitude adaptive control method and system based on reinforcement learning, and belongs to the technical field of rocket projectile attitude control, the method comprises the following steps: constructing an adaptive controller for controlling the attitude of a rocket projectile through a backstepping method, and realizing the self-adaption of a rocket to unknown disturbance in the adaptive controller through Actor-Critic (AC) based on reinforcement learning; the system can rapidly converge to an expected state, unknown nonlinearity is observed under the condition of not depending on an internal model, and tracking control over the attitude of the rocket projectile is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of rocket attitude control, and relates to a rocket attitude adaptive control method and system based on reinforcement learning. Background Art

[0002] The guidance and precision of rockets have become a research hotspot for long-range precision strikes. However, during the flight of a rocket, the mass, speed, and aerodynamic coefficients of the rocket body are constantly changing, resulting in rapid changes in the parameters of the rocket body. The dynamic characteristics of the rocket body are highly nonlinear and contain uncertain factors. In addition, the flight parameters are coupled with each other and change dramatically over time. It is a multivariable strongly coupled time-varying system, making it difficult to adaptively control the attitude of the rocket. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a rocket attitude adaptive control method and system based on reinforcement learning to solve the technical problem that the rocket attitude is difficult to adaptively control in the prior art.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A rocket attitude adaptive control method based on reinforcement learning comprises the following steps:

[0006] S1, establishing a kinematic model and a dynamic model of the rocket attitude, and transforming both the kinematic model and the dynamic model into a nonlinear model of the backstepping method, wherein the nonlinear model includes a first-order subsystem and a second-order subsystem, and the nonlinear model includes unknown interference and control input;

[0007] S2, constructing a virtual control law of the first-order subsystem and a tracking error derivative of the second-order subsystem, wherein the tracking error is provided with an unknown disturbance;

[0008] S3, fitting the unknown disturbance by executing the neural network to obtain a reinforcement learning adaptive controller, evaluating the result of executing the neural network by an evaluation neural network, wherein the evaluation neural network is evaluated by penalty function approximation; and the adaptive controller is proved to be stable by Lyapunov;

[0009] The execution neural network and the evaluation neural network both obtain update rates by using a gradient descent method, the update target of the execution neural network is to minimize the benefit error, and the update target of the evaluation neural network is to minimize the mean square error of the residual;

[0010] S4, the adaptive controller rocket outputs a control signal, and the control signal is used to control the yaw angle and the pitch angle to achieve rocket attitude adaptation.

[0011] A further improvement of the present invention is:

[0012] Preferably, in S1, the first-order subsystem and the second-order subsystem are respectively:

[0013]

[0014] Where, f1(x1)=(0 0) T , f2(x1,x2), g1(x1), g2(x1,x2) are nonlinear matrices, u is the control signal, d is the unknown interference, x1=(ψ θ), ψ, θ represent the yaw angle and pitch angle of the rocket, x2=(ω y4 ω z4 ),ω y4 ,ω z4 is the angular velocity of the rocket around the projectile in the projectile coordinate system.

[0015] Preferably, in S3, the virtual control of the first subsystem is:

[0016] x 2c =g1(x1) -1 (-k1e1-f1(x1)+x 1d ),k1>0 (13)

[0017] Among them, x 2c is the expected value, g1(x1) is the abbreviation of the nonlinear matrix, e1 is the tracking error of the rocket yaw and pitch angle, x 1d Represents the tracking expected value of the rocket's yaw and pitch angles.

[0018] Preferably, in S2, the tracking error derivative of the second-order subsystem is:

[0019]

[0020] in, is the derivative of the state variable, is the derivative of the virtual control quantity, d is the unknown disturbance, is an estimate of the unknown interference.

[0021] Preferably, in S3, the execution of the neural network to fit the unknown disturbance is:

[0022]

[0023] in,

[0024] Preferably, in S3, the update rate of the executed neural network is:

[0025]

[0026] Among them, Γ is the learning rate of the neural network, and Ω is the gain coefficient.

[0027] Preferably, in S3, the penalty function is approximated as:

[0028]

[0029] Among them, the basis function is bounded and satisfies ||Φ c ||≤Φ cM .

[0030] Preferably, the update rate of the evaluation neural network is:

[0031]

[0032] in,

[0033] Preferably, in S2, the control law for the rocket control is:

[0034]

[0035] Among them, g2 -1 The term (g1e1) compensates for the angular tracking error e1.

[0036] A rocket attitude adaptive control system based on reinforcement learning, comprising:

[0037] A nonlinear module is used to establish a kinematic model and a dynamic model of a rocket attitude, and to transform both the kinematic model and the dynamic model into a nonlinear model of a backstepping method, wherein the nonlinear model includes a first-order subsystem and a second-order subsystem, and the nonlinear model includes unknown interference and control input;

[0038] A subsystem module, used to construct a virtual control law of a first-order subsystem and a tracking error derivative of a second-order subsystem, wherein the tracking error is provided with an unknown disturbance;

[0039] A disturbance fitting module is used to fit the unknown disturbance by executing a neural network to obtain an adaptive controller of reinforcement learning, and to evaluate the result of executing the neural network by an evaluation neural network, wherein the evaluation neural network is evaluated by penalty function approximation; and the adaptive controller is proved to be stable by Lyapunov.

[0040] The execution neural network and the evaluation neural network both obtain update rates by using a gradient descent method, the update target of the execution neural network is to minimize the benefit error, and the update target of the evaluation neural network is to minimize the mean square error of the residual;

[0041] The control module is used to output a control signal to the rocket through the adaptive controller, and the control signal is used to control the yaw angle and the pitch angle to achieve rocket attitude adaptation.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention relates to a rocket attitude adaptive control algorithm based on reinforcement learning, which belongs to the research field of rocket attitude control. The Actor-Critic (AC) structure based on reinforcement learning is composed of an action network and an evaluation network. The evaluation network outputs an evaluation value of the state of the rocket body according to the state of the rocket, and the action network generates a corresponding elevator deflection angle according to the evaluation value output by the evaluation network, thereby realizing stable control of the rocket attitude without relying on the internal model of the rocket. This method can realize the fitting estimation of time-varying interference based on reinforcement learning, and control the system state, and the system state can quickly converge to the expected state. The present invention realizes tracking control of the rocket attitude by strengthening the learning of the Actor-Critic structure and observing unknown nonlinearity without relying on the internal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the present invention;

[0045] Figure 2 is a change curve of the state x of the present invention;

[0046] Figure 3 is the change curve of the control input u;

[0047] Figure 4 This is a comparison chart of the reinforcement learning estimated value d1 and the actual value;

[0048] Figure 5 This is a comparison chart of the reinforcement learning estimated value d2 and the actual value. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0050] In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention; the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; in addition, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0051] The first aspect of the present invention discloses a rocket attitude adaptive control method based on reinforcement learning, the method comprising the following steps:

[0052] Step 1: Establish and determine the rocket attitude dynamics model

[0053] The motion equations of the rolling rocket include: the center of mass dynamics and kinematics equations, the dynamics equations and kinematics equations of the rocket body rotating around the center of mass, etc. The attitude motion changes of the rolling rocket of the present invention make the following assumptions:

[0054] 1. During the movement of the rocket, it is assumed that the earth is flat and stationary.

[0055] 2. The rocket is regarded as a rigid body and the deformation of the projectile is not considered.

[0056] Select the state variable x2 = (ω y4 ω z4 ),ω y4 ,ω z4 is the angular velocity of the rocket around the projectile in the projectile coordinate system; control input u=(δ y δ z ) is the control torque, δ y δ z is the rudder angle; the system output is y=x1=(ψ θ), where ψ and θ represent the yaw angle and pitch angle of the rocket respectively. The rocket attitude motion and dynamics equations are established as:

[0057]

[0058] Among them, J x4 ,J y4 ,J z4 is the moment of inertia of the projectile axis; My4 ,M z4 are the sum of the static moment and damping moment of the projectile itself on the y-axis and z-axis respectively; ΔM y ,ΔM z are the external disturbance torques such as gusts on the y-axis and z-axis respectively; γ, ω x4 are the rotation angle and angular velocity of the projectile's longitudinal axis respectively; is the control coefficient. Simplify the above model and select the state variable x2=(ω y4 ω z4 ), and abstract the following nonlinear models, corresponding to the first-order subsystem and the second-order subsystem respectively:

[0059]

[0060] in,

[0061]

[0062] Among them: state x1 = (ψ θ) represents the yaw angle and pitch angle of the rocket, x2 = (ω y4 ω z4 ) is the angular velocity of the rocket around the projectile in the projectile coordinate system, f1(x1)=(0 0) T , f2(x1,x2), g1(x1), g2(x1,x2) are the abbreviations of the nonlinear matrix in formula (4); d is the comprehensive unknown interference such as aerodynamic parameter error, model interference and external interference.

[0063] The nonlinear model (3) is a rocket roll attitude anti-interference system, and the control signal u controls the output state x through state feedback. At the same time, d is considered as an external unknown interference in the present invention. For external slow-changing interference, the unknown interference d can be overcome by adaptively adjusting the input control signal u; however, for complex high-frequency interference d, a reinforcement learning adaptive controller needs to be designed. Therefore, a neural network model (execution network) is designed to fit the unknown interference d to compensate for the external interference.

[0064] The task of the control system of the present invention is to realize the target angle instruction y of the guidance system through adaptive adjustment of the input control signal u without exceeding the control amount of the control system, that is, the maximum torque provided by the rudder deflection angle. d Fast and stable tracking.

[0065] Step 2, define the rocket pitch angle and yaw angle tracking errors, and define the performance indicators that characterize the tracking errors.

[0066] According to the rocket roll attitude control system model (3), the system output is y = x1 = (ψ θ), where ψ and θ represent the yaw angle and pitch angle of the rocket respectively. The tracking error of the yaw and pitch angle of the rocket can be defined as:

[0067] e1=x1-x 1d (5)

[0068] Among them, x 1d represents the expected tracking value of the rocket's yaw and pitch angles, and x1 represents the yaw and pitch angles output by the system.

[0069] Since the purpose of this study is to track and control the posture, the performance indicators of reinforcement learning can be defined as follows:

[0070]

[0071] in, Q is a positive definite matrix, which represents the accumulation of control system errors.

[0072] Step 3: Backstepping controller design

[0073] Consider the uncertain posture motion model formed by equation (3) and make the following assumptions.

[0074] Assumption 1, tracking instruction y d and its second-order derivative exist and are bounded.

[0075] Assumption 2, uncertainty Bounded, that is is a positive constant with unknown upper bound.

[0076] The basic idea of ​​backstepping design is to decompose a complex nonlinear system into subsystems that do not exceed the system order, and then design Lyapunov functions and intermediate virtual control quantities for each subsystem, and "backward" to the entire system until the design of the entire control law is completed. Comprehensively consider the control law and the adaptive law to make the entire closed-loop system meet the expected dynamic and static performance indicators.

[0077] Define e1 = x1 - x 1d ,e2=x2-x 2c ,y d =x d =(ψθ) T It is the angle command expected by the system.

[0078] Taking the derivative of the error e1, we get:

[0079]

[0080] Design virtual control volume:

[0081] x 2c =g1 -1 (y d -f1-k1e1) (8)

[0082] when To avoid the singularity of g1, take θ=arcsin2(q1q2+q0q3).

[0083] Substituting (8) into (7) yields:

[0084]

[0085] Taking the derivative of e2, we get:

[0086]

[0087] Output from the reinforcement learning execution network The estimated value of the unknown interference d can be obtained Design the controller to control the input u, then:

[0088]

[0089] The designed control law is:

[0090]

[0091] where g2 -1 The term (g1e1) compensates for the angular tracking error e1.

[0092] Step 4: Based on the backstepping method, design the evaluation network and action network structures.

[0093] For the first-order system Design the virtual control law as follows:

[0094] x 2c =g1(x1) -1 (-k1e1-f1(x1)+x 1d ),k1>0 (13)

[0095] Enter virtual control into x 2c Substituting into the first-order subsystem we get:

[0096]

[0097] Defining the Second-Order Subsystem The tracking error is e2 = x2-x 2c , then the derivative of the tracking error of the second-order system can be obtained

[0098] The control input Substituting into the second-order subsystem, the derivative of the tracking error is:

[0099]

[0100] Due to the tracking error of the second-order subsystem Then formula (15) can be regarded as

[0101]

[0102] in, It represents the fitting error of the unknown disturbance d. Since d is an unknown function in the control dynamic process, the unknown d is fitted by designing a neural network model (execution network) to design the optimal control of the execution network. and the neural network structure that actually interferes with d.

[0103] According to the neural network fitting theory, we know that there is a parameter W a Make

[0104]

[0105] in, The basis function is bounded, that is, it satisfies ||Φ a ||≤Φ aM .definition W a The weighted expected error is Definition H a is the fitting error of the execution network neural network, so:

[0106]

[0107] To achieve Design the following reinforcement learning adaptive controller:

[0108]

[0109] Among them, k2>0 is the control gain.

[0110] Taking the derivative of e2 and substituting it into formula (10) we get:

[0111]

[0112] The integral penalty function of the control system is defined as:

[0113]

[0114] in, Q is a positive definite matrix, Q∈{Q|Q∈R,Q>0}.

[0115] Design and evaluate the neural network structure to approximate the penalty function:

[0116]

[0117] Among them, the basis function is bounded, that is, it satisfies ||Φ c ||≤Φ cM .

[0118] In the above process, the evaluation network is used to evaluate the output of the execution network. The execution network should achieve the maximum evaluation in continuous attempts, so as to continuously interact and find the optimal one.

[0119] Step 5, design and evaluate the neural network update rate and the network update rate of the neural network execution.

[0120] According to the influence of the control effect, the effect error objective function of the execution network module is constructed:

[0121]

[0122] Where Ω is the gain factor. E a The update target of executing the neural network weight is to make E a Minimum, so the update rate of the execution network weight (gradient descent, derivation):

[0123]

[0124] Among them, Γ is the learning rate of the neural network.

[0125] Construct the residual mean square error function for evaluating the neural network:

[0126]

[0127] Among them, E c To evaluate the network residual function, the update target of evaluating the neural network weights is to make E c Minimum, so according to the gradient descent method, the update rate of the evaluation network weight is obtained:

[0128]

[0129] in,

[0130] Approximation error of AC network for reinforcement learning and Defined as Then from (24) and (26) we can get:

[0131]

[0132] The approximation error ε c satisfy ε cM is a certain normal number.

[0133] Step 6, proof of control system stability based on Lyapunov.

[0134] Construct a Lyapunov function:

[0135]

[0136] In the formula, Approximation error for reinforcement learning AC neural network.

[0137] Taking the derivative of V and substituting it into equations (14) and (20) yields:

[0138]

[0139] For the fifth term of formula (29), use 2A T B≤||A|| 2 +||B|| 2 ,A∈R n ,B∈R n And the ||AB||≤||A||·||B|| inequality is processed as follows:

[0140]

[0141] Substituting formula (27) into the sixth and seventh terms in formula (29) yields:

[0142]

[0143]

[0144] Substituting formulas (32), (31), and (30) into formula (29), we obtain:

[0145]

[0146] When k1>0, k2>1, When, meet According to Lyapunov's stability theorem, the target controlled object x converges to 0 asymptotically and the weight errors of the execution network and the evaluation network Uniformly eventually bounded, where:

[0147]

[0148] The following is further verified by combining simulation examples:

[0149] In the rocket simulation, the simulation parameters are set as: main inertia J y4 =J z4 =4.59kg, the sum of the static moment and damping moment of the projectile itself M y4 =M z4=4.59kg.

[0150]

[0151] Among them, J x4 ,J y4 ,J z4 is the moment of inertia of the projectile axis; M y4 ,M z4 are the sum of the static moment and damping moment of the projectile itself on the y-axis and z-axis respectively; ΔM y ,ΔM z are the external disturbance torques such as gusts on the y-axis and z-axis respectively; γ, ω x4 are the rotation angle and angular velocity of the projectile's longitudinal axis respectively; is the control coefficient. After simplifying the above model, the following nonlinear system can be obtained:

[0152] System matrix: f1 = (0 0) T ,

[0153]

[0154] Unknown error d = 0.1sin(x2+5)

[0155] Controller gains: k1 = diag(2,2), k2 = diag(2.5,2.5)

[0156] System initial value: (x1,x2,ξ1,ξ2)=(8,10,5,9)

[0157] The simulation results are as follows Figure 2-Figure 5 As shown:

[0158] From the simulation results:

[0159] Depend on Figure 2 The simulation results show that by adjusting the parameters reasonably, the system state stabilizes at the expected value of [2, 2] within 3 seconds, which proves the stability of the system.

[0160] Depend on Figure 4-Figure 5 From the simulation results, it can be seen that the reinforcement learning fitter converges to the actual value in about 4 seconds, and its fitting converges to 0 within 4 seconds, which proves the effectiveness of the reinforcement learning neural network design.

[0161] This method takes into account the situation where the system has time-varying interference and provides a control method for system state feedback including reinforcement learning neural network design. This method can quickly eliminate time-varying interference and control the system state, so that the system state can quickly converge to the desired state.

[0162] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A rocket attitude adaptive control method based on reinforcement learning, characterized in that: The following steps are involved: S1, establishing a kinematic model and a dynamic model of the rocket attitude, and transforming both the kinematic model and the dynamic model into a nonlinear model of the backstepping method, wherein the nonlinear model includes a first-order subsystem and a second-order subsystem, and the nonlinear model includes unknown interference and control input; S2, constructing a virtual control law of the first-order subsystem and a tracking error derivative of the second-order subsystem, wherein the tracking error is provided with an unknown disturbance; S3, fitting the unknown disturbance by executing the neural network to obtain a reinforcement learning adaptive controller, evaluating the result of executing the neural network by an evaluation neural network, wherein the evaluation neural network is evaluated by penalty function approximation; and the adaptive controller is proved to be stable by Lyapunov; The execution neural network and the evaluation neural network both obtain update rates by using a gradient descent method, the update target of the execution neural network is to minimize the benefit error, and the update target of the evaluation neural network is to minimize the mean square error of the residual; S4, the adaptive controller rocket outputs a control signal, and the control signal is used to control the yaw angle and the pitch angle to achieve rocket attitude adaptation.

2. The rocket attitude adaptive control method based on reinforcement learning according to claim 1 is characterized in that: In S1, the first-order subsystem and the second-order subsystem are respectively: Where, f1(x1)=(0 0) T , f2(x1,x2), g1(x1), g2(x1,x2) are nonlinear matrices, u is the control signal, d is the unknown interference, x1=(ψ θ), ψ, θ represent the yaw angle and pitch angle of the rocket, x2=(ω y4 ω z4 ),ω y4 ,ω z4 is the angular velocity of the rocket around the projectile in the projectile coordinate system.

3. The rocket attitude adaptive control method based on reinforcement learning according to claim 1 is characterized in that: In S3, the virtual control of the first subsystem is: x 2c =g1(x1) -1 (-k1e1-f1(x1)+x 1d ),k1>0 (13) Among them, x 2c is the expected value, g1(x1) is the abbreviation of the nonlinear matrix, e1 is the tracking error of the rocket yaw and pitch angle, x 1d Represents the tracking expected value of the rocket's yaw and pitch angles.

4. The rocket attitude adaptive control method based on reinforcement learning according to claim 1 is characterized in that: In S2, the tracking error derivative of the second-order subsystem is: in, is the derivative of the state variable, is the derivative of the virtual control quantity, d is the unknown disturbance, is an estimate of the unknown interference.

5. The rocket attitude adaptive control method based on reinforcement learning according to claim 1 is characterized in that: In S3, the execution neural network fits the unknown disturbance as follows: in, 6. The rocket attitude adaptive control method based on reinforcement learning according to claim 3 is characterized in that: In S3, the update rate of the executed neural network is: Among them, Γ is the learning rate of the neural network, and Ω is the gain coefficient.

7. The rocket attitude adaptive control method based on reinforcement learning according to claim 1 is characterized in that: In S3, the penalty function is approximated as: Among them, the basis function is bounded and satisfies ||Φ c |≤Φ cM .

8. The rocket attitude adaptive control method based on reinforcement learning according to claim 7 is characterized in that: The update rate of the evaluation neural network is: in, 9. The rocket attitude adaptive control method based on reinforcement learning according to claim 1 is characterized in that: In S2, the control law of the rocket control is: Among them, g2 -1 The term (g1e1) compensates for the angular tracking error e1.

10. A rocket attitude adaptive control system based on reinforcement learning, characterized in that: include: A nonlinear module is used to establish a kinematic model and a dynamic model of a rocket attitude, and to transform both the kinematic model and the dynamic model into a nonlinear model of a backstepping method, wherein the nonlinear model includes a first-order subsystem and a second-order subsystem, and the nonlinear model includes unknown interference and control input; A subsystem module, used to construct a virtual control law of a first-order subsystem and a tracking error derivative of a second-order subsystem, wherein the tracking error is provided with an unknown disturbance; A disturbance fitting module is used to fit the unknown disturbance by executing a neural network to obtain an adaptive controller of reinforcement learning, and to evaluate the result of executing the neural network by an evaluation neural network, wherein the evaluation neural network is evaluated by penalty function approximation; and the adaptive controller is proved to be stable by Lyapunov. The execution neural network and the evaluation neural network both obtain update rates by using a gradient descent method, the update target of the execution neural network is to minimize the benefit error, and the update target of the evaluation neural network is to minimize the mean square error of the residual; The control module is used to output a control signal to the rocket through the adaptive controller, and the control signal is used to control the yaw angle and the pitch angle to achieve rocket attitude adaptation.

Images