Strong adaptive control law design method for launch vehicle nozzle thrust degradation condition

Abstract

The application discloses a kind of strong adaptation control law design methods under the working condition of launch vehicle nozzle thrust drop, the present control algorithm is overcome by the application, the training of control law is carried out under different thrust drop working condition by using reinforcement learning technology, so that the controller can adapt to the working condition of launch vehicle nozzle thrust drop, so as to realize the strong robust stability control of launch vehicle attitude. Meanwhile, the training environment established by the algorithm used by the application can add various other disturbances, and the strong adaptation ability of the controller to the added disturbance is realized through training, and the controller capacity can be continuously improved. The application solves the state estimation problem when the control quantity correlation coefficient changes, which is difficult to solve by ESO and other methods, and can more effectively design the compensation to compensate for the nozzle failure reasoning, with higher control accuracy and better effect. And it has the advantages of not needing to decouple the attitude, and realizing the automatic design of launch vehicle control law, reducing the workload of human design.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent fault-tolerant control of aircraft, in particular to a strong adaptive control law design method for a launch vehicle nozzle thrust under a degraded condition. BACKGROUND

[0002] Attitude control is an important guarantee for the safe and stable flight of a carrier vehicle and the successful completion of a mission, and therefore, designing an advanced carrier vehicle attitude control system with good robustness and strong adaptive ability has become a research hotspot in the field of aerospace in recent years. Due to the high flight Mach number during the reentry process of a carrier vehicle, the flight characteristics of a large envelope and a large airspace, the strong nonlinearity of the controlled model itself, large aerodynamic uncertainties, external disturbances, and severe attitude channel and attitude-orbit coupling, the attitude control of a carrier vehicle is extremely challenging. After a long distance flight through the active stage, the attitude control nozzle system as an actuator is prone to degradation failure during the attitude adjustment stage due to the influence of high dynamic pressure and thermal pressure, resulting in a deviation of the actual generated attitude control moment from the expected control command, thereby affecting the control accuracy and system reliability. Therefore, a carrier vehicle attitude control system must have high control accuracy and strong robustness, and in addition to achieving accurate tracking of the attitude, it must also ensure the ability to cope with partial failure of the actuator.

[0003] Therefore, the controller design process should fully consider the influence of the characteristics of the severe aerodynamic changes, strong disturbances and uncertainties, and actuator failures, so that the system can more accurately and quickly track the guidance command, while having strong robustness to external disturbances, model uncertainties, and actuator failures, and achieving safe and stable flight. SUMMARY

[0004] Therefore, the present application provides a strong adaptive control law design method for a launch vehicle nozzle thrust under a degraded condition, which can overcome the shortcomings of existing control algorithms, and provides a kind of intelligent strong adaptive control algorithm for a carrier vehicle, which can realize strong robust and stable control of the attitude of a launch vehicle under the condition of a degraded launch vehicle nozzle thrust, overcome the problem of insufficient attitude control ability caused by nozzle thrust failure, and realize strong adaptive attitude control of the launch vehicle through intelligent algorithm.

[0005] To achieve the above purpose, the technical scheme of the present application includes the following steps:

[0006] Step 1: Establish a three-channel attitude loop mathematical model of a launch vehicle under the condition of random failure of a nozzle thrust, establish a training environment and a Markov process under Python language according to the mathematical model, define the current attitude error and angular velocity of the launch vehicle as the state vector s t , and output the control amount of the launch vehicle generated by the intelligent agent as the action vector a tThe attitude error and angular velocity of the launch vehicle at the next moment are generated based on the launch vehicle's differential equations and used as the state vector s. t+1 The angle error and the L2 distance of the angular velocity after the control variable are applied are used as the reward r from the environmental feedback. t . s t With a t As input to the training environment, r t With s t+1 As the output of the training environment, the Markov process for the attitude control of the launch vehicle is defined as s t and a t Generate r t and s t+1 The process.

[0007] Step 2: Establish the TD3 reinforcement learning agent. The TD3 reinforcement learning agent includes 2 Actor networks and 4 Critic networks, and the state vector s t The action vector a is used as the input and output of the Actor execution network. t The reward r obtained by applying the Actor network to the environment t and the state vector s at the next time step t+1 With s t and a t The data samples are used to train the Critic execution network to predict the cumulative reward under the interaction between the agent and the environment, and the cumulative reward Q is predicted by the Critic network. t (s t a t The Actor network is guided to obtain the maximum cumulative reward value, i.e., the minimum cumulative control error. The Actor network contains one execution network for training and one target network for generating the expected value. The Critic network contains two execution networks for training and two target networks for generating the expected value.

[0008] Step 3: After the TD3 reinforcement learning algorithm agent training converges, the Actor network is used as the controller to realize the attitude control of the launch vehicle under the condition of nozzle thrust failure.

[0009] Preferably, step two specifically includes the following steps:

[0010] Step 201: Initialize the Actor with random weights and execute the network parameters ω eval Actor target network parameters ω tar The parameters θ of Critic execution network 1 and execution network 2 eval1 θ eval2 Critic target network 1 and target network 2 parameters θ tar1 θtar2 .

[0011] Step 202: The actor execution network interacts with the training environment, and the obtained Markov process vector [s t , a t , r t , s t+1 ] is used to fill the experience replay cache pool until a batch of data is filled, and then step 203 is entered to start training.

[0012] Step 203: Randomly initialize the initial state s0 in the training environment.

[0013] Step 204: The actor execution network generates an action a t at state s t t = μ t + N exp ; a t is the action vector at time t; μ t is the output of the actor execution network at time t; N exp is the Gaussian distributed random noise added in the training process, with a mean of 0 and a recommended standard deviation of 0.2 and decay following time, with the decay amplitude determined according to the need for exploration of the agent.

[0014] Step 205: Apply a t to the training environment to obtain the reward r t and the next state vector s t+1 in the environment.

[0015] Step 206: Pack and store the information [s t , a t , r t , s t+1 ] in the experience replay cache pool.

[0016] Step 207: Apply s t+1 to the actor target network to generate the next time action a t+1 , and apply s t+1 and a t+1 to the two Critic target networks to generate the next time cumulative return i is the number of the two Critic target networks; N reg is the Gaussian distributed regularization noise added, which is used to prevent overfitting of the cumulative return, with a mean of 0 and a standard deviation adjusted according to training requirements.

[0017] Step 208: Sample a batch of data from the experience replay cache pool, and according to the Bellman equation Q is estimated by generating a cumulative reward function using a Critic target network. t_d (s t a t ), and calculate the current Q value through the Critic network; where Q t_d Let be the value function of the expected cumulative return obtained at time t, and γ be the reward discount factor, representing the confidence of subsequent predicted rewards, with a value of 0 < γ < 1.

[0018] Step 209: Calculate the loss function for each Critic execution network. critic (t)=0.5(Q t_d (s t a t )-Q t (s t a t )) 2 Q t (s t a t ) represents the output of the corresponding Critic execution network;

[0019] Step 2010: Minimize the loss using the Adam optimizer critic The method (t) is implemented using gradient descent. Update the parameters of the Critic execution network; α is the learning rate of the Critic execution network.

[0020] Step 2011: If the Actor execution network update cycle is reached, then minimize the loss. actor The method (t) is implemented using gradient descent. Update the parameters of the Actor execution network; β is the learning rate of the Actor execution network.

[0021] Step 2012: According to the formula Update the target network parameters; whereby and These are the parameters for each execution network and the target network, respectively. n is the number of the execution network, representing Actor, Critic1, or Critic2, and η is the parameter update rate of the target network.

[0022] Step 2013: Repeat steps 204 to 2013 until one cycle of rocket flight mission is completed.

[0023] Step 2014: If the reward r t If the curve converges and reaches the expected value, then the training ends; otherwise, return to step 203.

[0024] Beneficial effects:

[0025] (1) The present application adopts a TD3 reinforcement learning algorithm to design an attitude control law of a launch vehicle offline, which can adapt to a large range of thrust failure conditions of the nozzle and achieve strong adaptive control of the launch vehicle.

[0026] (2) The training environment established by the algorithm used in the present application can add various other disturbances, and through training, the controller can achieve strong adaptability to the added disturbances, and the controller capability can be continuously improved.

[0027] (3) Compared with the ESO state observer method, the present application solves the state estimation problem when the control quantity correlation coefficient changes, which cannot be solved by the ESO method, and can more effectively design the compensation quantity to compensate for the nozzle failure reasoning, with higher control precision and better effect.

[0028] (4) The present application can achieve end-to-end control of the three-dimensional attitude of the launch vehicle without the need for attitude decoupling.

[0029] (5) The present application can achieve automatic design of the launch vehicle control law, reducing the workload of human design. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a principle block diagram of the TD3 algorithm;

[0031] Figure 2 is a principle block diagram of the neural network controller after training deployed on the control task;

[0032] Figure 3 is a reward change trend graph during the training process;

[0033] Figure 4 is an attitude angle and angular velocity control response curve graph in the fault-free state; Figure 4 (a) of is the attitude angle error, Figure 4 (b) of is the angular velocity

[0034] Figure 5 is a response curve graph in the 10% actuator failure fault state; Figure 5 (a) of is an attitude angle error response curve graph, Figure 5 (b) of is an angular velocity response curve graph;

[0035] Figure 6 is a response curve graph in the 30% actuator failure fault state; Figure 6 (a) of is an attitude angle error response curve graph; Figure 6 (b) of is an angular velocity response curve graph;

[0036] Figure 7 is a response curve graph in the 50% actuator failure fault state; Figure 7(a) is a response curve of attitude angle error; Figure 7 (b) is a response curve of angular velocity;

[0037] Figure 8 (a) is a response curve of attitude angle error; Figure 8 (b) is a response curve of angular velocity. Figure 8 DETAILED DESCRIPTION

[0038] The application will be described in detail below with reference to the accompanying drawings and examples.

[0039] The method for designing a strong adaptive control law under the working condition of thrust reduction of a launch vehicle nozzle comprises the following steps:

[0040] Step one: establish a three-channel attitude loop mathematical model of a launch vehicle under the working condition of random failure of nozzle thrust, establish a training environment and Markov process under Python language according to the mathematical model, define the current attitude error and angular velocity of the launch vehicle as state vectors s t , the control output of the launch vehicle generated by the intelligent agent as action vectors a t , the attitude error and angular velocity of the launch vehicle at the next time according to the differential equation of the launch vehicle as state vectors s t+1 , and the L2 distance of the angle error and angular velocity after the action of the control as the reward r t of the environment feedback. s t and a t are taken as the input of the training environment, and r t and s t+1 are taken as the output of the training environment, and the Markov process of the launch vehicle attitude control is defined as the process of generating r t and s t from s t and a t+1 .

[0041] First, the pitch angle θ, the yaw angle ψ and the roll angle φ are selected to establish the attitude kinematics model of the launch vehicle, which is expressed as follows:

[0042]

[0043] In the formula, [p, q, r] respectively represent the roll angular velocity, the yaw angular velocity and the pitch angular velocity, and the dynamic characteristics are as follows:

[0044]

[0045] In the formula, J xx , J yy , J zz ​J represents the moment of inertia of the vehicle about each axis in the body frame zx M represents the product of inertia between the z and x axes in the body frame x M y M z J represents the three-axis control torque. The actual torque acting on the control system can be represented as:

[0046]

[0047] where G(t) = diag{g1(t), g2(t), g3(t)}, 0 < g j (t)≤1, (j = 1, 2, 3) represents the driving efficiency on the three axes.

[0048] Considering the attitude control system of the vehicle under the condition of degenerative failure, the attitude model of the vehicle in the attitude adjustment stage is established as follows:

[0049]

[0050] where Ξ = [θ, ψ, φ] T represents the attitude angle vector, where θ ∈ (0°, 90°) since the research stage is the attitude adjustment stage in the atmosphere; ω = [p, q, r] T represents the angular velocity vector; u = F represents the control force of the three-axis equivalent RCS output; the rotation matrix S, the moment of inertia matrix J, and the cross product matrix ω × are represented as follows, respectively:

[0051]

[0052]

[0053]

[0054] A Markov process is established for the attitude control problem of the launch vehicle. In this problem, the control error of each attitude channel of the rocket is set as the state quantity s t of the environment, i.e.:

[0055] s t = e t = X d (t) - X(t)

[0056] where e t is the state error vector, including the angle control error and the current angular velocity of each channel (the units are set to ° or ° / s during training and testing). The output of the controller is taken as the action vector a t of the Markov process, which is defined as:

[0057] at = A = [δ θ , δ ψ , δ φ ] T

[0058] where δ θ , δ ψ , δ φ are the outputs of the pitch, yaw, roll attitude controllers respectively. To adapt the algorithm to the degraded condition of the launch vehicle actuators, the actuator capability is randomly degraded by 0%-80% during the training process.

[0059] The form of the reward function is set as:

[0060]

[0061] where P is a positive definite matrix that determines the weight of each reward dimension.

[0062] Step two: Establish a TD3 reinforcement learning agent, which includes 2 Actor networks (including 1 execution network for training and 1 target network for generating expected value) and 4 Critic networks (including 2 execution networks for training and 2 target networks for generating expected value). The state vector s t is input into the Actor execution network and the action vector a t is output. The reward r t obtained by the Actor network acting on the environment and the state vector s t+1 at the next time are used to form a data sample with s t and a t to train the Critic execution network to predict the cumulative return of the agent interacting with the environment, and the cumulative return Q t (s t , a t ) predicted by the Critic network guides the Actor network to obtain the maximum cumulative return value, i.e., the minimum cumulative control error.

[0063] In step two, the agent training method is as follows:

[0064] Step 201: Initialize the parameters ω eval of the Actor execution network, the parameters ω tar of the Actor target network, the parameters θ eval1 and θ eval2 of the Critic execution network 1 and execution network 2, and the parameters θ tar1 and θ tar2 of the Critic target network 1 and target network 2 with random weights.

[0065] Step 202: The actor execution network interacts with the training environment, and the obtained Markov process vector [s t , a t , r t , s t+1 ] is used to fill the experience replay cache pool until a batch of data is filled, and step 203 is entered to start training.

[0066] Step 203: Randomly initialize the initial state s0 in the training environment.

[0067] Step 204: The actor execution network generates an action a t = μ t + N t at state s exp at time t; a t is the action vector at time t; μ t is the output of the actor execution network at time t; N exp is the Gaussian distributed random noise added in the training process, with a mean of 0 and a recommended standard deviation of 0.2 and decay following time, and the decay amplitude is determined according to the need for exploration of the agent.

[0068] Step 205: Apply a t to the training environment to obtain the reward r t and the next state vector s t+1 in the environment.

[0069] Step 206: Pack and store the information [s t , a t , r t , s t+1 ] in the experience replay cache pool.

[0070] Step 207: Apply s t+1 to the actor target network to generate the next time action a t+1 , and apply s t+1 and a t+1 to the two Critic target networks to generate the next time cumulative return i and j are the numbers of the two Critic target networks; N reg is the Gaussian distributed regularization noise added, which is used to prevent overfitting of the cumulative return, with a mean of 0 and a standard deviation adjusted according to training requirements.

[0071] Step 208: Sample a batch of data from the experience replay cache pool, and according to the Bellman equation The cumulative reward function Q is estimated by generating the Critic target network (the Critic network has two target networks). t_d (s t a t ), and calculate the current Q value through the Critic network; where Q t_d Let be the value function of the expected cumulative return obtained at time t, and γ be the reward discount factor, representing the confidence of subsequent predicted rewards, with a value of 0 < γ < 1.

[0072] Step 209: Calculate the loss function for each Critic execution network. critic (t)=0.5(Q t_d (s t a t )-Q t (s t a t )) 2 Q t (s t a t ) represents the output of the corresponding Critic execution network.

[0073] Step 2010: Minimize the loss using the Adam optimizer critic The method (t) is implemented using gradient descent. Update the parameters of the Critic execution network; α is the learning rate of the Critic execution network.

[0074] Step 2011: If the Actor execution network update cycle is reached, then minimize the loss. actor The method (t) is implemented using gradient descent. Update the parameters of the Actor execution network; β is the learning rate of the Actor execution network.

[0075] Step 2012: According to the formula Update the target network parameters; whereby and These are the parameters for each execution network and the target network, respectively. n is the number of the execution network, representing Actor, Critic1, or Critic2, and η is the parameter update rate of the target network.

[0076] Step 2013: Repeat steps 204 to 2013 until one cycle of rocket flight mission is completed.

[0077] Step 2014: If the reward r t If the curve converges and reaches the expected value, then the training ends; otherwise, return to step 203.

[0078] After the above training steps are completed, it is considered that the controller converges, the Actor network is used as the controller, and the neural network control in the nozzle thrust decline condition can be realized. The controller deployment mode is as shown in Figure 2 .

[0079] Step three: after the TD3 reinforcement learning algorithm agent training converges, the Actor network (execution network or target network) is used as the controller to realize the launch vehicle attitude control in the nozzle thrust failure condition.

[0080] The TD3 reinforcement learning training framework as shown in Figure 1 is established, wherein the Actor network is used to generate the attitude control amount of the launch vehicle, and the Critic network is used to estimate the cumulative return of the launch vehicle in the working environment; the execution network of the Actor network and the Critic network is used to train by using the data in the environment, and the target network of the Actor network and the Critic network is used to estimate the subsequent cumulative return according to the state estimation of the next time state fed back by the environment, so as to realize the update of the evaluation network through the Bellman equation.

[0081] The following examples are for reference: take the initial attitude angle error Ξ(0)=[5°, 1.5°, -3.3°] T ; the expected attitude angle change over time is set to Ξ d =[0°, 0°, 0°] T ; the initial angular velocity ω(0)=[0, 0, 0] T ; the structure matrix B=diag{4, 10, 10}; the moment of inertia J is set as:

[0082]

[0083] The hyperparameters of the algorithm are taken as follows during the training process:

[0084]

[0085]

[0086] In order to realize the adaptation of the algorithm to the decline condition of the launch vehicle execution mechanism capacity, the execution mechanism capacity is randomly reduced by 0%-80% during the training process.

[0087] The agent reaches the convergence state after 200 episodes, and the total reward change curve in the training process is as shown in Figure 3 .

[0088] Take the actor network under the 200th episode as a strong adaptive neural network controller, respectively in the fault-free state, the 10% actuator failure fault state, the 30% actuator failure fault state, the 50% actuator failure fault state, and the 70% actuator failure fault state, and take the full thrust control instruction as ±500N, Figures 4-8 The attitude control curve is for thrust reduction. As can be seen from the figure, the controller trained by the algorithm can quickly converge to the expected value under different thrust reduction conditions, achieving robustness to the thrust reduction conditions of the launch vehicle.

[0089] To sum up, the above is only a preferred embodiment of the present application, not for limiting the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for designing a strong adaptive control law for a launch vehicle nozzle thrust degradation condition, characterized in that, Comprising the following steps: Step one: Establish the three-channel attitude loop mathematical model of launch vehicle under the reasoning random failure condition of nozzle, according to the mathematical model to establish the training environment and Markov process under Python language, define the current attitude error and angular velocity of launch vehicle as state vector , the control output of launch vehicle generated by intelligent agent as action vector , the next time attitude error and angular velocity of launch vehicle generated by the differential equation of launch vehicle as state vector , the L2 distance of angle error and angular velocity after the action of control as the reward of environment feedback ; the and as the input of training environment, the and as the output of training environment, then the Markov process of launch vehicle attitude control is defined as the process of generating and from and ; First, the pitch angle , the yaw angle and the roll angle are selected to establish the attitude kinematics model of the carrier, which is expressed as follows: wherein respectively denote roll, yaw and pitch angular velocities, whose dynamic characteristics are as follows: wherein , , denotes the moment of inertia of the carrier about each axis in the body system, denotes the product of inertia of the body system axis and axis; , , denotes the three-axis control torque; The torque actually acting on the control system is expressed as: In the formula, , indicates the driving efficiency on the three axes; Considering the attitude control system of the spacecraft under the condition of degenerative failure, the attitude model of the spacecraft in the attitude adjustment segment is established as follows: wherein, represents the attitude angle vector, wherein, since the research phase is the attitude adjustment segment outside the atmosphere, there are ; represents the angular velocity vector; represents the control force of the triaxial equivalent RCS output; the rotation matrix , the moment of inertia matrix , the cross product matrix respectively represent as follows: A Markov process is established for the problem of attitude control of a launch vehicle. In this problem, the control errors of each attitude channel of the vehicle are set as the state variables of the environment That is: where is the state error vector, including the angle control error of each channel and the current angular velocity, the angle control error is set to ° during training and testing, and the unit of angle is ° / s; the output of the controller is taken as the action vector of the Markov process , which is defined as: wherein are the outputs of the pitch, yaw, and roll attitude controllers, respectively; to adapt the algorithm to the degraded condition of the launch vehicle actuators, the actuator capability is randomly degraded from 0% to 80% during the training process; The form of the reward function is set as: wherein is a positive definite matrix, determining the weight of each reward dimension; Step two: establish a TD3 reinforcement learning algorithm agent, the TD3 reinforcement learning agent includes two actor networks and four critic networks, input the state vector as the actor execution network and output the action vector , the reward obtained by the actor network acting on the environment and the state vector of the next moment , and composing a data sample to train the critic execution network to realize the prediction of the cumulative return under the interaction of the agent and the environment, and the cumulative return predicted by the critic network guides the actor network to obtain the maximum cumulative return value, that is, the minimum cumulative control error; the actor network includes one execution network for training and one target network for generating expected value, and the critic network includes two execution networks for training and two target networks for generating expected value; Step three: after the TD3 reinforcement learning algorithm intelligent agent training converges, the actor network is used as a controller to realize the attitude control of the launch vehicle under the condition of failed nozzle thrust.

2. The method of designing a robust adaptive control law for a launch vehicle nozzle in a thrust down condition according to claim 1, wherein, The step two specifically includes the following steps: Step 201: Initialize the Actor execution network parameters with random weights , Actor target network parameters , Critic execution network 1, execution network 2 parameters , Critic target network 1, target network 2 parameters ; Step 202: the network interacts with the training environment by using the Actor to fill the experience replay buffer pool with the obtained Markov process vector until a batch of data is filled, and then step 203 is entered to start training; Step 203: Randomly initialize the initial state in the training environment ; Step 204: the network is executed by the actor to generate an action in the state ; is an action vector at time t; is an output of the actor execution network at time t; is a Gaussian distribution random noise added in the training process, the mean of the noise is 0, and the recommended standard deviation is 0.2 and decays with time, and the decay amplitude is determined according to the exploration needs of the agent; Step 205: obtaining acts on the training environment, in which a reward is obtained and the next time state vector ; Step 206: Pack and store the information into the experience replay buffer pool. information into the experience replay buffer pool. Step 207: the Actor target network is acted on to generate the next moment action and the Critic target network is acted on to generate the next moment cumulative reward and the two Critic target networks are acted on to generate the next moment cumulative reward , i is the number of Critic target networks; is the added Gaussian distribution regularization noise, which is used to prevent the cumulative reward from overfitting, and the mean of the noise is 0 and the standard deviation is adjusted according to the training requirements;​​ Step 208: Sample a batch of data from the experience replay buffer, and generate cumulative return function estimates according to Bellman equation by Critic target network , and compute current Q value by Critic execution network; wherein is the value function of the expected cumulative return obtained at the moment is a reward discount factor, representing the credibility of the subsequent predicted reward, and takes a value ; Step 209: calculating the loss function of each Critic execution network wherein is the output of the corresponding Critic execution network; Step 2010: updating the Critic execution network parameters by the Adam optimizer with the method of minimizing by the gradient descent method ; learning rate for the Critic execution network; Step 2011: if the Actor execution network update period is reached, update the parameters of the Actor execution network using the method of minimizing by gradient descent with a learning rate of for the Actor execution network; Step 2012: performing the update of the target network parameters according to the formula wherein and are the parameters of the respective execution network and target network, is the number of the execution network, representing Actor, Critic1 or Critic2, is the parameter update rate of the target network; Step 2013: repeat steps 204~step 2013 until the rocket flight task of a cycle is completed. Step 2014: If reward If the curve converges and reaches the expected value, end the training; otherwise, return to step 203.

Images