Finite time sliding mode attitude control method for quad-rotor unmanned aerial vehicle based on reinforcement learning

Abstract

The invention discloses a quadrotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning, and the method comprises the steps: building a quadrotor unmanned aerial vehicle attitude dynamic model considering external disturbance and model uncertainty, and rewriting the quadrotor unmanned aerial vehicle attitude dynamic model into a standard state equation form; a controller nominal item based on a nonsingular integral terminal sliding mode method is designed, an arrival control law item containing a time-varying matrix is designed, and the sum of the arrival control law item and the nominal item is an attitude controller; based on the reinforcement learning principle, an extended Markov decision process is constructed, a corresponding state set, an action set, action implementation mapping, a reward function and the like are designed, and a time-varying matrix is obtained through reinforcement learning training and is set online. According to the method, the dynamic performance and the steady-state performance of the system are effectively improved, and meanwhile high robustness is achieved under turbulence disturbance.

Description

TECHNICAL FIELD

[0001] The application relates to a finite-time sliding mode attitude control method for a quadrotor unmanned aerial vehicle based on reinforcement learning, and belongs to the technical field of unmanned aerial vehicle attitude control. BACKGROUND

[0002] In recent years, the application range of unmanned aerial vehicles is very wide, and the unmanned aerial vehicles are related to surveying and mapping, inspection, logistics, emergency and the like, and the attitude performance directly affects the task effect. The unmanned aerial vehicle attitude control is affected by external factors such as wind field turbulence, at the same time, the system model is affected by parameter uncertainty and unmodeled dynamics, so that the controller design has great challenges.

[0003] In the prior art, the PID control structure is simple, but the precision is low. The model predictive control can process constraints, but the calculation amount is large. The sliding mode control has strong anti-disturbance, and the non-singular terminal sliding mode control can converge in a finite time, but the adaptability to strong disturbance and model uncertainty is weak. In the prior art, an adaptive method, a neural network or an extended observer is introduced to improve the control performance, but there are problems such as complex calculation and slow convergence speed. Therefore, the attitude control method capable of converging in a finite time and having adaptive adjustment ability in the face of complex external disturbance and model uncertainty is urgently needed to be researched. SUMMARY

[0004] The technical problem to be solved by the application is to provide a finite-time sliding mode attitude control method for a quadrotor unmanned aerial vehicle based on reinforcement learning, so as to improve the attitude control performance under the influence of external factors such as turbulence and under the system model uncertainty.

[0005] The application adopts the following technical solutions to solve the above technical problem: The finite-time sliding mode attitude control method for a quadrotor unmanned aerial vehicle based on reinforcement learning comprises the following steps: Step 1, considering external disturbance and model uncertainty, establishing an attitude dynamics model of the quadrotor unmanned aerial vehicle, and transforming the model into a standard state equation form; Step 2, constructing a non-singular integral terminal sliding mode function, and designing a nominal item of the attitude controller; at the same time, designing a reaching control law item containing a time-varying matrix, and the sum of the reaching control law item and the nominal item is the attitude controller; Step 3, constructing an extended Markov decision process, depicting the optimization setting of the time-varying matrix based on reinforcement learning under attitude tracking, defining the three-axis attitude angle error and its change rate of the quadrotor unmanned aerial vehicle as states, the output of the policy network in reinforcement learning as actions, taking the three-axis attitude angle error and its change rate of the quadrotor unmanned aerial vehicle as inputs of the policy network, designing an action implementation mapping, mapping the output of the policy network to the control parameter action, so as to obtain the time-varying matrix; The three-axis attitude angle error and its change rate of the quadrotor unmanned aerial vehicle and the output of the strategy network are taken as the input of the value network in reinforcement learning, a reward function for control performance is designed, the output of the strategy network is evaluated and fed back to the strategy network, and then the next action is generated, so that the attitude control of the quadrotor unmanned aerial vehicle is realized. Step 4, in the training process of the strategy network and the value network, the deep deterministic gradient algorithm is used for parameter updating.

[0006] Compared with the prior art, the above technical scheme has the following technical effects: 1. The quadrotor unmanned aerial vehicle attitude dynamics model is constructed according to model uncertainty and external disturbances such as turbulence, and compared with the traditional modeling, the model is more accurate.

[0007] 2. The controller structure is optimized, and a corresponding extended Markov decision process is designed to optimize the control effect. Simulation results show that the method has better rapidity and robustness, and has strong adaptability under turbulence disturbance. DETAILED DESCRIPTION

[0008] Figure 1 is a flowchart of the quadrotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning of the application; Figure 2 is a reinforcement learning strategy network and action mapping structure; Figure 3 is a reinforcement learning value network structure; Figure 4 is a reinforcement learning framework diagram; Figure 5 is a comparison of attitude angle response control effects under turbulence disturbance; Figure 6 is a comparison of attitude angle error control effects under turbulence disturbance. DETAILED DESCRIPTION

[0009] The embodiments of the application will be described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the drawings are exemplary and are only used to explain the application, and cannot be explained as a limitation of the application.

[0010] As Figure 1 shown, the application proposes a quadrotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning, and the specific steps are as follows: Step 1, considering external disturbance and model uncertainty, the quadrotor unmanned aerial vehicle attitude dynamics model is established as follows: , In the formula, , , are the roll angle, pitch angle, yaw angle of the quadrotor respectively, , , are the roll angular velocity, pitch angular velocity, yaw angular velocity of the quadrotor respectively, , , are the roll angular acceleration, pitch angular acceleration, yaw angular acceleration of the quadrotor respectively, , , are the moments of inertia of the quadrotor about the axes respectively, is the moment of inertia of the rotor, is the yaw axis counter-torque sum, , , are the roll control moment, pitch control moment, yaw control moment respectively, are the aerodynamic damping coefficients of the roll channel, pitch channel, yaw channel respectively; , , are the external disturbances of the roll channel, pitch channel, yaw channel respectively. The total disturbance contains external disturbances and model uncertainties, which is .

[0011] For the convenience of designing the control algorithm, the above attitude dynamics model is rewritten as the standard state equation form: , where, ; are the angular velocity state variables of the roll angle, pitch angle, yaw angle respectively; are the angular acceleration of the roll angle, pitch angle, yaw angle respectively.

[0012] Step 2, construct a non-singular integral terminal sliding mode function, and design the nominal term of the attitude controller. In order to improve the dynamic performance and disturbance rejection ability of the attitude system, the reaching control law term containing time-varying matrix based on reinforcement learning technology is designed. The total control amount of the controller is the sum of the nominal term and the control law term, which cooperates to improve the rapidity and robustness. The specific steps are as follows: Define the attitude angle tracking error as: , where, is the attitude angle state vector; is the expected attitude angle vector.

[0013] Define the vector as: is the attitude angular velocity state vector; is the desired attitude angular velocity vector, is a diagonal matrix satisfying

[0014] Construct a nonsingular integral terminal sliding mode function to ensure converges to zero in finite time is the sliding mode function vector, denotes the current time, denotes any time between 0 and is a diagonal matrix satisfying is the power index vector, is an all-one vector with the same dimension as To avoid singularity, satisfy is an odd number, and

[0015] The derivative of is: is the total disturbance, including external disturbances and model uncertainties.

[0016] Let , ignoring the total disturbance term, the nominal controller is: To improve the rapidity and robustness of the control system, a time-varying reaching control law based on reinforcement learning is designed. The time-varying reaching control law based on reinforcement learning is expressed as: is the moment of inertia matrix of the quadrotor unmanned aerial vehicle body, and​​​​​​​​​​​​​​​​​​​​​​​​​​ The matrix is ​​time-varying and will be adjusted using reinforcement learning techniques. Let be the sliding mode function vector. As parameters, satisfying ; Its dynamic satisfaction .in, and Element-wise computation. The following time-varying matrix based on reinforcement learning is defined. and : , , Among them, control parameters , , , , , .

[0017] In summary, the attitude controller is designed as follows: , Right now .

[0018] Step 3: Construct an extended Markov decision process for online tuning of the time-varying matrix in Step 2. Define a set of states and actions, using the attitude angle errors of each axis and their rates of change as states. Design actions to achieve mapping, using the mapped control parameter actions as components of the time-varying matrix. Design a reward function for control performance. Design a policy network and a value network and determine their inputs and outputs. During the training phase, an experience replay mechanism is used to enhance exploratory power by superimposing behavioral noise, continuously optimizing the time-varying matrix. Details are as follows: To characterize the time-varying parameter update mechanism based on policy optimization under attitude tracking, the following extended Markov decision process is constructed: , In the formula, For a set of states, For a set of actions, As a state transition kernel, For instant rewards, As a discount factor, Mapping is implemented for actions. This is a behavioral noise process.

[0019] Specifically, the state set is defined as a 6-dimensional vector. Based on the UAV's attitude angle tracking of the target, the three-axis attitude angle error and its rate of change are taken as the state: , In the formula, , , This represents the attitude angle tracking error across the three axes. , , This represents the rate of change of attitude angle tracking error across the three axes.

[0020] Define the action set as a 6-dimensional vector, which will be the output of the policy network: , In the formula, This is the vector output by the policy network.

[0021] Control parameter action From action Mapping is achieved through actions. The process of adding behavioral noise to the action output of the policy network during the training phase is obtained. This enhances the exploratory nature of the process. The mapping yields... The process is represented as: , In the formula, The mapped control parameter action vector, To prevent the control parameter from approaching 0, a lower limit of the constant is set. For the action scaling matrix, specifically: , , For the first The scaling factor of each control parameter corresponds to, in turn to, the scaling factor of each control parameter. ; To control the lower bound vector of the action parameter, .

[0022] Therefore, action mapping right The Jacobian matrix can be expressed as: , Mapped control parameter action vector Represented as: , Based on this, a time-varying matrix of the following form is formed. and : .

[0023] The state transition in the decision-making process is represented as follows: given the current state and control parameters, the action... The system transforms into the next state through the control law and external disturbances. : , Design a reward function based on the control performance objective and considering the control output performance. as follows: , , , , In the formula, , , For the weight parameters, satisfying , , , This is the attitude angle error vector. This is the vector of attitude angle error rate of change. This allows the error to converge while adjusting the control energy across different error ranges, resulting in a more stable system performance when the error is small. Specifically, For a piecewise function, it is represented as: , In the above formula, the constant , ; Let L be the 2-norm of the attitude angle error vector at the end of the current decision. To control the measurement of energy, , Specifically defined as: , In the formula, This is the attitude angle error vector at the end of the current decision. This is the three-axis control torque vector generated by the control law at the time of the current decision.

[0024] To adjust the time-varying matrix online and Use reinforcement learning strategies applicable to the action space to optimize and The update mechanism. Design strategy networks such as... Figure 2 As shown, its input is the attitude angle tracking error vector. and attitude angle tracking error change rate The output of the policy network is a vector. , Obtained through action mapping The network is designed with two hidden layers, each containing 256 neurons, activated by ReLU, and the output layer is activated by tanh; the value network is designed as follows: Figure 3 As shown, its input is a state vector. and the vector output by the policy network The output is The network is designed with two hidden layers, each containing 256 neurons, activated by ReLU. The policy network and value network work together to quantitatively evaluate and optimize the updates of control parameters.

[0025] Step 4: During training, the policy network and value network are updated using a deep deterministic gradient algorithm. The specific training process is as follows: The policy network is updated using the policy gradient method: , In the formula, This represents the batch sample size. For value function, Output actions for the policy network. The objective function for accumulating expected rewards throughout the training process is expressed as follows: , In the formula, For the first Instant rewards for each time step As a discount factor, For a finite time step, In policy networks The mathematical expectation operator for the state-action trajectory distribution.

[0026] Minimize target loss This updates the value network. , In the formula, This is the current state. For instant rewards, As a termination marker, For the next state, The target policy network corresponding to the policy network. This is the target value function corresponding to the value network.

[0027] The parameters of the target policy network and the target value network are updated using a soft update method: , In the formula, The parameters of the current value network, , These are the parameters for the target policy network and the target value network, respectively. .

[0028] Training employs an experience replay mechanism, storing the state, action, reward, and next state of each interaction in an experience pool. Network updates are performed through mini-batch random sampling, improving training stability. The reinforcement learning framework is as follows: Figure 4 As shown. Through the above process, the time-varying matrix is ​​made... and We continuously adjust and optimize the drone attitude controller online.

[0029] The effectiveness of the control method of the present invention is verified through simulation experiments. Specifically, the relevant parameters of the quadcopter UAV are shown in Table 1.

[0030] Table 1. Relevant parameters of UAVs

[0031] Based on the attitude dynamics modeling of the quadcopter UAV, the controller is designed according to step 2. The relevant parameters of the controller are shown in Table 2.

[0032] Table 2 Controller Parameters

[0033] Step 3 involves reinforcement learning training, with the sampling step size of the reinforcement learning agent set to 0.1 s. The parameter selection for reinforcement learning is shown in Table 3.

[0034] Table 3 Reinforcement Learning Related Parameters

[0035] The desired attitude angles for the roll, pitch, and yaw axes are set as follows, representing a multi-stage step response: , , , The numerical simulation step size is set to 0.004s, and the simulation time is 120s.

[0036] The simulation process uses the standard Dryden turbulence model to simulate the effects of stochastic wind fields in real flight environments. For each round of the reinforcement learning training process, the stochastic standard Dryden turbulence is independent to ensure regularity and the universality of the trained policy.

[0037] This invention provides a finite-time sliding mode attitude control method for quadrotor UAVs based on reinforcement learning. The method was experimentally compared with a non-singular terminal sliding mode control method and a nominal sliding mode control method without online tuning, yielding the following attitude angle response: Figure 5 As shown, the attitude angle error is as follows: Figure 6As shown in Table 4, the integral of squared error (ISE) was calculated to quantitatively analyze the control effect. Figure 5 , Figure 6 The quantitative analysis results show that the method of the present invention has better speed and robustness, and is particularly adaptable to turbulent disturbances.

[0038] Table 4 Comparison of Integral Squared Error (ISE) (degrees / deg)

[0039] Based on the same inventive concept, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned reinforcement learning-based finite-time sliding mode attitude control method for quadrotor UAVs.

[0040] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based finite-time sliding mode attitude control method for quadrotor unmanned aerial vehicles.

[0041] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0042] This invention is described with reference to flowchart illustrations of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each step in the flowchart, and combinations of steps in the flowchart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the steps in the flowchart. Figure 1 A device for a function specified in one or more processes.

[0043] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0044] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0045] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A finite-time sliding mode attitude control method for quadrotor UAVs based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Considering external disturbances and model uncertainties, establish the attitude dynamics model of the quadcopter UAV and transform it into the standard state equation form; Step 2: Construct a non-singular integral terminal sliding mode function and design the nominal term of the attitude controller; at the same time, design the arrival control law term containing a time-varying matrix. The sum of the arrival control law term and the nominal term is the attitude controller. Step 3: Construct an extended Markov decision process to characterize the time-varying matrix optimization tuning based on reinforcement learning under attitude tracking. Define the three-axis attitude angle error of the quadcopter UAV and its rate of change as states, and the output of the policy network in reinforcement learning as actions. Use the three-axis attitude angle error of the quadcopter UAV and its rate of change as the input of the policy network, design actions to achieve mapping, and map the output of the policy network to control parameter actions, thereby obtaining the time-varying matrix. Using the three-axis attitude angle error and its rate of change of a quadrotor UAV and the output of the policy network as inputs to the value network in reinforcement learning, a reward function for control performance is designed to evaluate the output of the policy network and feed it back to the policy network, thereby generating the next action and realizing attitude control of the quadrotor UAV. Step 4: During the training process, the policy network and value network use a deep deterministic gradient algorithm to update their parameters.

2. The finite-time sliding mode attitude control method for a quadrotor UAV based on reinforcement learning according to claim 1, characterized in that, The specific process of step 1 is as follows: The attitude dynamics model of the quadrotor UAV considering external disturbances and model uncertainties is established as follows: ， In the formula, and These are the roll angle, pitch angle, and yaw angle of a quadcopter drone. and These are the roll rate, pitch rate, and yaw rate of the quadcopter drone. and These are the roll acceleration, pitch acceleration, and yaw acceleration of the quadcopter UAV, respectively. and The body surrounds and Moment of inertia of the shaft, Let be the moment of inertia of the rotor. This is the resultant of the yaw axis counter-torque. and These are the roll control torque, pitch control torque, and yaw control torque, respectively. and These are the aerodynamic damping coefficients for the roll, pitch, and yaw channels, respectively. and These are external disturbances in the roll channel, pitch channel, and yaw channel, respectively; total disturbance Including external disturbances and model uncertainties, for superscript Indicates transpose; The above attitude dynamics model is transformed into the following standard state equation form: ， In the formula, ; and The angular velocity state variables are roll angle, pitch angle, and yaw angle, respectively. and These are the angular accelerations for roll, pitch, and yaw, respectively.

3. The finite-time sliding mode attitude control method for a quadrotor UAV based on reinforcement learning according to claim 2, characterized in that, The specific process of step 2 is as follows: Define attitude angle tracking error vector for: ， In the formula, , and These are the tracking errors for roll angle, pitch angle, and yaw angle, respectively. The attitude angle state vector, ; Let be the desired attitude angle vector. , and These are the desired roll angle, pitch angle, and yaw angle, respectively. Define vector for: ， In the formula, , and Both represent intermediate variables. The attitude angular velocity state vector, ; Let be the desired attitude angular velocity vector. , and These are the desired roll rate, pitch rate, and yaw rate, respectively. It is a diagonal matrix. ,parameter and satisfy , , ; Constructing a non-singular integral terminal sliding mode function ,make sure In a limited time Converging to zero: ， In the formula, , and These are sliding mode functions for roll angle, pitch angle, and yaw angle, respectively. Indicates the current moment. Represents 0 to At any time between, It is a diagonal matrix. ,parameter and satisfy , , ; Let be a vector of exponents. , To and A vector of all 1s with the same dimension. and All are odd numbers, and , , ; right Differentiate and let Ignoring the total disturbance, the nominal term of the attitude controller is obtained. for: ， In the formula, The moment of inertia matrix of the quadcopter drone body. , , Let be the vector of attitude angle error change rate. , and These are the rates of change of tracking errors for roll angle, pitch angle, and yaw angle, respectively. Design a time-varying arrival control law based on reinforcement learning. for: ， In the formula, As parameters, satisfying ; For symbolic functions, , and Both represent intermediate variables, satisfying ; and All are time-varying matrices: ， ， In the formula, and All of these are control parameters. , , , , , ; Then the attitude controller for: ， In the formula, , and These are the control moments for the roll, pitch, and yaw channels, respectively. 。 4. The finite-time sliding mode attitude control method for a quadrotor UAV based on reinforcement learning according to claim 3, characterized in that, The specific process of step 3 is as follows: The following extended Markov decision process is constructed to characterize the time-varying matrix update mechanism based on reinforcement learning under pose tracking. : ， In the formula, For a set of states, For a set of actions, As a state transition kernel, For instant rewards, As a discount factor, Mapping is implemented for actions. This is a behavioral noise process; Define the state set as a 6-dimensional vector. Based on the UAV's attitude angle tracking of the target, take the three-axis attitude angle error and its rate of change as the state: ， In the formula, It is a real vector space; Define the action set as a 6-dimensional vector: ， In the formula, and All are parameters; Control parameter action Depend on Mapping is achieved through actions. get: ， In the formula, The mapped control parameter action vector, To prevent the control parameter from approaching the lower limit of a constant of 0, For the action scaling matrix, , , For the first The scaling factor for each control parameter, with the first to sixth control parameters being respectively: ; To control the lower bound vector of the action parameter, , These are the lower bounds of the action for each control parameter; Action mapping pairs The Jacobian matrix is ​​expressed as: ， In the formula, Represents the hyperbolic secant function; Mapped control parameter action vector Represented as: ， Thus, the time-varying matrix is ​​obtained. and ; The state transition in the decision-making process is represented as follows: given the current state and control parameter actions After being controlled by the control law and subjected to external disturbances, it transforms into the next state. ; Design a reward function based on the control performance objective and considering the control output performance. as follows: ， ， ， ， In the formula, and All are weighted parameters, satisfying , , ; For a piecewise function, it is represented as: ， In the formula, All are constants. , ; The attitude angle error vector at the end of the current decision. The 2-norm, To control the measurement of energy, , This is the three-axis control torque vector generated by the control law at the time of the current decision. , and These are the control torques generated by the control law during the current decision-making process for the roll, pitch, and yaw channels, respectively.

5. The finite-time sliding mode attitude control method for a quadrotor UAV based on reinforcement learning according to claim 1, characterized in that, The specific process of step 4 is as follows: The policy network is updated using the policy gradient method: ， In the formula, To adjust the policy network parameters gradient operator, This represents the batch sample size. To gradient operator, For value function, These are the parameters of the current policy network. Output actions for the policy network. This is the current state. The first sample obtained from the experience replay pool One action sample, To accumulate the expected reward objective function throughout the training process, , In policy networks The expected value operator for the state-action trajectory distribution is given below. For the first Instant rewards for each time step , As a discount factor, For a finite time step; Value networks minimize target loss Update: ， In the formula, For instant rewards, As a termination marker, For the next state, The target policy network corresponding to the policy network. The target value function corresponding to the value network; The parameters of the target policy network and the target value network are updated using a soft update method: ， In the formula, The parameters of the current value network, , These are the parameters for the target policy network and the target value network, respectively. .

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the finite-time sliding mode attitude control method for a quadrotor unmanned aerial vehicle based on reinforcement learning as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the finite-time sliding mode attitude control method for a quadrotor unmanned aerial vehicle based on reinforcement learning as described in any one of claims 1 to 5.

Images