Four-rotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning

Abstract

The application discloses a quad-rotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning, and comprises the following steps: establishing a quad-rotor unmanned aerial vehicle attitude dynamics model considering external disturbance and model uncertainty, and rewriting the model into a standard state equation form; designing a controller nominal term based on a non-singular integral terminal sliding mode method, designing a reaching control law term containing a time-varying matrix, and the sum of the reaching control law term and the nominal term being an attitude controller; based on the principle of reinforcement learning, constructing an extended Markov decision process, designing corresponding state sets, action sets, action implementation mapping and reward functions, and the time-varying matrix being obtained through reinforcement learning training and being adjusted online. The method effectively improves the dynamic performance and the steady-state performance of the system, and has strong robustness under turbulent disturbance.

Description

TECHNICAL FIELD

[0001] The application relates to a finite-time sliding mode attitude control method for a quadrotor unmanned aerial vehicle based on reinforcement learning and belongs to the technical field of unmanned aerial vehicle attitude control. BACKGROUND

[0002] In recent years, the application range of unmanned aerial vehicles is very wide, involves surveying and mapping, inspection, logistics, emergency and the like, and the attitude performance directly affects the task effect. The unmanned aerial vehicle attitude control is affected by external factors such as wind field turbulence, at the same time, the system model is affected by parameter uncertainty and unmodeled dynamics, so that the controller design has great challenges.

[0003] In the prior art, the PID control structure is simple, but the precision is low. The model predictive control can process constraints, but the calculation amount is large. The sliding mode control has strong anti-disturbance, and the non-singular terminal sliding mode control can converge in a finite time, but the adaptability to strong disturbance and model uncertainty is weak. In the prior art, an adaptive method, a neural network or an extended observer is introduced to improve the control performance, but there are problems such as complex calculation and slow convergence speed. Therefore, the attitude control method capable of converging in a finite time and having adaptive adjustment ability in the face of complex external disturbance and model uncertainty is urgently needed to be researched. SUMMARY

[0004] The technical problem to be solved by the application is to provide a finite-time sliding mode attitude control method for a quadrotor unmanned aerial vehicle based on reinforcement learning, so as to improve the attitude control performance under the influence of external factors such as turbulence and under the system model uncertainty.

[0005] The application adopts the following technical solutions to solve the above technical problem:

[0006] The finite-time sliding mode attitude control method for a quadrotor unmanned aerial vehicle based on reinforcement learning comprises the following steps:

[0007] Step 1, considering external disturbance and model uncertainty, an attitude dynamics model of the quadrotor unmanned aerial vehicle is established, and is transformed into a standard state equation form;

[0008] Step 2, a non-singular integral terminal sliding mode function is constructed, and a nominal term of the attitude controller is designed; meanwhile, a reaching control law term containing a time-varying matrix is designed, and the sum of the reaching control law term and the nominal term is the attitude controller;

[0009] Step 3, an extended Markov decision process is constructed, the optimization setting of the time-varying matrix based on reinforcement learning under attitude tracking is described, the three-axis attitude angle error and its change rate of the quadrotor unmanned aerial vehicle are defined as states, the output of the policy network in reinforcement learning is defined as an action, the three-axis attitude angle error and its change rate of the quadrotor unmanned aerial vehicle are taken as inputs of the policy network, an action is designed to realize mapping, the output of the policy network is mapped to the control parameter action, so that the time-varying matrix is obtained.

[0010] The three-axis attitude angle error and its rate of change of the quadrotor unmanned aerial vehicle and the output of the strategy network are taken as the input of the value network in reinforcement learning, a reward function for control performance is designed, the output of the strategy network is evaluated and fed back to the strategy network, and then the next action is generated, so as to realize the attitude control of the quadrotor unmanned aerial vehicle.

[0011] In the training process of the strategy network and the value network, the deep deterministic gradient algorithm is used for parameter updating.

[0012] Compared with the prior art, the above technical scheme has the following technical effects:

[0013] 1. The quadrotor unmanned aerial vehicle attitude dynamics model is constructed according to the model uncertainty and external disturbances such as turbulence, and is more accurate than the traditional modeling.

[0014] 2. The controller structure is optimized, and a corresponding extended Markov decision process is designed to optimize the control effect. Simulation results show that the method has better rapidity and robustness, and has strong adaptability under turbulence disturbance. DETAILED DESCRIPTION

[0015] Figure 1 is a flow chart of the quadrotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning of the present application;

[0016] Figure 2 is a reinforcement learning strategy network and action mapping structure;

[0017] Figure 3 is a reinforcement learning value network structure;

[0018] Figure 4 is a reinforcement learning framework diagram;

[0019] Figure 5 is a comparison of attitude angle response control effects under turbulence disturbance;

[0020] Figure 6 is a comparison of attitude angle error control effects under turbulence disturbance. DETAILED DESCRIPTION

[0021] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be explained as a limitation of the present application.

[0022] As shown in Figure 1 The present application proposes a quadrotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning, and the specific steps are as follows:

[0023] Step 1, considering external disturbance and model uncertainty, the attitude dynamics model of quadrotor UAV is established as follows:

[0024] ,

[0025] In the formula, 、 、 Roll angle, pitch angle, yaw angle of quadrotor UAV, 、 、 Roll angular velocity, pitch angular velocity, yaw angular velocity of quadrotor UAV, 、 、 Roll angular acceleration, pitch angular acceleration, yaw angular acceleration of quadrotor UAV, 、 、 The moment of inertia of the body around the axis, The moment of inertia of the rotor, The yaw axis counter torque, 、 、 Roll control moment, pitch control moment, yaw control moment, Roll channel, pitch channel, yaw channel aerodynamic damping coefficient; 、 、 Roll channel external disturbance, pitch channel external disturbance, yaw channel external disturbance. The total disturbance contains external disturbance and model uncertainty, which is .

[0026] In order to design the control algorithm, the above attitude dynamics model is rewritten as the standard state equation form:

[0027] ,

[0028] In the formula, ; Roll angle, pitch angle, yaw angle angular velocity state variable; Roll angle, pitch angle, yaw angle angular acceleration.

[0029] Step 2, construct non singular integral terminal sliding mode function, design the nominal term of attitude controller. In order to improve the dynamic performance and anti disturbance ability of attitude system, the reaching control law term with time varying matrix based on reinforcement learning technology is designed. The total control amount of the controller is the sum of the nominal term and the control law term, which cooperates to realize the improvement of rapidity and robustness. The specific is as follows:

[0030] Define the attitude angle tracking error as

[0031] ,

[0032] where is the attitude angle state vector; is the desired attitude angle vector.

[0033] Define the vector as

[0034] ,

[0035] where is the attitude angular rate state vector; is the desired attitude angular rate vector, is a diagonal matrix satisfying , , .

[0036] Construct the nonsingular integral terminal sliding mode function to ensure converges to zero in finite time :

[0037] ,

[0038] where is the sliding mode function vector, denotes the current time, denotes any time between 0 and , is a diagonal matrix satisfying , , ; is the power index vector, is an all-ones vector of the same dimension as . To avoid singularity, satisfy , , , , , is an odd number, and , , .

[0039] Differentiate to get

[0040] ,

[0041] where , is the total disturbance, including the external disturbance and model uncertainty.

[0042] Let , ignoring the total disturbance term, the nominal term of the controller can be obtained is:

[0043] ,

[0044] To improve the rapidity and robustness of the control system, a time-varying reaching control law based on reinforcement learning is designed. The time-varying reaching control law based on reinforcement learning is expressed as:

[0045] ,

[0046] In the formula, is the moment of inertia matrix of the quadrotor unmanned aerial vehicle body, and is a time-varying matrix, which will be adjusted by reinforcement learning technology. is the sliding mode function vector, is a parameter, satisfying ; , whose dynamics satisfy . Among them, and are calculated element by element. Define the following time-varying matrix based on reinforcement learning and :

[0047] ,

[0048] ,

[0049] Among them, the control parameter , , , , , .

[0050] Summarizing the above, the attitude controller is designed as follows:

[0051] ,

[0052] That is

[0053] .

[0054] Step 3: Construct an extended Markov decision process for online tuning of the time-varying matrix in Step 2. Define a set of states and actions, using the attitude angle errors of each axis and their rates of change as states. Design actions to achieve mapping, using the mapped control parameter actions as components of the time-varying matrix. Design a reward function for control performance. Design a policy network and a value network and determine their inputs and outputs. During the training phase, an experience replay mechanism is used to enhance exploratory power by superimposing behavioral noise, continuously optimizing the time-varying matrix. Details are as follows:

[0055] To characterize the time-varying parameter update mechanism based on policy optimization under attitude tracking, the following extended Markov decision process is constructed:

[0056] ,

[0057] In the formula, For a set of states, For a set of actions, As a state transition kernel, For instant rewards, As a discount factor, Mapping is implemented for actions. This is a behavioral noise process.

[0058] Specifically, the state set is defined as a 6-dimensional vector. Based on the UAV's attitude angle tracking of the target, the three-axis attitude angle error and its rate of change are taken as the state:

[0059] ,

[0060] In the formula, , , This represents the attitude angle tracking error across the three axes. , , This represents the rate of change of attitude angle tracking error across the three axes.

[0061] Define the action set as a 6-dimensional vector, which will be the output of the policy network:

[0062] ,

[0063] In the formula, This is the vector output by the policy network.

[0064] Control parameter action From action Mapping is achieved through actions. The process of adding behavioral noise to the action output of the policy network during the training phase is obtained. This enhances the exploratory nature of the process. The mapping yields... The process is represented as:

[0065] ,

[0066] where, is the mapped control parameter action vector, is a constant lower bound to prevent control parameters from tending to 0; is the action scaling matrix, specifically , , is the scaling factor of the th control parameter, corresponding to in turn; is the control parameter action lower bound vector, .

[0067] Thus, the action mapping is the Jacobian matrix of , which can be expressed as:

[0068] ,

[0069] The mapped control parameter action vector is expressed as:

[0070] ,

[0071] Accordingly, the time-varying matrix and are formed as follows:

[0072] .

[0073] The state transition in the decision-making process is expressed as: given the current state and control parameter action , the system is transformed into the next state by the control law and external disturbance:

[0074] ,

[0075] According to the control performance target, while considering the control output performance, the reward function is designed as follows:

[0076] ,

[0077] ,

[0078] ,

[0079] ,

[0080] where, , , is a weight parameter satisfying , , , is a vector of attitude angle error, is a vector of attitude angle error rate. The control energy is adjusted in different error intervals, and the system performs more smoothly when the error is small. Wherein, is a piecewise function, expressed as:

[0081] ,

[0082] In the above formula, the constant , ; is the 2-norm of the attitude angle error vector at the end of the current decision, is a measure of control energy, , is specifically defined as:

[0083] ,

[0084] In the formula, is the 2-norm of the attitude angle error vector at the end of the current decision, is the three-axis control torque vector generated by the control law at the current decision.

[0085] In order to online tune the time-varying matrix and , the reinforcement learning strategy suitable for action space is used to optimize the update mechanism of and . The policy network is designed as shown in Figure 2 , the input is the attitude angle tracking error vector and the attitude angle tracking error rate , the output of the policy network is the vector , After action mapping, we get , the network is designed with 2 layers of hidden layers, each containing 256 neurons, activated by ReLU, and the output layer is activated by tanh; the value network is designed as shown in Figure 3 , the input is the state vector and the vector output by the policy network , the output is , the network is designed with 2 layers of hidden layers, each containing 256 neurons, activated by ReLU. The policy network and the value network cooperate to quantitatively evaluate and optimize the control parameter update.

[0086] Step 4, the parameters of the policy network and the value network are updated by using the deep deterministic gradient algorithm during the training process. The specific training process is as follows:

[0087] The policy network is updated by using the policy gradient method:

[0088] ,

[0089] In the formula, is the number of batch samples, is the value function, is the action output by the policy network. is the cumulative reward expected target function during the entire training process, which is expressed as:

[0090] ,

[0091] In the formula, is the immediate reward at the th time step, is the discount factor, is the limited time step, is the mathematical expectation operator of the state-action trajectory distribution under the policy network .

[0092] The target loss is minimized , so as to update the value network:

[0093] ,

[0094] In the formula, is the current state, is the immediate reward, is the termination flag, is the next state, is the target policy network corresponding to the policy network, is the target value function corresponding to the value network.

[0095] The parameters of the target policy network and the target value network are updated by using the soft update method:

[0096] ,

[0097] In the formula, is the parameter of the current value network, , are the parameters of the target policy network and the target value network respectively, .

[0098] Training employs an experience replay mechanism, storing the state, action, reward, and next state of each interaction in an experience pool. Network updates are performed through mini-batch random sampling, improving training stability. The reinforcement learning framework is as follows: Figure 4 As shown. Through the above process, the time-varying matrix is ​​made... and We continuously adjust and optimize the drone attitude controller online.

[0099] The effectiveness of the control method of the present invention is verified through simulation experiments. Specifically, the relevant parameters of the quadcopter UAV are shown in Table 1.

[0100] Table 1. Relevant parameters of UAVs

[0101]

[0102] Based on the attitude dynamics modeling of the quadcopter UAV, the controller is designed according to step 2. The relevant parameters of the controller are shown in Table 2.

[0103] Table 2 Controller Parameters

[0104]

[0105] Step 3 involves reinforcement learning training, with the sampling step size of the reinforcement learning agent set to 0.1 s. The parameter selection for reinforcement learning is shown in Table 3.

[0106] Table 3 Reinforcement Learning Related Parameters

[0107]

[0108] The desired attitude angles for the roll, pitch, and yaw axes are set as follows, representing a multi-stage step response:

[0109] ,

[0110] ,

[0111] ,

[0112] The numerical simulation step size is set to 0.004s, and the simulation time is 120s.

[0113] The simulation process uses the standard Dryden turbulence model to simulate the effects of stochastic wind fields in real flight environments. For each round of the reinforcement learning training process, the stochastic standard Dryden turbulence is independent to ensure regularity and the universality of the trained policy.

[0114] This invention provides a finite-time sliding mode attitude control method for quadrotor UAVs based on reinforcement learning. The method was experimentally compared with a non-singular terminal sliding mode control method and a nominal sliding mode control method without online tuning, yielding the following attitude angle response: Figure 5 As shown, the attitude angle error is as follows: Figure 6 As shown in Table 4, the integral of squared error (ISE) was calculated to quantitatively analyze the control effect. Figure 5 , Figure 6 The quantitative analysis results show that the method of the present invention has better speed and robustness, and is particularly adaptable to turbulent disturbances.

[0115] Table 4 Comparison of Integral Squared Error (ISE) (degrees / deg)

[0116]

[0117] Based on the same inventive concept, this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned reinforcement learning-based finite-time sliding mode attitude control method for quadrotor UAVs.

[0118] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based finite-time sliding mode attitude control method for quadrotor unmanned aerial vehicles.

[0119] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This invention is described with reference to flowchart illustrations of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each step in the flowchart, and combinations of steps in the flowchart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the steps in the flowchart. Figure 1 A device for a function specified in one or more processes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 The function specified in one or more processes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0123] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for finite-time sliding mode attitude control of quadrotor unmanned aerial vehicles based on reinforcement learning, characterized in that, Comprising the following steps: Step 1, considering external disturbance and model uncertainty, a quadrotor unmanned aerial vehicle attitude dynamics model is established, and is transformed into a standard state equation form; Step 2, a non-singular integral terminal sliding mode function is constructed, and a nominal item of the attitude controller is designed; meanwhile, a reaching control law item containing a time-varying matrix is designed, and the sum of the reaching control law item and the nominal item is the attitude controller; the specific process is as follows: Defining a pose angle tracking error vector is: ， wherein, , , are the tracking errors of roll, pitch and yaw angles, respectively, is the attitude angle state vector, , , , are the roll, pitch and yaw angles of the quadrotor UAV, respectively; is the desired attitude angle vector, , , are the desired roll, pitch and yaw angles, respectively. Definition of vector is: ， wherein , , all represent intermediate variables, is the attitude angular velocity state vector, , , are the angular velocity state variables of the roll angle, the pitch angle and the yaw angle, respectively; is the desired attitude angular velocity vector, , , are the desired roll angular velocity, the pitch angular velocity and the yaw angular velocity, respectively, is a diagonal matrix, , the parameters and satisfy , , ; Constructing nonsingular integral terminal sliding mode function , ensuring convergence to zero in finite time ​ ， wherein, , and are the sliding mode functions of roll angle, pitch angle and yaw angle, respectively, denotes the current time, denotes any time between 0 and , is a diagonal matrix, , parameters and satisfy , , ; is a power exponent vector, , is an all-one vector with the same dimension as , and are both odd numbers, and , , ; For derivative, and let , ignoring total disturbance, the nominal term of the attitude controller is: ， wherein, is the moment of inertia matrix of the quadrotor airframe, , and are the moments of inertia of the airframe about and axes, , is the rate of change of the attitude angle error vector, , and are the rate of change of the roll, pitch and yaw angle tracking errors, respectively. Design of time-varying arrival control law based on reinforcement learning For: ， where is a parameter satisfying ; is a sign function, , and are intermediate variables satisfying ; and are time-varying matrices: ， ， wherein and are control parameters, , , , , , ; Then the attitude controller is: ， wherein , and are the control moments for roll, pitch and yaw channels, respectively, and ， wherein and are roll, pitch and yaw angular velocities of the quadcopter, respectively, and are roll, pitch and yaw angular accelerations of the quadcopter, respectively. Step 3, an extended Markov decision process is constructed to depict the time-varying matrix optimization setting based on reinforcement learning under attitude tracking, the three-axis attitude angle error and its rate of change of the quadrotor unmanned aerial vehicle are defined as states, the output of the policy network in reinforcement learning is an action, the three-axis attitude angle error and its rate of change of the quadrotor unmanned aerial vehicle are taken as inputs of the policy network, an action implementation mapping is designed, the output of the policy network is mapped to a control parameter action, so as to obtain the time-varying matrix; The three-axis attitude angle error and its rate of change of the quadrotor unmanned aerial vehicle and the output of the policy network are taken as inputs of the value network in reinforcement learning, a reward function for control performance is designed, the output of the policy network is evaluated and fed back to the policy network, and then the next action is generated, so as to realize the attitude control of the quadrotor unmanned aerial vehicle; The specific process of step 3 is as follows: A Markov decision process is constructed as follows to characterize the time-varying matrix updating mechanism based on reinforcement learning under pose tracking : ， wherein is a set of states, is a set of actions, is a state transition kernel, is an immediate reward, is a discount factor, is an action realization mapping, is a behavior noise process; The state set is defined as a 6-dimensional vector, and the three-axis attitude angle error and its rate of change are taken as states according to the attitude angle tracking target of the unmanned aerial vehicle: ， In the formula, is a real vector space; The action set is defined as a 6-dimensional vector: ， wherein and are parameters; Controlling parameter action By Mapping by action Obtained: ， wherein, is the mapped control parameter action vector, is a constant lower limit to prevent the control parameter from tending to 0, is the action scaling matrix, , , is the scaling coefficient of the first control parameter, the first to the sixth control parameters are ; is the control parameter action lower limit vector, , are the action lower limits of the respective control parameters, respectively. Action mapping pair The Jacobian matrix representation of the action mapping pair is given by: ， wherein denotes the hyperbolic secant function; mapped control parameter action vector is represented as: ， Thus, a time-varying matrix is obtained and ; The state transition in the decision process is represented as: given the current state and control parameter action , the next state is transformed from the control law and external disturbance. According to the control performance target, a reward function is designed while considering the control output performance As follows: ， ， ， ， wherein and are weight parameters satisfying , , ; is a piecewise function represented as: ， wherein are constants, , ; is the 2-norm of the attitude angle error vector at the end of the current decision , is a measure of the control energy, , is the three-axis control torque vector resulting from the control law at the current decision, , and are the control torques in the roll, pitch, and yaw channels, respectively, resulting from the control law at the current decision. In the training process of the policy network and the value network, the deep deterministic gradient algorithm is used for parameter updating. 2.The finite time sliding mode attitude control method for quadrotor unmanned aerial vehicles based on reinforcement learning according to claim 1, wherein, The specific process of step 1 is as follows: The quadrotor unmanned aerial vehicle attitude dynamics model considering external disturbance and model uncertainty is established as follows: ， where is the moment of inertia of the rotor, is the total yaw damping, and are the roll, pitch and yaw control moments, respectively, and are the aerodynamic damping coefficients of the roll, pitch and yaw channels, respectively; and are the external disturbances of the roll, pitch and yaw channels, respectively; the total disturbance includes the external disturbances and model uncertainties, and is the superscript denotes the transpose. The above attitude dynamics model is transformed into the following standard state equation form: ， wherein and are the angular accelerations of roll, pitch and yaw angles, respectively. 3.The finite time sliding mode attitude control method for quadrotor unmanned aerial vehicles based on reinforcement learning according to claim 1, wherein, The specific process of step 4 is as follows: The policy network is updated by a policy gradient method: ， In the formula, To adjust the policy network parameters gradient operator, For the batch sample size, To gradient operator, For value function, These are the parameters of the current policy network. Output actions for the policy network. This is the current state. The first sample obtained from the experience replay pool One action sample, To accumulate the expected reward objective function throughout the training process, , In policy networks The expected value operator for the state-action trajectory distribution is given below. For the first Instant rewards for each time step , As a discount factor, For a finite time step; The value network minimizes the objective loss Update: ， wherein is an immediate reward, is a terminal flag, is a next state, is a target policy network corresponding to the policy network, is a target value function corresponding to the value network; The parameters of the target policy network and the target value network are updated in a soft update manner: ， wherein are parameters of the current value network, , are parameters of the target policy network and the target value network, respectively, .

4. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The processor executes the computer program to realize the steps of the quadrotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning according to any one of claims 1 to 3.

5. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 4. The computer program is executed by the processor to realize the steps of the quadrotor unmanned aerial vehicle finite time sliding mode attitude control method based on reinforcement learning according to any one of claims 1 to 3.

Images