Quad-rotor online reinforcement learning attitude control method with safety guarantee

Abstract

The invention relates to a quadrotor on-line reinforcement learning attitude control method with safety guarantee, which comprises the following steps: S1, acquiring quadrotor attitude control flight data, and respectively acquiring nominal dynamic representation and initial Gaussian process dynamic representation of quadrotor attitude; s2, predicting a long-term task return based on the initial Gaussian process dynamics representation; s3, solving the gradient of the long-term task return to the control strategy parameters, and updating the control strategy by adopting a gradient descent method; s4, solving a model prediction control problem through nominal dynamics, and carrying out the safety filtering of the control strategy; s5, updating the training data set and the corresponding Gaussian process dynamics representation by using a correlation-based data selection criterion; and S6, substituting the updated Gaussian process dynamics representation into the step S2, and continuously repeating the steps S2-S5 in the flight process to realize online reinforcement learning control of the quadrotor inner ring attitude.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle control, and in particular to a quad-rotor online reinforcement learning attitude control method with safety assurance. Background Art

[0002] With the widespread application of quadrotor aircraft in scenarios such as transportation, search and rescue, researchers have proposed many new quadrotor control methods. Precise control of the underlying attitude of quadrotors is the basis for achieving various flight missions. Model-based control methods represented by model predictive control have attracted widespread attention due to their stability, constraint flexibility and predictability. However, due to the influence of wind interference, load changes and aerodynamic effects, quadrotors flying in natural environments have nonlinear dynamics that are difficult to accurately model, and model errors will lead to a decrease in the performance of control methods. With the rapid development of machine learning algorithms, methods such as reinforcement learning and imitation learning have shown good results in quadrotor control. Among them, model-based reinforcement learning methods can simultaneously learn dynamic models and control strategies from interaction data. Compared with model-free reinforcement learning methods, they have higher sampling efficiency and stronger environmental adaptability than model predictive control. However, due to the large model error in the initial learning stage, dangerous control signals may be generated.

[0003] In summary, the existing low-level attitude control methods of quadrotors have defects such as large model error, low data efficiency, lack of environmental adaptability, and lack of safety assurance. Therefore, the present invention proposes a quadrotor online reinforcement learning attitude control method with safety assurance. Summary of the invention

[0004] The purpose of the present invention is to provide a quadrotor online reinforcement learning attitude control method with safety guarantee in view of the defects of the prior art, so as to ensure the real-time performance and safety performance of the controller's online learning.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A quadrotor online reinforcement learning attitude control method with safety assurance, comprising:

[0007] Step S1, collecting quadrotor attitude control flight data, and obtaining the nominal dynamics representation and initial Gaussian process dynamics representation of the quadrotor attitude respectively;

[0008] Step S2, predicting long-term task rewards based on the initial Gaussian process dynamics representation;

[0009] Step S3, solving the gradient of the long-term task reward to the control strategy parameters, and updating the control strategy using the gradient descent method;

[0010] Step S4, performing safety filtering on the control strategy by solving the model predictive control problem through nominal dynamics;

[0011] Step S5, updating a training data set and a corresponding Gaussian process dynamics representation using a correlation-based data selection criterion, wherein the training data set includes attitude control flight data collected in real time during the quadrotor flight process in step S1;

[0012] Step S6, substitute the updated Gaussian process dynamics representation into step S2, and repeat steps S2-S5 continuously during the flight to achieve online reinforcement learning control of the quadrotor inner loop attitude.

[0013] Optionally, collecting the quadrotor attitude control flight data in step S1 includes:

[0014] The control strategy is randomly given, and the reference attitude signal, actual attitude signal and motor PWM signal output by the PID controller are collected during the flight of the quadrotor.

[0015] Optionally, in step S1, the nominal dynamics of the quadrotor attitude is expressed as:

[0016]

[0017] in, is the derivative of the rotation matrix from the machine system to the Earth system, is the rotation matrix from the machine system to the earth system, ω b =[ω x ,ω y ,ω z ] T is the three-axis angular velocity of the body, [·] × Represents the skew-symmetric matrix of the vector, J = [J x ,J y ,J z ] T is the three-axis moment of inertia of the machine body, is the derivative of the body angular velocity, τ T =[τ x ,τ y ,τ z ] T is the propeller's three-axis torque, τ D is the gyro torque, and T is the transpose of the matrix.

[0018] Optionally, in step S1, the initial Gaussian process dynamics of the quadrotor attitude is expressed as:

[0019] x t+1 =f(x t ,a t )

[0020]

[0021] Among them, x t is the current quadrotor attitude, x t+1 is the state of the quadrotor at the next moment, a t is the action output by the control strategy at the current moment, i.e., the motor thrust, (x t ,a t ) is the quadrotor state action pair at the current moment, μ t+1 and∑ t+1 They represent the mean and variance of the state distribution at the next moment, Δx is the change in the quadrotor state, is the mean value of the state change, var f [Δx] is the variance of the state change.

[0022] Optionally, predicting the long-term task reward based on the initial Gaussian process dynamics representation in step S2 includes:

[0023] Given a task objective and a reward function, wherein the task objective is to maximize the cumulative reward, and the reward function is an exponential quadratic function;

[0024] The maximum cumulative reward is solved based on the initial Gaussian process dynamics representation to obtain a long-term task reward representation.

[0025] Optionally, the long-term task reward is expressed as:

[0026]

[0027] Among them, R π (θ) is the long-term task reward, represents the expectation of reward, r is the instantaneous reward at a certain moment, and T is the length of the future time step considered.

[0028] Optionally, the method of updating the control strategy using the gradient descent method in step S3 is:

[0029] θ←θ+lr·(dR π (θ) / dθ)

[0030] Among them, θ is the control strategy parameter, lr is the learning rate, R π (θ) is the long-term task return, dR π (θ) / dθ is the gradient of the long-term task reward with respect to the control policy parameters.

[0031] Optionally, the model predictive control problem in step S4 is:

[0032]

[0033] Among them, u ref (k) is the action output by the reinforcement learning controller at time k, x k+1 is the state of the quadrotor at time k+1, x k is the state of the quadrotor at time k, u k is the action of the quadrotor at time k, ΔT is the discrete time step length for solving the optimization problem, x0 is the initial state of the quadrotor, x init is the current starting state of the quadrotor, x min and x max is the safety boundary of the quadrotor state, u min and u max It is the safety boundary of the quadrotor action.

[0034] Optionally, in step S5, updating the training data set and the corresponding Gaussian process dynamics representation using a correlation-based data selection criterion includes:

[0035] Calculate the distance metric between the collection point data in the training data set, and obtain the minimum value of the distance metric and the corresponding collection point data;

[0036] Calculating the distance metric between the new collection point data and the training data set, and if the distance metric is greater than the minimum distance metric value, replacing the collection point data corresponding to the minimum distance metric value with the new collection point data to complete the updating of the training data set;

[0037] A new Gaussian process dynamics representation is obtained based on the updated training data set, and the update of the Gaussian process dynamics representation is completed.

[0038] The beneficial effects of the present invention are:

[0039] The present invention provides a quad-rotor online reinforcement learning attitude control method with safety guarantee. Aiming at the characteristics of nonlinear uncertainty in the dynamics of a quad-rotor unmanned aerial vehicle, a Gaussian process dynamics model representation method without prior knowledge is designed. Control strategy evaluation and control strategy improvement considering long-term uncertainty are realized based on Gaussian process dynamics. Flight data is used for learning dynamics and control strategy at the same time, which can realize efficient use of data. Aiming at the time complexity problem of Gaussian process model construction, a data priority selection mechanism based on data correlation is proposed, which can ensure the real-time performance of the controller's online learning. Aiming at the potential dangerous control signals that may be caused by model errors in the initial learning stage, a safety filtering layer is designed to ensure the safety of the learning-based controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a schematic diagram of using a quadrotor fixed on a platform to collect flight data according to an embodiment of the present invention;

[0042] Figure 2 This is a basic flow chart of the online reinforcement learning attitude control of a quadrotor according to an embodiment of the present invention;

[0043] Figure 3 This is a flow chart of a quadrotor online reinforcement learning attitude control method with safety assurance according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] This embodiment provides a quadrotor online reinforcement learning attitude control method with safety assurance, including:

[0047] Step S1, collecting quadrotor attitude control flight data, and obtaining the nominal dynamics representation and initial Gaussian process dynamics representation of the quadrotor attitude respectively;

[0048] Step S2, predicting long-term task rewards based on the initial Gaussian process dynamics representation;

[0049] Step S3, solving the gradient of the long-term task reward to the control strategy parameters, and updating the control strategy using the gradient descent method;

[0050] Step S4, performing safety filtering on the control strategy by solving the model predictive control problem through nominal dynamics;

[0051] Step S5, updating the training data set and the corresponding Gaussian process dynamics representation using a correlation-based data selection criterion, wherein the training data set includes attitude control flight data collected in real time during the quadrotor flight process in step S1;

[0052] Step S6, substitute the updated Gaussian process dynamics representation into step S2, and repeat steps S2-S5 continuously during the flight to achieve online reinforcement learning control of the quadrotor inner loop attitude.

[0053] Specifically, in view of the characteristics of nonlinear uncertainty in the dynamics of quadcopters, this embodiment designs a Gaussian process dynamics model representation method that does not require prior knowledge, and implements control strategy evaluation and control strategy improvement based on Gaussian process dynamics that take into account long-term uncertainty. Flight data is used to learn dynamics and control strategies at the same time, which can achieve efficient use of data. In view of the time complexity problem of Gaussian process model construction, a data priority selection mechanism based on data correlation is proposed to ensure the real-time performance of the controller's online learning. In view of the potential dangerous control signals that may be caused by model errors in the initial learning stage, a safety filtering layer is designed to ensure the safety of the learning-based controller.

[0054] Combine the following Figure 1-Figure 3 A four-rotor online reinforcement learning attitude control method with safety guarantee proposed in this embodiment is described in detail. Figure 2 , Figure 3 As shown in the figure, first, uncertainty dynamics learning is performed, then long-term reward prediction and estimation are performed, and finally, policy optimization is performed based on policy gradient. Specifically, the following steps are included:

[0055] Step S1, collecting quadrotor attitude control flight data, and obtaining the nominal dynamics representation and initial Gaussian process dynamics representation of the quadrotor attitude respectively;

[0056] Collecting the quadrotor attitude control flight data includes: randomly giving a control strategy, collecting the reference attitude signal, actual attitude signal and motor PWM signal output by the PID controller during the quadrotor flight.

[0057] Specifically, Figure 1 As shown in the figure, based on the real quadrotor bench test and the traditional PID controller, the attitude control flight data is collected and the system is identified to obtain the nominal dynamics. First, the traditional PID controller is used to stabilize the quadrotor fixed on the desktop bench. The remote control is used to give a random expected attitude signal, and the quadrotor can track the corresponding reference signal. During the flight, the reference attitude signal, the actual attitude signal, and the PWM signals of the four motors output by the PID controller are recorded.

[0058] (1) Nominal dynamics representation:

[0059] The attitude dynamics related parameters are identified from the flight data. The attitude dynamics of the quadrotor are expressed as follows:

[0060]

[0061] where τ T =[τ x ,τ y ,τ z ] T The three-axis torque provided to the propeller is expressed as:

[0062]

[0063] in, is the derivative of the rotation matrix from the machine system to the Earth system, is the rotation matrix from the machine system to the earth system, ω b =[ω x ,ω y ,ω z ] T is the three-axis angular velocity of the body, [·] × Represents the skew-symmetric matrix of the vector, J = [J x ,J y ,J z ] T is the three-axis moment of inertia of the machine body, is the derivative of the body angular velocity, τ T =[τ x ,τ y ,τ z ] T is the propeller's three-axis torque, τ D is the gyro torque, T is the transpose of the matrix, c M is the propeller torque coefficient, c T is the thrust coefficient of the propeller, and T1, T2, T3, and T4 represent the thrust of the four motors.

[0064] (2) Gaussian process dynamics representation:

[0065] like Figure 3 As shown, the quadrotor attitude dynamics represented by the Gaussian process is learned based on the collected data. Consider the following quadrotor attitude dynamics model:

[0066] x t+1 =f(x t ,a t )

[0067] Among them, xt +1 is the state of the quadrotor at the next moment, x tis the current quadrotor attitude, x t =[φ,θ,ψ] T represents the roll angle, pitch angle and yaw angle of the quadrotor, a t =[T1,T2,T3,T4] T It represents the thrust of the four motors, which can be calculated based on the motor PWM signal output by the PID controller at the current moment.

[0068] According to the collected flight data, the attitude dynamics are represented by a non-parametric Gaussian process model that can represent uncertainty. As training input, the state change As training output, the square exponential function is used as the kernel function:

[0069]

[0070] Among them, σ f ,σ w ,δ 12 is a hyperparameter.

[0071] Then the state of the system at the next moment satisfies the following Gaussian distribution:

[0072]

[0073] The expectation and variance are:

[0074]

[0075] in, is the training input matrix, y=[Δx1,…,Δx n ] is the training output matrix, K is the kernel matrix of the training data, I is the identity matrix, x t is the current quadrotor attitude, x t+1 is the state of the quadrotor at the next moment, a t is the action output by the control strategy at the current moment, i.e., the motor thrust, (x t ,a t ) is the quadrotor state action pair at the current moment, μ t+1 and∑ t+1 They represent the mean and variance of the state distribution at the next moment, Δx is the change in the quadrotor state, is the mean value of the state change, var f [Δx] is the variance of the state change.

[0076] Step S2, predicting long-term task rewards based on the initial Gaussian process dynamics representation;

[0077] Given a task objective and a reward function, where the task objective is to maximize the cumulative reward and the reward function is an exponential quadratic function, the maximum cumulative reward is solved based on the initial Gaussian process dynamics representation to obtain a long-term task reward representation.

[0078] Specifically, the long-term task return considering uncertainty is predicted based on Gaussian process dynamics, which is the strategy evaluation process. The goal of the quadrotor attitude control task is to construct a control strategy u=π(x) to make the quadrotor track the given reference attitude input. The task goal is expressed as maximizing the cumulative reward, that is:

[0079]

[0080] Among them, R π (θ) is the long-term task reward, represents the expectation of reward, r is the instantaneous reward at a certain moment, and T is the length of the future time step considered.

[0081] Choose the reward function to be an exponential quadratic function in is the reference pose input. Need to find the Gaussian distribution of the current state According to the Gaussian process model, we can calculate p(x t-1 ,a t-1 ) predict p(x t ). Given the joint distribution p(x t-1 )=p(x t-1 ,a t-1 ),but:

[0082] p(Δx)=∫∫p(f(x t )|x t )p(x t )dfdx t

[0083] From this we can get p(x t ) distribution:

[0084]

[0085] Step S3, solving the gradient of the long-term task reward to the control strategy parameters, and updating the control strategy using the gradient descent method;

[0086] In the previous step, we obtained the analytical expression of the long-term return, and we can calculate the long-term return R π The gradient dR of (θ) with respect to the control strategy parameter θ π (θ) / dθ, and perform the policy optimization process according to the gradient descent method, that is:

[0087] θ←θ+lr·(dR π (θ) / dθ)

[0088] Among them, θ is the control strategy parameter, lr is the learning rate, R π (θ) is the long-term task return, dR π (θ) / dθ is the gradient of the long-term task reward with respect to the control policy parameters.

[0089] Step S4, performing safety filtering on the control strategy by solving the model predictive control problem through nominal dynamics;

[0090] A safety filter layer is implemented by solving the model predictive control problem based on nominal dynamics to perform safety assurance on the learning-based controller. A safety filter layer is constructed using the model predictive control framework to filter the unsafe actions output by the learning policy during training and deployment and replace it with the closest safe action. The problem is transformed into solving the following constrained optimization problem:

[0091]

[0092] Among them, u ref (k) is the action output by the reinforcement learning controller at time k, x k+1 is the state of the quadrotor at time k+1, x k is the state of the quadrotor at time k, u k is the action of the quadrotor at time k, ΔT is the discrete time step length for solving the optimization problem, x0 is the initial state of the quadrotor, x init is the current starting state of the quadrotor, x min and x max is the safety boundary of the quadrotor state, u min and u max It is the safety boundary of the quadrotor action.

[0093] Step S5, updating the training data set and the corresponding Gaussian process dynamics representation using a data selection criterion based on data correlation;

[0094] Calculate the distance metric between the collected point data in the training data set, obtain the minimum value of the distance metric and the corresponding collected point data; calculate the distance metric between the new collected point data and the training data set, if the distance metric is greater than the minimum value of the distance metric, replace the collected point data corresponding to the minimum value of the distance metric with the new collected point data, and complete the update of the training data set; obtain a new Gaussian process dynamics representation based on the updated training data set, and complete the update of the Gaussian process dynamics representation.

[0095] Specifically, the data selection criteria based on data relevance in this embodiment include:

[0096] Since the computation time of the Gaussian process model increases exponentially with the growth of the data volume, this embodiment selects a set of fixed-volume data that contributes most to the model accuracy from all the data to balance the prediction accuracy and computational real-time performance of the Gaussian process model. Specifically, a distance metric is used to represent the coverage of the state space by the training data. The greater the coverage, the more accurately this set of data can represent the system dynamics. and the original dataset The distance measure between can be written as:

[0097]

[0098] The linear correlation coefficient vector w is obtained by minimizing γ, Get w * =K -1 k, that is:

[0099] γ new =kk·K -1 k

[0100] The process of data selection is as follows:

[0101] Initialize an empty data dictionary with capacity L

[0102] When the number of points in the training data set is less than L, the new points are directly added to the data dictionary;

[0103] Otherwise, if the γ of the new point new Greater than the minimum distance metric value γ in the data set min , use the new point to replace the minimum point.

[0104] Step S6, substitute the updated Gaussian process dynamics representation into step S2, and repeat steps S2-S5 continuously during the flight to achieve online reinforcement learning control of the quadrotor inner loop attitude.

[0105] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A quadrotor online reinforcement learning attitude control method with safety assurance, characterized in that: include: Step S1, collecting quadrotor attitude control flight data, and obtaining the nominal dynamics representation and initial Gaussian process dynamics representation of the quadrotor attitude respectively; Step S2, predicting long-term task rewards based on the initial Gaussian process dynamics representation; Step S3, solving the gradient of the long-term task reward to the control strategy parameters, and updating the control strategy using the gradient descent method; Step S4, performing safety filtering on the control strategy by solving the model predictive control problem through nominal dynamics; Step S5, updating a training data set and a corresponding Gaussian process dynamics representation using a correlation-based data selection criterion, wherein the training data set includes attitude control flight data collected in real time during the quadrotor flight process in step S1; Step S6, substitute the updated Gaussian process dynamics representation into step S2, and repeat steps S2-S5 continuously during the flight to achieve online reinforcement learning control of the quadrotor inner loop attitude.

2. The method for online reinforcement learning attitude control of a quadrotor with safety assurance according to claim 1 is characterized in that: The step S1 of collecting the quadrotor attitude control flight data includes: The control strategy is randomly given, and the reference attitude signal, actual attitude signal and motor PWM signal output by the PID controller are collected during the flight of the quadrotor.

3. The method for online reinforcement learning attitude control of a quadrotor with safety assurance according to claim 1 is characterized in that: In step S1, the nominal dynamics of the quadrotor attitude is expressed as: in, is the derivative of the rotation matrix from the machine system to the Earth system, is the rotation matrix from the machine system to the earth system, ω b =[ω x ,ω y ,ω z ] T is the three-axis angular velocity of the body, [·] × Represents the skew-symmetric matrix of the vector, J = [J x , J y , J z ] T is the three-axis moment of inertia of the machine body, is the derivative of the body angular velocity, τ T =[τ x , τ y , τ z ] T is the propeller's three-axis torque, τ D is the gyro torque, and T is the transpose of the matrix.

4. The method for online reinforcement learning attitude control of a quadrotor with safety assurance according to claim 1 is characterized in that: In step S1, the initial Gaussian process dynamics of the quadrotor attitude is expressed as: x t+1 =f(x t ,a t ) Among them, x t is the current quadrotor attitude, x t+1 is the state of the quadrotor at the next moment, a t is the action output by the control strategy at the current moment, i.e., the motor thrust, (x t , a t ) is the quadrotor state action pair at the current moment, μ t+1 and∑ t+1 They represent the mean and variance of the state distribution at the next moment, Δx is the change in the quadrotor state, is the mean value of the state change, var f [Δx] is the variance of the state change.

5. The method for online reinforcement learning attitude control of a quadrotor with safety assurance according to claim 1, characterized in that: The step S2 predicts the long-term task reward based on the initial Gaussian process dynamics representation, including: Given a task objective and a reward function, wherein the task objective is to maximize the cumulative reward, and the reward function is an exponential quadratic function; The maximum cumulative reward is solved based on the initial Gaussian process dynamics representation to obtain a long-term task reward representation.

6. The method for online reinforcement learning attitude control of a quadrotor with safety assurance according to claim 5 is characterized in that: The long-term task reward is expressed as: Among them, R π (θ) is the long-term task reward, represents the expectation of reward, r is the instantaneous reward at a certain moment, and T is the length of the future time step considered.

7. The method for online reinforcement learning attitude control of a quadrotor with safety assurance according to claim 1, characterized in that: The method for updating the control strategy using the gradient descent method in step S3 is: θ←θ+lr·(dR π (θ) / dθ) Among them, θ is the control strategy parameter, lr is the learning rate, R π (θ) is the long-term task return, dR π (θ) / dθ is the gradient of the long-term task reward with respect to the control policy parameters.

8. The method for online reinforcement learning attitude control of a quadrotor with safety assurance according to claim 1, characterized in that: The model predictive control problem in step S4 is: minutes u ||uu ref (k)|| 2 s.t.x k+1 =x k +f RK4 (x k ,u k ,ΔT) x0=x init x min ≤x k ≤x max in min in k in max Among them, u ref (k) is the action output by the reinforcement learning controller at time k, x k+1 is the state of the quadrotor at time k+1, x k is the state of the quadrotor at time k, u k is the action of the quadrotor at time k, ΔT is the discrete time step length for solving the optimization problem, x0 is the initial state of the quadrotor, x init is the current starting state of the quadrotor, x min and x max is the safety boundary of the quadrotor state, u min and u max It is the safety boundary of the quadrotor action.

9. The method for online reinforcement learning attitude control of a quadrotor with safety assurance according to claim 1, characterized in that: The step S5 uses the correlation-based data selection criteria to update the training data set and the corresponding Gaussian process dynamics representation, including: Calculate the distance metric between the collection point data in the training data set, and obtain the minimum value of the distance metric and the corresponding collection point data; Calculating the distance metric between the new collection point data and the training data set, and if the distance metric is greater than the minimum distance metric value, replacing the collection point data corresponding to the minimum distance metric value with the new collection point data to complete the updating of the training data set; A new Gaussian process dynamics representation is obtained based on the updated training data set, and the update of the Gaussian process dynamics representation is completed.

Images