Quadrotor online reinforcement learning attitude control method with security guarantee
By employing an online reinforcement learning attitude control method and utilizing a Gaussian process dynamics model and model predictive control, the model error and safety issues in quadrotor attitude control were resolved, achieving efficient and safe attitude control.
Patent Information
- Application Number
- CN202411933794.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-11-25
- Estimated Expiration
- 2044-12-26
AI Technical Summary
Existing quadrotor attitude control methods suffer from drawbacks such as large model errors, low data efficiency, lack of environmental adaptability, and lack of safety guarantees.
A safe online reinforcement learning attitude control method for quadrotors is adopted. By collecting flight data, nominal and Gaussian process dynamics are obtained, long-term mission rewards are predicted, gradients are solved and control strategies are updated, safety filtering is performed by combining model predictive control, and the training dataset is updated using data relevance selection criteria to achieve online reinforcement learning control.
It effectively handles the nonlinear uncertainties of quadcopter UAV dynamics, ensuring the real-time performance and security of the controller's online learning, and improving data utilization efficiency and environmental adaptability.
Smart Images
Figure CN119759082B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a safe online reinforcement learning attitude control method for quadrotors. Background Technology
[0002] With the widespread application of quadrotor aircraft in transportation, search and rescue, and other scenarios, researchers have proposed many novel quadrotor control methods. Precise low-level attitude control of the quadrotor is fundamental to achieving various flight missions. Model-based control methods, represented by model predictive control (MMC), have attracted widespread attention due to their stability, constraint flexibility, and predictability. However, due to the influence of wind disturbances, load variations, and aerodynamic effects, quadrotors flying in natural environments exhibit nonlinear dynamics that are difficult to model accurately, and model errors can lead to a decline in the performance of control methods. With the rapid development of machine learning algorithms, reinforcement learning and imitation learning methods have shown promising results in quadrotor control. Among them, model-based reinforcement learning methods can simultaneously learn dynamic models and control strategies from interactive data, exhibiting higher sampling efficiency compared to model-free reinforcement learning methods and stronger environmental adaptability compared to MMC. However, due to large model errors in the initial learning phase, dangerous control signals may be generated.
[0003] In summary, existing quadrotor attitude control methods suffer from drawbacks such as large model errors, low data efficiency, lack of environmental adaptability, and lack of safety guarantees. Therefore, this invention proposes a quadrotor online reinforcement learning attitude control method with safety guarantees. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a quadcopter online reinforcement learning attitude control method with safety guarantees, ensuring the real-time performance and safety performance of the controller's online learning.
[0005] To achieve the above objectives, the present invention provides the following solution:
[0006] A safety-assured online reinforcement learning attitude control method for quadrotors includes:
[0007] Step S1: Collect quadrotor attitude control flight data, and obtain the nominal dynamic representation and initial Gaussian process dynamic representation of the quadrotor attitude respectively;
[0008] Step S2: Predict long-term task rewards based on the initial Gaussian process dynamics representation;
[0009] Step S3: Solve for the gradient of the long-term task reward with respect to the control policy parameters, and update the control policy using the gradient descent method;
[0010] Step S4: Perform safety filtering on the control strategy by predicting the control problem through the nominal dynamics solution model;
[0011] Step S5: Update the training dataset and the corresponding Gaussian process dynamics representation using a correlation-based data selection criterion, wherein the training dataset includes the attitude control flight data collected in real time during the quadcopter flight in step S1.
[0012] Step S6: Substitute the updated Gaussian process dynamics representation into step S2, and repeat steps S2-S5 continuously during flight to achieve online reinforcement learning control of the quadrotor's inner ring attitude.
[0013] Optionally, the acquisition of quadcopter attitude control flight data in step S1 includes:
[0014] A randomized control strategy is given, and reference attitude signals, actual attitude signals, and motor PWM signals output by the PID controller are collected during the flight of the quadcopter.
[0015] Optionally, in step S1, the nominal dynamics of the quadrotor attitude are expressed as:
[0016]
[0017] in, The derivative of the rotation matrix from the machine system to the Earth system. The rotation matrix from the machine system to the Earth system, ω b =[ω x ,ω y ,ω z ] T For the three-axis angular velocities of the machine, [·] × Let J = [J] be a skew-symmetric matrix representing vectors. x J y J z ] T The moment of inertia of the machine body along its three axes. τ is the derivative of the body's angular velocity. T =[τ x ,τ y ,τ z ] T τ is the triaxial torque of the propeller. D Let T be the gyroscopic torque, and T be the transpose of the matrix.
[0018] Optionally, in step S1, the initial Gaussian process dynamics of the quadrotor attitude is expressed as:
[0019] x t+1 =f(x) t ,a t )
[0020]
[0021] Where, x t Given the current quadcopter attitude, x t+1 For the next moment, a t The action output by the control strategy at the current moment, i.e., the motor thrust, (x t ,a t ) represents the current quadcopter state action pair, μ t+1 and ∑ t+1 Let represent the mean and variance of the state distribution at the next moment, respectively, and Δx be the change in the state of the quadrotor. var is the mean of the state changes. f [Δx] represents the variance of the state change.
[0022] Optionally, the prediction of long-term task reward based on the initial Gaussian process dynamics representation in step S2 includes:
[0023] Given a task objective and a reward function, where the task objective is to maximize the cumulative reward and the reward function is an exponential quadratic function;
[0024] The maximum cumulative reward is solved based on the initial Gaussian process dynamics representation to obtain the long-term task reward representation.
[0025] Optionally, the long-term task reward is represented as:
[0026]
[0027] Among them, R π (θ) represents the reward for long-term tasks. Let r represent the expected reward, r be the instantaneous reward at a certain moment, and T be the length of the future time step under consideration.
[0028] Optionally, the method for updating the control strategy using gradient descent in step S3 is as follows:
[0029] θ←θ+lr·(dR π (θ) / dθ)
[0030] Where θ is the control policy parameter, lr is the learning rate, and R... π (θ) represents the long-term task reward, dR π (θ) / dθ is the gradient of the long-term task reward with respect to the control policy parameters.
[0031] Optionally, the model predictive control problem in step S4 is:
[0032]
[0033] Among them, u ref (k) represents the action output by the reinforcement learning controller at time k, x k+1 Let x be the state of the quadcopter at time k+1. k Let u be the state of the quadcopter at time k. k Let x be the motion of the quadcopter at time k, ΔT be the discrete time step length for solving the optimization problem, x0 be the initial state of the quadcopter, and x init This is the current initial state of the quadcopter, x min and x max For the safety boundary of the quadcopter configuration, u min and u max The safety boundary for quadcopter operation.
[0034] Optionally, step S5, which uses a correlation-based data selection criterion to update the training dataset and the corresponding Gaussian process dynamics representation, includes:
[0035] Calculate the distance metric between the data points collected in the training dataset, and obtain the minimum distance metric and the corresponding data point data.
[0036] Calculate the distance metric between the new collection point data and the training dataset. If the distance metric is greater than the minimum distance metric, replace the collection point data corresponding to the minimum distance metric with the new collection point data to complete the update of the training dataset.
[0037] A new Gaussian process dynamics representation is obtained based on the updated training dataset, thus completing the update of the Gaussian process dynamics representation.
[0038] The beneficial effects of this invention are as follows:
[0039] This invention provides a safe online reinforcement learning attitude control method for quadrotors. Addressing the nonlinear uncertainties in quadrotor UAV dynamics, it designs a Gaussian process dynamics model representation method that requires no prior knowledge. Based on Gaussian process dynamics, it implements control strategy evaluation and improvement considering long-term uncertainties. Flight data is used simultaneously for learning dynamics and control strategies, enabling efficient data utilization. To address the time complexity of Gaussian process model construction, a data priority selection mechanism based on data correlation is proposed, ensuring the real-time performance of the controller's online learning. For potential dangerous control signals caused by model errors in the initial learning stage, a safety filtering layer is designed to ensure the safety of the learning-based controller. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0041] Figure 1 This is a schematic diagram of flight data acquisition using a quadcopter fixed on a test bench according to an embodiment of the present invention;
[0042] Figure 2 This is a basic flowchart of the online reinforcement learning attitude control for a quadcopter according to an embodiment of the present invention;
[0043] Figure 3 This is a flowchart illustrating a quadrotor online reinforcement learning attitude control method with security assurance, according to an embodiment of the present invention. Detailed Implementation
[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0046] This embodiment provides a safe online reinforcement learning attitude control method for quadrotors, including:
[0047] Step S1: Collect quadrotor attitude control flight data, and obtain the nominal dynamic representation and initial Gaussian process dynamic representation of the quadrotor attitude respectively;
[0048] Step S2: Predict long-term task rewards based on the initial Gaussian process dynamics representation;
[0049] Step S3: Solve for the gradient of the long-term task reward with respect to the control policy parameters, and update the control policy using the gradient descent method;
[0050] Step S4: Perform safety filtering on the control strategy by predicting the control problem through the nominal dynamics solution model;
[0051] Step S5: Update the training dataset and the corresponding Gaussian process dynamics representation using the correlation-based data selection criterion. The training dataset includes the attitude control flight data collected in real time during the quadcopter flight in step S1.
[0052] Step S6: Substitute the updated Gaussian process dynamics representation into step S2, and repeat steps S2-S5 continuously during flight to achieve online reinforcement learning control of the quadrotor's inner ring attitude.
[0053] Specifically, this embodiment addresses the nonlinear uncertainty of quadcopter UAV dynamics by designing a Gaussian process dynamics model representation method that requires no prior knowledge. Based on Gaussian process dynamics, it implements control strategy evaluation and improvement considering long-term uncertainties. Flight data is used simultaneously for learning dynamics and control strategies, enabling efficient data utilization. To address the time complexity of Gaussian process model construction, a data priority selection mechanism based on data correlation is proposed, ensuring the real-time performance of the controller's online learning. To address potential dangerous control signals caused by model errors in the initial learning stage, a safety filtering layer is designed to ensure the safety of the learning-based controller.
[0054] The following is combined Figures 1-3 This embodiment provides a detailed description of a safety-assured online reinforcement learning attitude control method for quadrotors, such as... Figure 2 , Figure 3 As shown, the process first involves uncertainty dynamics learning, then long-term return prediction and estimation, and finally policy optimization based on policy gradients. Specifically, it includes the following steps:
[0055] Step S1: Collect quadrotor attitude control flight data, and obtain the nominal dynamic representation and initial Gaussian process dynamic representation of the quadrotor attitude respectively;
[0056] The acquisition of quadcopter attitude control flight data includes: randomly given control strategy, acquisition of reference attitude signal, actual attitude signal and motor PWM signal output by PID controller during quadcopter flight.
[0057] Specifically, such as Figure 1 As shown, nominal dynamics are obtained by collecting attitude control flight data and performing system identification based on real quadcopter bench tests and a traditional PID controller. First, a traditional PID controller is used to stabilize the quadcopter fixed on a desktop bench. A random desired attitude signal is given by the remote controller, and the quadcopter can track the corresponding reference signal. During flight, the reference attitude signal, the actual attitude signal, and the PWM signals of the four motors output by the PID controller are recorded.
[0058] (1) Nominal dynamic representation:
[0059] Attitude dynamics parameters were identified from the flight data, and the quadrotor attitude dynamics are represented as follows:
[0060]
[0061] Where τ T =[τ x ,τ y ,τ z ] T The three-axis torque provided to the propeller is expressed as:
[0062]
[0063] in, The derivative of the rotation matrix from the machine system to the Earth system. The rotation matrix from the machine system to the Earth system, ω b =[ω x ,ω y ,ω z ] T For the three-axis angular velocities of the machine, [·] × Let J = [J] be a skew-symmetric matrix representing vectors. x J y J z ] T The moment of inertia of the machine body along its three axes. τ is the derivative of the body's angular velocity. T =[τ x ,τ y ,τ z ] T τ is the triaxial torque of the propeller. D Let T be the gyroscopic torque, T be the transpose of the matrix, and c be the torque. M c is the torque coefficient of the propeller. T T1, T2, T3, and T4 represent the thrust coefficients of the propeller and the thrust of the four motors, respectively.
[0064] (2) Dynamic representation of Gaussian process:
[0065] like Figure 3 As shown, the quadrotor attitude dynamics are learned using a Gaussian process representation based on the collected data. Consider the following quadrotor attitude dynamics model:
[0066] x t+1 =f(x) t ,a t )
[0067] Among them, xt +1 For the next moment, x tGiven the current quadcopter attitude, x t =[φ,θ,ψ] T This indicates the roll, pitch, and yaw angles of a quadcopter. t =[T1,T2,T3,T4] T This represents the thrust of the four motors, which can be calculated based on the motor PWM signal output by the PID controller at the current moment.
[0068] Based on the collected flight data, attitude dynamics are represented using a non-parametric Gaussian process model that can represent uncertainty. The current state-action pair is shown. As training input, state change As the training output, the squared exponential function is used as the kernel function:
[0069]
[0070] Where, σ f ,σ w ,δ 12 This is a hyperparameter.
[0071] The system's state at the next time step then follows a Gaussian distribution:
[0072]
[0073] Where the expected value and variance are respectively:
[0074]
[0075] in, The training input matrix is y = [Δx1, ..., Δx]. n [] is the training output matrix, K is the kernel matrix of the training data, I is the identity matrix, x t Given the current quadcopter attitude, x t+1 For the next moment, a t The action output by the control strategy at the current moment, i.e., the motor thrust, (x t ,a t ) represents the current quadcopter state action pair, μ t+1 and ∑ t+1 Let represent the mean and variance of the state distribution at the next moment, respectively, and Δx be the change in the state of the quadrotor. var is the mean of the state changes. f [Δx] represents the variance of the state change.
[0076] Step S2: Predict long-term task rewards based on the initial Gaussian process dynamics representation;
[0077] Given a task objective and a reward function, where the task objective is to maximize the cumulative reward and the reward function is an exponential quadratic function; the solution to maximize the cumulative reward is obtained based on the initial Gaussian process dynamics representation, thus obtaining the long-term task reward representation.
[0078] Specifically, the strategy evaluation process involves predicting long-term task rewards considering uncertainties based on Gaussian process dynamics. The objective of the quadrotor attitude control task is to construct a control policy u = π(x) that enables the quadrotor to track a given reference attitude input. The task objective is expressed as maximizing the cumulative reward, i.e.:
[0079]
[0080] Among them, R π (θ) represents the reward for long-term tasks. Let r represent the expected reward, r be the instantaneous reward at a certain moment, and T be the length of the future time step under consideration.
[0081] Choose an exponential quadratic function as the reward function. in It is the reference attitude input. And... We need to find the Gaussian distribution of the current state. According to the Gaussian process model, we can determine p(x) t-1 ,a t-1 Predict p(x) t Given the joint distribution p(x) t-1 )=p(x t-1 ,a t-1 ),but:
[0082] p(Δx)=∫∫p(f(x t )∣x t )p(x t )dfdx t
[0083] Therefore, p(x) t Distribution of )
[0084]
[0085] Step S3: Solve for the gradient of the long-term task reward with respect to the control policy parameters, and update the control policy using the gradient descent method;
[0086] Having obtained the analytical expression for the long-term return in the previous step, we can now calculate the long-term return R. π (θ) gradient dR with respect to the control strategy parameter θ π (θ) / dθ, and perform policy optimization according to the gradient descent method, that is:
[0087] θ←θ+lr·(dR π (θ) / dθ)
[0088] Where θ is the control policy parameter, lr is the learning rate, and R... π (θ) represents the long-term task reward, dR π (θ) / dθ is the gradient of the long-term task reward with respect to the control policy parameters.
[0089] Step S4: Perform safety filtering on the control strategy by predicting the control problem through the nominal dynamics solution model;
[0090] A safety filter layer is implemented based on nominal dynamics to solve the model predictive control problem, ensuring the safety of the learning-based controller. A safety filter layer is constructed using a model predictive control framework to filter unsafe actions output by the learned policy during training and deployment, replacing them with the closest safe action. The problem is transformed into solving the following constrained optimization problem:
[0091]
[0092] Among them, u ref (k) represents the action output by the reinforcement learning controller at time k, x k+1 Let x be the state of the quadcopter at time k+1. k Let u be the state of the quadcopter at time k. k Let x be the motion of the quadcopter at time k, ΔT be the discrete time step length for solving the optimization problem, x0 be the initial state of the quadcopter, and x init This is the current initial state of the quadcopter, x min and x max For the safety boundary of the quadcopter configuration, u min and u max The safety boundary for quadcopter operation.
[0093] Step S5: Update the training dataset and the corresponding Gaussian process dynamics representation using a data selection criterion based on data relevance;
[0094] Calculate the distance metric between the data points collected in the training dataset, obtain the minimum distance metric and the corresponding data point; calculate the distance metric between the new data point and the training dataset, if the distance metric is greater than the minimum distance metric, replace the data point corresponding to the minimum distance metric with the new data point, and complete the update of the training dataset; obtain a new Gaussian process dynamics representation based on the updated training dataset, and complete the update of the Gaussian process dynamics representation.
[0095] Specifically, the data selection criteria based on data relevance in this embodiment include:
[0096] Since the computation time of Gaussian process models increases exponentially with the amount of data, this embodiment selects a fixed-size dataset that contributes the most to the model's accuracy from all data, thus balancing the prediction accuracy and real-time computation of the Gaussian process model. Specifically, a distance metric is used to represent the degree to which the training data covers the state space; a greater degree of coverage indicates that the dataset accurately represents the system dynamics. A new dataset... and the original dataset The distance metric between them can be written as:
[0097]
[0098] The linear correlation coefficient vector w is obtained by minimizing γ. Let Get w * =K -1 k, that is:
[0099] γ new =kk·K -1 k
[0100] The data selection process is as follows:
[0101] Initialize an empty data dictionary of capacity L.
[0102] When the number of points in the training dataset is less than L, the new points are directly added to the data dictionary;
[0103] Otherwise, if the new point's γ new Greater than the minimum distance metric γ in the dataset min Replace the minimum point with a new point.
[0104] Step S6: Substitute the updated Gaussian process dynamics representation into step S2, and repeat steps S2-S5 continuously during flight to achieve online reinforcement learning control of the quadrotor's inner ring attitude.
[0105] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. A quadrotor online reinforcement learning attitude control method with safety assurance, characterized in that, include: Step S1: Collect quadrotor attitude control flight data, and obtain the nominal dynamic representation and initial Gaussian process dynamic representation of the quadrotor attitude respectively; Step S2: Predict long-term task rewards based on the initial Gaussian process dynamics representation; Step S3: Solve for the gradient of the long-term task reward with respect to the control policy parameters, and update the control policy using the gradient descent method; Step S4: Perform safety filtering on the control strategy by predicting the control problem through the nominal dynamics solution model; Step S5: Update the training dataset and the corresponding Gaussian process dynamics representation using a correlation-based data selection criterion, wherein the training dataset includes the attitude control flight data collected in real time during the quadcopter flight in step S1. Step S6: Substitute the updated Gaussian process dynamics representation into step S2, and repeat steps S2-S5 continuously during flight to achieve online reinforcement learning control of the quadrotor's inner ring attitude.
2. The quadrotor online reinforcement learning attitude control method with safety assurance according to claim 1, characterized in that, The acquisition of quadrotor attitude control flight data in step S1 includes: A randomized control strategy is given, and reference attitude signals, actual attitude signals, and motor PWM signals output by the PID controller are collected during the flight of the quadcopter.
3. The quadrotor online reinforcement learning attitude control method with safety assurance according to claim 1, characterized in that, In step S1, the nominal dynamics of the quadrotor attitude are expressed as follows: in, The derivative of the rotation matrix from the machine system to the Earth system. The rotation matrix from the machine system to the Earth system, ω b =[ω x ω y ω z ] T For the three-axis angular velocities of the machine, [·] × Let J = [J] be a skew-symmetric matrix representing vectors. x J y J z ] T The moment of inertia of the machine body along its three axes. τ is the derivative of the body's angular velocity. T =[τ x , τ y , τ z ] T τ is the three-axis torque of the propeller. D Let T be the gyroscopic torque, and T be the transpose of the matrix.
4. The quadrotor online reinforcement learning attitude control method with safety assurance according to claim 1, characterized in that, In step S1, the initial Gaussian process dynamics of the quadrotor attitude are expressed as follows: x t+1 =f(x t ,a t ) Where, x t Given the current quadcopter attitude, x t+1 For the next moment, a t The action output by the control strategy at the current moment, i.e., the motor thrust, (x t a t ) represents the current quadcopter state action pair, μ t+1 and ∑ t+1 Let represent the mean and variance of the state distribution at the next moment, respectively, and Δx be the change in the state of the quadrotor. var is the mean of the state changes. f [Δx] represents the variance of the state change.
5. The quadrotor online reinforcement learning attitude control method with safety assurance according to claim 1, characterized in that, The step S2, which predicts the long-term task reward based on the initial Gaussian process dynamics representation, includes: Given a task objective and a reward function, where the task objective is to maximize the cumulative reward and the reward function is an exponential quadratic function; The maximum cumulative reward is solved based on the initial Gaussian process dynamics representation to obtain the long-term task reward representation.
6. The quadrotor online reinforcement learning attitude control method with safety assurance according to claim 5, characterized in that, The reward for the long-term task is expressed as follows: Among them, R π (θ) represents the reward for long-term tasks. Let r represent the expected reward, r be the instantaneous reward at a certain moment, and T be the length of the future time step under consideration.
7. The quadrotor online reinforcement learning attitude control method with safety assurance according to claim 1, characterized in that, The method for updating the control strategy using gradient descent in step S3 is as follows: θ←θ+lr·(dR π (θ) / dθ) Where θ is the control policy parameter, lr is the learning rate, and R... π (θ) represents the long-term task reward, dR π (θ) / dθ is the gradient of the long-term task reward with respect to the control policy parameters.
8. The quadrotor online reinforcement learning attitude control method with safety assurance according to claim 1, characterized in that, The model predictive control problem in step S4 is: minutes u ||uu ref (k)|| 2 s.t.x k+1 =x k +f RK4 (x k ,u k ,ΔT) x0=x init x min ≤x k ≤x max in min in k in max Among them, u ref (k) represents the action output by the reinforcement learning controller at time k, x k+1 Let x be the state of the quadcopter at time k+1. k Let u be the state of the quadcopter at time k. k Let x be the motion of the quadcopter at time k, ΔT be the discrete time step length for solving the optimization problem, x0 be the initial state of the quadcopter, and x init This is the current initial state of the quadcopter, x min and x max For the safety boundary of the quadcopter configuration, u min and u max The safety boundary for quadcopter operation.
9. The quadrotor online reinforcement learning attitude control method with safety assurance according to claim 1, characterized in that, Step S5, which uses a correlation-based data selection criterion to update the training dataset and the corresponding Gaussian process dynamics representation, includes: Calculate the distance metric between the data points collected in the training dataset, and obtain the minimum distance metric and the corresponding data point data. Calculate the distance metric between the new collection point data and the training dataset. If the distance metric is greater than the minimum distance metric, replace the collection point data corresponding to the minimum distance metric with the new collection point data to complete the update of the training dataset. A new Gaussian process dynamics representation is obtained based on the updated training dataset, thus completing the update of the Gaussian process dynamics representation.