Quad-rotor unmanned aerial vehicle flight attitude control method based on data driving
Through data-driven LSTM neural network and reinforcement learning method, combined with Soft Actor Critic algorithm, the problems of limitations of physical models and uncertainty in training results in attitude control of quadrotor drone are solved, achieving higher accuracy and stability Attitude control.
Patent Information
- Application Number
- CN202411203105.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, the attitude control method of the four-rotor UAV based on physical models has limitations, and the reinforcement learning algorithm has uncertainty in the results and difficulty in reproducing during the training process, which hinders its wide application in practical applications.
Using a data-driven method, the LSTM neural network is designed in combination with reinforcement learning, through the construction of data sets and designing Markov decision-making process, the attitude control is used using the Soft Actor Critic algorithm, and it is established on the GymFC simulation flight framework for training and optimization.
It significantly improves the accuracy of pose estimation, reduces pose control errors, enhances the intelligence of the system, and solves the problems of traditional methods in model integrity and training stability.
Smart Images

Figure CN118938990A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for controlling the flight attitude of an unmanned aerial vehicle, and in particular to a method for controlling the flight attitude of a quad-rotor unmanned aerial vehicle based on data drive. Background Art
[0002] In recent years, the development prospects of unmanned aerial vehicles (UAVs) are broad, and the promotion of military and civilian use has deepened people's understanding of them. Unmanned aerial vehicles (UAVs) are unmanned aircraft controlled by remote control and self-contained programs. Due to the characteristics of strong flexibility, low cost and high reliability, UAVs are widely used in various fields, such as disaster relief, agricultural plant protection, aerial photography and mapping, inspection and patrol, etc.
[0003] The most common multi-rotor aircraft is a quad-rotor drone, which can quickly adjust the lift to achieve various flight postures by adjusting the speed of the four propellers. The quad-rotor drone is a typical multi-input and multi-output, under-driven and strongly coupled nonlinear system.
[0004] Moreover, during the flight, due to the small size and light load of the quadcopter, the interference from external uncertainties will have a significant impact on the flight performance. In addition, the dynamic model of the quadcopter is relatively complex, and some dynamic model parameters cannot be accurately measured and modeled. When the load changes, the parameters such as the moment of inertia of the aircraft may be perturbed. These uncertain interference factors also increase the difficulty of designing the flight control system, and are also one of the issues that must be considered in the design of the flight control system. Attitude control is the basis and key point of the entire flight control system of the quadcopter. Excellent attitude control is a prerequisite for ensuring the stable flight of the aircraft. At the same time, attitude control performance also directly determines the quality of the position control of the aircraft. Therefore, a large number of scientific researchers and enterprises have conducted extensive research in related fields. Aircraft attitude estimation is a major research hotspot in this field. It is very important to accurately estimate the attitude of the aircraft when the aircraft performs a mission. Providing accurate attitude in real time to keep the quadcopter in a stable flight attitude and avoid unnecessary casualties and losses has practical research significance.
[0005] In UAV control tasks, they can be divided into high-level and low-level tasks according to the task type. The former include path planning, autonomous obstacle avoidance, landing, etc.; the latter include position and attitude control. Common UAV flight control systems are expressed as graphs with inner and outer loop structures. When the desired position and yaw angle are given, the trajectory tracker calculates the desired attitude angle and reference thrust of the UAV, and then the attitude controller tracks and controls it. Usually, the outer loop calculates at a lower rate, while the inner loop needs to dynamically control the attitude at a much faster speed than the position tracking calculation. Therefore, UAV attitude control not only needs to ensure accuracy, but also has high requirements in terms of speed.
[0006] At present, the control technologies used by everyone mainly include active disturbance rejection control, adaptive control, backstepping control, data-driven control, active disturbance rejection control, sliding mode control, etc.
[0007] When considering a large range of dynamic, static rotation and translational motion, traditional attitude fusion algorithms need to adjust the fusion weights between the gyroscope, accelerometer and magnetometer according to different motion states, and involve complex model compensation and parameter adjustment, which limits the achievable accuracy.
[0008] In order to solve the limitations of the physical model-based method, a deep learning-based attitude control method for quadcopters was proposed, using the long short-term memory (LSTM) neural network for attitude estimation. Although reinforcement learning has the above-mentioned significant advantages over traditional control strategies, it is difficult to train in actual applications. This is mainly reflected in the sensitivity of most algorithms to their own parameters, reward calculation methods, random number seeds, and observation space settings, which leads to significant differences in training results. This makes the reproduction of related work more uncertain and difficult to achieve the expected results, which seriously hinders the further application of reinforcement learning technology. Summary of the invention
[0009] The purpose of the present invention is to solve the limitations of the physical model method and the problem that there are obvious differences in the training results of the reinforcement learning algorithm, which hinders the reinforcement learning technology, and propose a data-driven quadrotor drone flight attitude control method.
[0010] The above purpose is achieved through the following technical solutions:
[0011] A data-driven flight attitude control method for a quad-rotor unmanned aerial vehicle comprises designing an LSTM neural network and constructing a data set for attitude estimation. The overall structure of the LSTM neural network is as follows: the attitude angle is divided into a horizontal attitude angle and a yaw angle, and a LSTM neural network is used to take a gyroscope output, an accelerometer output, and the horizontal attitude angles at the previous n moments as inputs to solve the horizontal attitude angle at the current moment, and another LSTM neural network is used to take a gyroscope output, a magnetometer output, and the yaw angles at the previous n moments as inputs to solve the yaw angle at the current moment, thereby finally achieving full attitude estimation;
[0012] The data in the data set are MEMS and inertial measurement system sensor data installed on the drone, and drone attitude information output by the Vicon system; the sensor data include gyroscope output, accelerometer output and magnetometer output.
[0013] Furthermore, the process of designing the LSTM neural network includes:
[0014] First, the steps to build the LSTM neural network prediction model;
[0015] Second, the steps to determine the loss function and evaluation index;
[0016] Third, a quadcopter attitude control method is designed based on the reinforcement learning method; first, the Markov decision process of the problem is defined by designing the state space, action space and reward function of the problem, and a reinforcement learning model is established. It is expected to be implemented on the simulation platform in combination with the GymFC simulation flight framework; secondly, the proximal strategy optimization algorithm is used to determine the reward function through analysis and comparison of simulation training and evaluation results.
[0017] Furthermore, the steps of constructing the LSTM neural network prediction model are specifically as follows:
[0018] The LSTM-based posture estimation framework includes an input layer, a hidden layer, and an output layer. The input layer is used to reconstruct the input and output features of the time series and divide the training set and the test set. At the same time, the data set is normalized and mapped to the range of 0 to 1.
[0019] The hidden layer contains several LSTM units, which are mainly used to learn the special mapping relationship between input features and output features; reasonable adjustments are made according to the training results;
[0020] The output layer is used to output the final result. The output value is obtained according to the forward propagation calculation method, and the value is back-propagated into the training network to update the weights and the updated weights are sent to the hidden layer to update the model. It is iterated in sequence until the weights converge. Finally, the prediction results are denormalized to obtain the final posture estimation result.
[0021] Furthermore, the steps of determining the loss function and the evaluation index are specifically as follows:
[0022] The loss function is used to represent the cumulative error between the estimated value and the true value. Smooth-L1-loss is selected as the loss function, and the calculation formula is: Where: n is the sequence length; is the true value yi and the predicted value The difference between
[0023] The optimizer uses an adaptive momentum estimator; the data is repeatedly iterated through the above units and the weights are updated to minimize the loss function;
[0024] The evaluation index of attitude estimation output is the root mean square error S RMSE , S RMSE Sensitive to outliers, the fewer unreasonable points in the posture estimation output, the closer the result is to the true value, then S RMSE The smaller the value, the RMSE The calculation formula is:
[0025]
[0026] Furthermore, the Markov decision process is specifically as follows:
[0027] Markov decision process is a classic expression of sequential decision process; the action performed at a certain moment not only affects the immediate reward and the next state, but also affects the future benefits and states, thus generating delayed benefits; the current state and action of the decision process are only affected by the state and action of the previous moment, and have nothing to do with other moments before; the second-order to multi-order Markov properties are deduced by analogy;
[0028] MDP is an ideal framework for studying the interaction process in RL. The agent moves at time t according to the current state s. t and policy execution action a t , the environment is affected by a t The impact of returns a numerical reward (Reward) representing its performance, and is represented by the state s t Convert to s t+1 ; Continuous interaction enables the agent to learn the strategy that maximizes the cumulative reward;
[0029] Define Gt as a specific function to obtain the reward sequence after time t, which represents the expected return, and introduces the form of discount factor expressed as G t =r t +γr t+1 +γ 2 r t+2 +...+γ T-1 rT-1 +γ T r T ,The discount factor indicates that the impact of the action performed at time t on the future will gradually decrease over time;
[0030] In the form of the discount factor, T is the final time, r t Represents the reward value obtained at time t; if the interaction task between the agent and the environment is continuous, then T is ∞, and this task is called a continuous task; otherwise T is a finite value, and the interaction process is divided into a series of subsequences containing terminal states. Tasks with this characteristic are called episodic tasks, and the subsequences are called episodes.
[0031] Furthermore, the Markov decision process of defining the problem by designing the state space of the problem is specifically:
[0032] The state space is the source of the agent's perception of external information in the reinforcement learning interaction framework. The information that can be obtained and used by the agent is measured by the sensor. For the entire external environment, this is a partially perceived process, so the acquired state is equivalent to a subset of the complete space. In the attitude control process, in order to achieve roll θ, pitch As well as the tracking control of the three angular velocities of yaw ψ, the state space includes three angular velocity errors, namely:
[0033]
[0034] In addition, if there is a time delay between the action and the state, the historical state value needs to be used as part of the neural network input, and the state space is set to consist of the current angular velocity error and the error at the previous moment:
[0035]
[0036] Furthermore, the Markov decision process of defining the problem by designing the action space of the problem is specifically as follows: the action space is a set of actions that the agent can take, corresponding to the control signal of the motor; in the actual task, the motor speed is adjusted by adjusting the PWM output of the electric speed controller, thereby achieving the purpose of posture control; the control signal is expressed as the allowed energy output ratio, and the value range is [0,1]; thus, the action space can be expressed as:
[0037] A=<a1,a2,a3,a4>
[0038] Among them: a i ,i=1,2,3,4 represents the four motor control signals of the drone.
[0039] Furthermore, the Markov decision process of the problem is defined by designing the reward function of the problem, specifically: the reward function determines the performance of the final strategy; in the posture control task, the tracking error is used as a measure of the size of the reward, and the two are inversely correlated, that is, the larger the tracking error, the smaller the reward value, so its value is set to a negative value to indicate the penalty for inaccurate instruction tracking; that is:
[0040] r∞-|e|.
[0041] Furthermore, the method further comprises a reinforcement learning algorithm, and the reinforcement learning algorithm selects a Soft ActorCritic algorithm;
[0042] As an off-policy and random strategy algorithm, Soft Actor Critic involves increasing the entropy of actions on the basis of maximizing expected returns, that is, completing the task with as many possible actions as possible to achieve diversity of solutions.
[0043] The beneficial effects of the present invention are:
[0044] The present invention is aimed at the requirements of intelligent operation of quad-rotor UAVs and combines the technical difficulties of current UAV flight attitude control, and adopts data-driven methods including deep learning and reinforcement learning to realize the flight attitude control of quad-rotor UAVs, thereby improving their intelligence. Specifically:
[0045] (1) Compared with the traditional attitude estimation algorithm, a quadrotor UAV attitude control method based on deep learning is proposed to address the limitations of the method based on the physical model. The error of the attitude estimation algorithm based on the LSTM neural network is greatly reduced, and the accuracy is improved.
[0046] (2) Research on attitude control of quadrotor drone based on reinforcement learning. Taking quadrotor drone as the research object, its underactuation and high coupling characteristics become a challenge for flight controller design. At the same time, it is susceptible to various unstable factors in practical applications, which increases the difficulty of the design process; in addition, with the development of artificial intelligence technology, the design of flight control is also developing towards autonomous control systems. The method of designing controllers based on modeling has difficulties in terms of model completeness, comprehensiveness and accuracy. To address this problem, a reinforcement learning-based attitude control scheme is studied, and the strategy is trained and optimized through data-driven learning. The deep reinforcement learning (DRL) method is adopted. Considering the advantages of unlabeled data learning, generalization and learning ability of unmodeled factors, and the achievements in various robot control tasks, the model-free deep reinforcement learning method provides a possibility and reference for solving the above problems.
[0047] (3) Based on the modeling of the attitude control problem, the GymFC framework is used to design and implement a reinforcement learning simulation platform; secondly, the proximal strategy optimization algorithm is used to study the effects of three reward methods and related parameters on training and control effects; finally, in response to the problem that some algorithms have strong unstable performance during the training process, which leads to poor experimental repeatability, a UAV attitude control based on the SAC algorithm is proposed. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a flow chart of the method involved in the present invention;
[0049] Figure 2 It is a diagram of the overall structure of the LSTM neural network involved in the present invention;
[0050] Figure 3 The present invention relates to a Markov decision process for attitude control of a quad-rotor drone;
[0051] Figure 4 It is a diagram of the LSTM-based posture estimation framework involved in the present invention;
[0052] Figure 5 It is a Markov decision process involved in the present invention. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] In the attitude control of a quadrotor drone, various reward functions are designed by mapping the attitude angular velocity tracking error to the state space in the Markov decision process, and the motor control signal is simplified to the energy proportion in the interval [0,1] as the representation of the action space, so as to establish a reinforcement learning model for attitude control. At the same time, a simulation platform is built based on the simulation flight framework GymFC. In the simulation experiment, the proximal policy optimization algorithm (PPO) is applied to discuss and compare the effects of various reward settings and related parameters on the training and attitude control effects, and determine the reward calculation method with outstanding performance. In view of the fact that TD3 and PPO have different degrees of differences in training results in this task, resulting in poor experimental repeatability, a drone attitude control based on the SAC algorithm combined with the maximum entropy framework is proposed.
[0055] By defining the Markov decision process of the problem through the design of the state space, action space and reward function of the problem, the reinforcement learning model is established, and the simulation platform is implemented in combination with the GymFC simulation flight framework. Secondly, using the proximal policy optimization algorithm (PPO), the influence of various reward methods and related parameters, such as normalization factor and learning rate, on the training and control results is discussed. By analyzing and comparing the simulation training and evaluation results, the reward function with outstanding performance is determined. Finally, for the problem that the training results of Twin Delayed Deep Deterministic Policy Gradient (TD3) and Proximal Policy Optimization Algorithm (PPO) are obviously different to different degrees, attitude control based on Soft Actor Critic (SAC) is proposed. Specific implementation method one:
[0057] See also Figure 1-Figure 5 , a data-driven quad-rotor drone flight attitude control method in this embodiment, such as Figure 1 As shown, the method includes designing an LSTM neural network and constructing a data set for posture estimation. The overall structure of the LSTM neural network is as follows Figure 2 As shown in the figure, the attitude angle is divided into the horizontal attitude angle and the yaw angle, and an LSTM neural network is used to take the gyroscope output, the accelerometer output and the horizontal attitude angle of the previous n moments as input to solve the horizontal attitude angle at the current moment, and another LSTM neural network is used to take the gyroscope output, the magnetometer output and the yaw angle of the previous n moments as input to solve the yaw angle at the current moment, and finally the full attitude estimation is realized;
[0058] The dataset used is collected from the sensor suite installed on the quadrotor "Pelican", equipped with a Vicon capture system, and the aircraft is actually flying in an indoor environment of 10m×10m×10m. The dataset contains three subsets, "1LoopDown", "2LoopsDown", and "3LoopsDown", which correspond to three real flights respectively, with increasing flight time, and the postures involved in the flight process can basically meet the postures involved in normal flight.
[0059] The data in the dataset are mainly MEMS and inertial measurement unit (IMU) sensor data installed on the drone, and drone attitude information output by the Vicon system; the sensor data include gyroscope output, accelerometer output and magnetometer output. Specific implementation method 2:
[0061] Different from the first specific implementation, the present implementation is a data-driven quadrotor drone flight attitude control method, and the process of designing the LSTM neural network includes:
[0062] First, the steps to build the LSTM neural network prediction model;
[0063] Second, the steps to determine the loss function and evaluation index;
[0064] Third, reinforcement learning technology is regarded as the solution closest to artificial intelligence, especially when combined with deep learning (called deep reinforcement learning, DRL), which trains neural networks that can complete target tasks in a data-driven way, and thus gradually receives widespread attention and application. Neural networks show memory and learning capabilities through constantly changing parameters, which makes it possible to learn unmodeled factors and thus improve actual performance. At the same time, the numerous parameters of the fully connected network structure also provide strong anti-interference capabilities. Unlike supervised learning, RL is not limited by labeled data, and the network is continuously optimized through data collected during the interaction process; RL also allows learning in a model-free manner, thus avoiding complex modeling work. Usually, RL is applied to real-world scenarios after simulation training, which can save time and the cost of related equipment, and in some specific applications, it can also improve the safety of experiments.
[0065] Although reinforcement learning has the above significant advantages over traditional control strategies, there are difficulties in training in actual application. This is mainly reflected in the sensitivity of most algorithms to their own parameters, reward calculation methods, random number seeds, and observation space settings, which leads to significant differences in training results. This makes the reproduction of related work more uncertain and difficult to achieve the expected results, which seriously hinders the further application of reinforcement learning technology.
[0066] The theoretical framework of UAV attitude control based on reinforcement learning is as follows Figure 3 As shown in the figure. First, by defining the Markov decision process (MDP) through the design of the state space, action space and reward function, a reinforcement learning model for posture control is established. Secondly, the reward function is further analyzed and designed in a specific and multi-faceted manner. The algorithm used in this experiment is introduced to lay the foundation for the specific simulation implementation later.
[0067] In the attitude control of quad-rotor drones, by mapping the attitude angular velocity tracking error to the state space in the Markov decision process, various reward functions are designed, and the energy proportion of the motor control signal is simplified to the interval [0,1] as the representation of the action space, so as to establish a reinforcement learning model for attitude control. At the same time, a simulation platform is built based on the simulation flight framework GymFC. In the simulation experiment part, the proximal policy optimization algorithm (PPO) is applied to discuss and compare the effects of various reward settings and related parameters on training and attitude control effects, and determine the reward calculation method with outstanding performance.
[0068] A quadcopter attitude control method is designed based on the reinforcement learning method. First, the Markov decision process of the problem is defined by designing the state space, action space and reward function of the problem, and a reinforcement learning model is established. It is expected to be implemented on the simulation platform in combination with the GymFC simulation flight framework. Secondly, the proximal policy optimization algorithm (PPO) is used to discuss the effects of various reward methods and related parameters, such as normalization factor and learning rate, on training and control results. It is expected that through the analysis and comparison of simulation training and evaluation results, a reward function with outstanding performance can be determined. Specific implementation method three:
[0070] Different from the second specific implementation mode, in the data-driven quadrotor drone flight attitude control method of this implementation mode, the steps of constructing the LSTM neural network prediction model are as follows:
[0071] The LSTM-based posture estimation framework includes an input layer, a hidden layer, and an output layer, such as Figure 4 As shown, Figure 4 In: Adam (adaptive moment estimation) is an adaptive momentum estimator;
[0072] The input layer is used to reconstruct the input and output features of the time series, divide the training set and the test set, and normalize the data set to map the data to the range of 0 to 1;
[0073] The hidden layer contains several LSTM units, which are mainly used to learn the special mapping relationship between input features and output features; this layer is reasonably adjusted according to the training effect;
[0074] The output layer is used to output the final result. The output value is obtained according to the forward propagation calculation method, and the value is back-propagated into the training network to update the weights and the updated weights are sent to the hidden layer to update the model. It is iterated in sequence until the weights converge. Finally, the prediction results are denormalized to obtain the final posture estimation result. Specific implementation method four:
[0076] Different from the third specific implementation mode, in the data-driven quadrotor drone flight attitude control method of this implementation mode, the steps of determining the loss function and the evaluation index are specifically as follows:
[0077] The loss function is used to represent the cumulative error between the estimated value and the true value. In order to prevent outliers, smooth-L1-loss is selected as the loss function. This loss function is less likely to have outliers than the mean square error. The calculation formula is: Where: n is the sequence length; is the true value yi and the predicted value The difference between
[0078] The optimizer uses the widely used adaptive momentum estimator (Adam); the data is repeatedly iterated through the above units and the weights are updated to reduce the loss function, so that the loss function reaches the ultimate goal of minimizing the loss function;
[0079] The evaluation index of attitude estimation output is root mean square error (RMSE) S RMSE , S RMSE Sensitive to outliers, the fewer unreasonable points in the posture estimation output, the closer the result is to the true value, then S RMSE The smaller the value, the RMSE The calculation formula is:
[0080] Specific implementation method five:
[0082] Different from the fourth specific implementation mode, the present implementation mode is a data-driven quadrotor drone flight attitude control method, and the Markov decision process is specifically as follows:
[0083] Markov Decision Process (MDP) Figure 5 As shown in the figure, it is a classic expression of the sequential decision-making process; the action performed at a certain moment not only affects the immediate reward and the next state, but also affects the future benefits and states, thereby generating delayed benefits; the first-order Markov property of the decision-making process shows that the current state and action are only affected by the state and action of the previous moment, and have nothing to do with other moments before; the second-order to multi-order Markov properties are deduced in the same way. This is easy to understand and in line with reality. For example, a person's mood is greatly affected by yesterday, but has little to do with the mood of the day before yesterday or even longer ago.
[0084] MDP is an ideal framework for studying the interaction process in RL. Figure 2 The agent (Agent) at time t according to the current state st and Policy execution action a t , the environment is affected by a t The impact of returns a numerical reward (Reward) representing its performance, and is represented by the state s t Convert to s t+1 ; The purpose of continuous interaction is to enable the agent to learn the strategy that maximizes the cumulative reward;
[0085] Due to the delayed nature of the benefits, Gt is defined as a specific function for obtaining the reward sequence after time t, which represents the expected benefit. The discount factor is introduced in the form of G t =r t +γr t+1 +γ 2 r t+2 +...+γ T-1 r T-1 +γ T r T ,The discount factor indicates that the impact of the action performed at time t on the future will gradually decrease over time;
[0086] In the form of the discount factor, T is the final time, r t Represents the reward value obtained at time t; if the interaction task between the agent and the environment is continuous, then T is ∞, and this task is called a continuous task; otherwise T is a finite value, and the interaction process is divided into a series of subsequences containing terminal states, such as playing a chess game or a game. Tasks with this feature are called episode tasks, and the subsequences are called episodes (Episode, or rounds in this article). Specific implementation method six:
[0088] Different from the fifth specific implementation, the present implementation is a data-driven quadcopter flight attitude control method, wherein the Markov decision process of the problem is defined by designing the state space of the problem, specifically:
[0089] The state space is the source of the agent's perception of external information in the reinforcement learning interaction framework. The information that can be obtained and used by the agent is measured by the sensor. For the entire external environment, this is a partially perceived process, so the acquired state is equivalent to a subset of the complete space. In the attitude control process, in order to achieve roll θ, pitch As well as the tracking control of the three angular velocities of yaw ψ, the state space includes three angular velocity errors, namely:
[0090]
[0091] In addition, if there is a delay between the action and the state, the historical state value needs to be used as part of the neural network input. Considering the delay caused by communication during the simulation implementation, the state space is set to consist of the current angular velocity error and the error at the previous moment:
[0092] Specific implementation method seven:
[0094] Different from the sixth specific implementation, the present implementation is a data-driven method for controlling the flight attitude of a quadcopter drone. The Markov decision process of the problem is defined by designing the action space of the problem. Specifically, the action space is a set of actions that the agent can take, which corresponds to the control signal of the motor in this problem. In actual tasks, the motor speed is adjusted by adjusting the PWM output of the ESC to achieve the purpose of attitude control. To simplify the process, and because most reinforcement learning algorithms rely on Gaussian distribution (centered at zero and with a standard deviation of 1) for continuous actions, if the action space is forgotten to be normalized when using a custom environment, this may impair learning and be difficult to debug. In addition, in order to reduce the difficulty of space exploration, the action space also needs to be normalized. Therefore, the control signal is expressed as the allowed energy output ratio, and the value range is [0,1]. Thus, the action space can be expressed as:
[0095] A=<a1,a2,a3,a4>
[0096] Among them: a i ,i=1,2,3,4 represents the four motor control signals of the drone. Specific implementation eight:
[0098] Different from the seventh specific implementation mode, the present implementation mode is a data-driven quadcopter flight attitude control method. The Markov decision process of the problem is defined by designing the reward function of the problem. Specifically, the goal of reinforcement learning is to obtain the maximum cumulative reward, so the reward function largely determines the performance of the final strategy; in the attitude control task, the goal is to achieve accurate and rapid tracking and control of the angular velocity of the quadcopter; therefore, the basic idea is to use the tracking error as a measure of the size of the reward. At the same time, since the two are inversely correlated, that is, the larger the tracking error, the smaller the reward value, so its value is set to a negative value to indicate the penalty for inaccurate instruction tracking; that is:
[0099] r∞-|e|. Specific implementation method nine:
[0101] Different from the eighth specific implementation mode, the data-driven flight attitude control method of the quad-rotor unmanned aerial vehicle in this implementation mode, on the basis of establishing the attitude control problem model, needs to determine the reinforcement learning algorithm before the simulation experiment, and the method also includes the reinforcement learning algorithm, such as Figure 3 As shown in the figure, the role of the algorithm in the interactive learning process is to update the agent's strategy. Combined with the proximal policy optimization algorithm (PPO), the influence of different reward function settings on the final control effect is studied. For the problem that the training results produced by the TD3 and PPO methods are quite different and not conducive to reproduction, the SAC-based drone attitude control is proposed. The RL algorithms used are introduced one by one below, including:
[0102] First, Proximal Policy Optimization Algorithm;
[0103] For random strategies, the most widely used on-policy algorithm is the Proximal Policy Optimization Algorithm (PPO), which is developed based on the Trust Region Policy Optimization (TRPO) algorithm. In the policy gradient method, its parameter σ is calculated according to σ t+1 =σ t +α·g update, when the learning step size α is not selected properly, it is easy to cause the new strategy to perform worse or even cause the training to crash; TRPO applies the idea of Trust Region Algorithm to reinforcement learning, which is more stable and converges faster than the policy gradient algorithm; as a constrained optimization problem, the solution process of TRPO involves a very large amount of calculation; PPO improves on its basis, simplifies the amount of calculation while limiting the update range, and its optimization objectives have two forms, penalty form and clip form; currently, the more commonly used is the clip form;
[0104] Second, Twin Delayed Deep Deterministic Policy Gradient algorithm;
[0105] In the deterministic policy method, the optimization of the policy is basically achieved through Q-Learning, that is, by maximizing the value of the action value function. In this process, the experience data is generated by the previous old policy, so it is an Off-Policy algorithm;
[0106] Deep Deterministic Policy Gradient (DDPG) is a deterministic policy algorithm used in the attitude control task of quadrotors, but the results are not good, mainly because it is often fragile in terms of hyperparameters and other types of tuning. A common problem is that DDPG greatly overestimates the Q value when learning the Q function, which causes the policy learned by relying on this function to perform poorly. To solve this problem, the Twin Delayed DDPG (TD3) algorithm is applied to this control task.
[0107] Third, the reinforcement learning algorithm uses the Soft Actor Critic algorithm;
[0108] As an off-policy and random strategy algorithm, Soft Actor Critic (SAC) involves increasing the entropy of actions on the basis of maximizing expected returns, that is, completing tasks with as many possible actions as possible to achieve diversity of solutions; in practical problems, such as the optimal path for a robot from point A to point B, there is more than one, but for deterministic algorithms, the learning process ends after an optimal solution is obtained. Therefore, SAC has advantages for such problems: first, the learned strategy can be used as the initialization of more complex specific tasks, because through the maximum entropy method, the strategy can usually learn multiple ways to solve the task. Therefore, such a strategy is more conducive to learning new tasks; second, SAC has stronger exploration ability and can more easily find better modes under multi-modal rewards, such as completing robot control tasks under the condition of reducing energy consumption; at the same time, SAC also has stronger robustness and stronger generalization ability. Because it is necessary to explore various optimal possibilities in different ways, it is easier to make adjustments when facing interference. It is worth mentioning that it has been observed in the work that the introduction of entropy will lead to better exploration, and compared with the traditional objective function, the new objective function can increase the learning speed.
[0109] Previously, there have been related works that combined the maximum entropy concept with On-Policy (such as PPO, which uses the entropy value of the action as part of the Actor loss when implemented) or Off-Policy (such as Soft Q-Learning, SQL), but the problem of low sample utilization in On-Policy has not been solved, and the Off-Policy version requires the use of complex approximate inference in the continuous action space, such as the SQL algorithm needs to use Stein Variational Gradient. SAC has the advantages of both high sample utilization and stability.
[0110] The embodiments of the present invention disclose preferred embodiments, but are not limited thereto. A person skilled in the art can easily understand the spirit of the present invention based on the above embodiments and make different extensions and changes. However, as long as they do not deviate from the spirit of the present invention, they are all within the protection scope of the present invention.
Claims
1. A data-driven quadrotor drone flight attitude control method, characterized in that: The method includes designing an LSTM neural network and constructing a data set for attitude estimation. The overall structure of the LSTM neural network is as follows: the attitude angle is divided into a horizontal attitude angle and a yaw angle, and an LSTM neural network is used to take a gyroscope output, an accelerometer output, and the horizontal attitude angles at the previous n moments as inputs to solve the horizontal attitude angle at the current moment, and another LSTM neural network is used to take a gyroscope output, a magnetometer output, and the yaw angles at the previous n moments as inputs to solve the yaw angle at the current moment, thereby finally realizing full attitude estimation; The data in the data set are MEMS and inertial measurement system sensor data installed on the drone, and drone attitude information output by the Vicon system; the sensor data include gyroscope output, accelerometer output and magnetometer output.
2. The data-driven quadrotor drone flight attitude control method according to claim 1, characterized in that: The process of designing the LSTM neural network includes: First, the steps to build the LSTM neural network prediction model; Second, the steps to determine the loss function and evaluation index; Third, a quadcopter attitude control method is designed based on the reinforcement learning method; first, the Markov decision process of the problem is defined by designing the state space, action space and reward function of the problem, and a reinforcement learning model is established. It is expected to be implemented on the simulation platform in combination with the GymFC simulation flight framework; secondly, the proximal strategy optimization algorithm is used to determine the reward function through analysis and comparison of simulation training and evaluation results.
3. The data-driven quad-rotor UAV flight attitude control method according to claim 2, characterized in that: The steps of constructing the LSTM neural network prediction model are specifically as follows: The LSTM-based posture estimation framework includes an input layer, a hidden layer, and an output layer. The input layer is used to reconstruct the input and output features of the time series and divide the training set and the test set. At the same time, the data set is normalized and mapped to the range of 0 to 1. The hidden layer contains several LSTM units, which are mainly used to learn the special mapping relationship between input features and output features; reasonable adjustments are made according to the training results; The output layer is used to output the final result. The output value is obtained according to the forward propagation calculation method, and the value is back-propagated into the training network to update the weights and the updated weights are sent to the hidden layer to update the model. It is iterated in sequence until the weights converge. Finally, the prediction results are denormalized to obtain the final posture estimation result.
4. The data-driven quad-rotor UAV flight attitude control method according to claim 3 is characterized in that: The steps of determining the loss function and the evaluation index are specifically as follows: The loss function is used to represent the cumulative error between the estimated value and the true value. Smooth-L1-loss is selected as the loss function, and the calculation formula is: Where: n is the sequence length; is the true value yi and the predicted value The difference between The optimizer uses an adaptive momentum estimator; the data is repeatedly iterated through the above units and the weights are updated to minimize the loss function; The evaluation index of attitude estimation output is the root mean square error S RMSE , S RMSE Sensitive to outliers, the fewer unreasonable points in the posture estimation output, the closer the result is to the true value, then S RMSE The smaller the value, the RMSE The calculation formula is:
5. The data-driven quad-rotor UAV flight attitude control method according to claim 4, characterized in that: The Markov decision process is specifically: Markov decision process is a classic expression of sequential decision process; the action performed at a certain moment not only affects the immediate reward and the next state, but also affects the future benefits and states, thus generating delayed benefits; the current state and action of the decision process are only affected by the state and action of the previous moment, and have nothing to do with other moments before; the second-order to multi-order Markov properties are deduced by analogy; MDP is an ideal framework for studying the interaction process in RL. The agent moves at time t according to the current state s. t and policy execution action a t , the environment is affected by a t The impact of returns a numerical reward (Reward) representing its performance, and is represented by the state s t Convert to s t+1 ; Continuous interaction enables the agent to learn the strategy that maximizes the cumulative reward; Define Gt as a specific function to obtain the reward sequence after time t, which represents the expected return, and introduces the form of discount factor expressed as G t =r t +γr t+1 +γ 2 r t+2 +...+γ T-1 r T-1 +γ T r T ,The discount factor indicates that the impact of the action performed at time t on the future will gradually decrease over time; In the form of the discount factor, T is the final time, r t Represents the reward value obtained at time t; if the interaction task between the agent and the environment is continuous, then T is ∞, and this task is called a continuous task; otherwise T is a finite value, and the interaction process is divided into a series of subsequences containing terminal states. Tasks with this characteristic are called episodic tasks, and the subsequences are called episodes.
6. The data-driven quad-rotor UAV flight attitude control method according to claim 5, characterized in that: The Markov decision process of defining the problem by designing the state space of the problem is specifically: The state space is the source of the agent's perception of external information in the reinforcement learning interaction framework. The information that can be obtained and used by the agent is measured by the sensor. For the entire external environment, this is a partially perceived process, so the acquired state is equivalent to a subset of the complete space. In the attitude control process, in order to achieve roll θ, pitch As well as the tracking control of the three angular velocities of yaw ψ, the state space includes three angular velocity errors, namely: In addition, if there is a time delay between the action and the state, the historical state value needs to be used as part of the neural network input, and the state space is set to consist of the current angular velocity error and the error at the previous moment:
7. The data-driven quad-rotor UAV flight attitude control method according to claim 6, characterized in that: The Markov decision process of defining the problem by designing the action space of the problem is as follows: the action space is the set of actions that the agent can take, corresponding to the control signal of the motor; in the actual task, the motor speed is adjusted by adjusting the PWM output of the electric speed controller, thereby achieving the purpose of posture control; the control signal is expressed as the allowed energy output ratio, and the value range is [0,1]; thus, the action space can be expressed as: A=<a1,a2,a3,a4> Among them: a i ,i=1,2,3,4 represents the four motor control signals of the drone.
8. The data-driven quad-rotor UAV flight attitude control method according to claim 7, characterized in that: The Markov decision process of the problem is defined by designing the reward function of the problem. Specifically, the reward function determines the performance of the final strategy. In the posture control task, the tracking error is used as a measure of the size of the reward. The two are inversely correlated, that is, the larger the tracking error, the smaller the reward value. Therefore, its value is set to a negative value to indicate the penalty for inaccurate instruction tracking; that is: r∞-|e|。 9. The data-driven quadrotor drone flight attitude control method according to claim 8, characterized in that: The method also includes a reinforcement learning algorithm, and the reinforcement learning algorithm uses a Soft Actor Critic algorithm; As an off-policy and random strategy algorithm, Soft Actor Critic involves increasing the entropy of actions on the basis of maximizing expected returns, that is, completing the task with as many possible actions as possible to achieve diversity of solutions.
Citation Information
Cited By
Unmanned aerial vehicle freedom degree attitude test analysis system and method
CN121005102A
Multi-rotor trajectory tracking optimization method based on MLP acceleration estimation and reinforcement learning
CN121028574A
