Unmanned aerial vehicle control method and device based on deep reinforcement learning, terminal and medium

Abstract

The invention discloses an unmanned aerial vehicle control method and device based on deep reinforcement learning, a terminal and a medium. When the system has a steady-state error and the action output by the strategy network according to the system state cannot further enable the quadrotor unmanned aerial vehicle to move to the target position, the data obtained by accumulating the deviation between the actual track and the expected track in the system state is used as an actual position error item, namely the system state after integration; when a steady-state error exists in the system and the action output by the strategy network according to the system state cannot further enable the quadrotor unmanned aerial vehicle to move to a target position, a position error item can be continuously accumulated and increased, so that the strategy network is driven to generate a larger action value, the quadrotor unmanned aerial vehicle is converged to a target value, and the control precision is improved; therefore, the flight stability of the unmanned aerial vehicle under the condition of complete failure of the execution mechanism is improved.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and in particular to a drone control method, device, terminal and medium based on deep reinforcement learning. Background Art

[0002] UAV, also known as unmanned aircraft, is an unmanned aircraft controlled by radio remote control equipment and self-contained program control devices, or operated completely or intermittently autonomously by an onboard computer. Taking the more common quad-rotor UAV as an example, this type of UAV can take off and land vertically, has excellent maneuverability, can quickly change direction in complex environments, and has a simple mechanical structure, convenient operation and maintenance, and high system reliability; it has outstanding payload capacity and can carry a large number of sensors or loads. Therefore, quad-rotor UAVs are widely used in military and civilian fields, such as aerial photography and photography, infrastructure exploration and monitoring, search and rescue and disaster response, pesticide spraying and forest management, power line inspection and communication relay, etc. in the civilian field.

[0003] In order to achieve autonomous flight and high reliability of quadrotor drones, it is crucial to accurately control the drone's position and ensure that the flight attitude responds quickly and is in a stable state. However, in actual applications, when drones perform tasks in complex environments, they are easily affected by harsh environmental factors such as heavy rain and strong winds. When an actuator fails completely, the existing fault-tolerant control method is difficult to maintain effective stability. Summary of the invention

[0004] The present application provides a drone control method, device, terminal and medium based on deep reinforcement learning, which are used to solve the technical problem that the existing fault-tolerant control mechanism is difficult to maintain effective stability when facing the situation of complete failure of the actuator.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a drone control method based on deep reinforcement learning, comprising:

[0006] Acquire real-time operation data of the drone, wherein the real-time operation data specifically includes: actual trajectory, rotation speed and movement speed;

[0007] When an actuator of the drone fails, a drone control model based on deep reinforcement learning is used, with the accumulated value of the deviation between the actual trajectory and the preset expected trajectory as the position error, and the position error, the rotation speed, the movement speed and the preset coordinate system rotation matrix as state variables, so as to update the policy parameters of the proximal policy optimization function and the action variables of each effective actuator of the drone according to the current state variables in combination with the preset proximal policy optimization function, and perform reinforcement learning on the drone control model according to the preset reward and penalty function;

[0008] The lift parameters of each rotor in the UAV are calculated according to the action variables, and then the control parameters of the UAV are obtained through calculation of the UAV dynamics model according to the lift parameters, so that the control parameters are used for controlling the UAV.

[0009] Preferably, the reward-penalty function specifically includes: a position reward sub-function, a posture reward sub-function, an angular velocity reward sub-function and a crash penalty sub-function.

[0010] Preferably, when the UAV has an actuator failure, the reward-penalty function further includes: a continuous action change reward sub-function.

[0011] Preferably, the updating of the policy parameters of the proximal policy optimization function and the action variables of each effective actuator of the drone according to the current state variables in combination with a preset proximal policy optimization function specifically includes:

[0012] According to the current state variables, combined with the importance sampling ratio calculation formula in the preset proximal strategy optimization function, the probability distribution ratio of the current strategy parameters to the previous round of strategy parameters is calculated;

[0013] According to the probability distribution ratio, combined with the objective function in the proximal policy optimization function, the policy parameters of the proximal policy optimization function and the action variables of each effective actuator of the drone are updated with the output value of the objective function.

[0014] Preferably, it also includes:

[0015] Recording cache data obtained by multiple rounds of calculations of the drone control model through a preset cache unit, wherein the cache data includes: state variables at time t, action variables at time t, reward variables at time t+1, and state variables at time t+1;

[0016] When the amount of the cached data reaches a preset threshold, the policy parameters of the proximal policy optimization function are updated in batches according to the cached data, and reinforcement learning is performed on the drone control model according to a preset reward and penalty function.

[0017] Preferably, it also includes:

[0018] When the UAV has no actuator failure, the deviation between the actual trajectory and the preset expected trajectory is used as the position error through PID feedback control, so as to calculate the expected acceleration of the UAV according to the position error;

[0019] According to the expected acceleration, in combination with a preset lift correction formula, a lift control correction amount is obtained, so as to control the lift correction amount according to the lift control correction amount.

[0020] Preferably, when the drone has no actuator failure, the reward-penalty function further includes: an energy reward sub-function.

[0021] At the same time, the second aspect of the present application provides a drone control device based on deep reinforcement learning, including:

[0022] An operation data acquisition unit, used to acquire real-time operation data of the UAV, wherein the real-time operation data specifically includes: actual trajectory, rotation speed and movement speed;

[0023] A reinforcement learning optimization unit is used for, when an actuator of the UAV fails, using a UAV control model based on deep reinforcement learning, taking the accumulated value of the deviation between the actual trajectory and the preset expected trajectory as the position error, taking the position error, the rotation speed, the movement speed and the preset coordinate system rotation matrix as state variables, updating the action variables of each effective actuator of the UAV according to the current state variables in combination with a preset proximal strategy optimization function, and performing reinforcement learning on the UAV control model according to a preset reward and penalty function;

[0024] The control parameter determination unit is used to calculate the lift parameters of each rotor in the UAV according to the action variables, and then obtain the control parameters of the UAV through calculation of the UAV dynamics model according to the lift parameters, so as to use the control parameters for controlling the UAV.

[0025] A third aspect of the present application provides a drone control terminal based on deep reinforcement learning, comprising: a memory and a processor;

[0026] The memory is used to store program codes, and the program codes correspond to a drone control method based on deep reinforcement learning as provided in the first aspect of the present application;

[0027] The processor is used for reading and executing the program code.

[0028] The fourth aspect of the present application provides a computer-readable storage medium, in which a program code is stored. When the program code is executed by a processor, a drone control method based on deep reinforcement learning as provided in the first aspect of the present application can be implemented.

[0029] It can be seen from the above technical solutions that this application has the following advantages:

[0030] The solution provided in the present application is that when the drone is in a situation where the actuator fails, because when there is a steady-state error in the system and the action output by the policy network according to the system state cannot further make the quadcopter move to the target position, the data obtained by accumulating the deviation between the actual trajectory and the expected trajectory in the system state is used as the actual position error term, that is, the integrated system state. When there is a steady-state error in the system and the action output by the policy network according to the system state cannot further make the quadcopter move to the target position, the position error term will continue to accumulate and increase, thereby driving the policy network to generate a larger action value so that the quadcopter converges to the target value and increases the control accuracy, thereby improving the flight stability of the drone when facing a complete failure of the actuator. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A flowchart of an embodiment of a drone control method based on deep reinforcement learning provided in this application.

[0033] Figure 2 This is the basic framework diagram of deep reinforcement learning.

[0034] Figure 3 Schematic diagram of the coordinate system relationship between the inertial coordinate system and the body coordinate system.

[0035] Figure 4 A logical block diagram illustrating the control training of a quadrotor drone based on a combination of proximal strategy optimization algorithm and deep reinforcement learning provided in this application.

[0036] Figure 5 This is a network structure diagram of the strategy neural network part of the quadrotor drone control model based on the combination of proximal strategy optimization algorithm and deep reinforcement learning provided in this application.

[0037] Figure 6 This is a network structure diagram of the evaluation neural network part of the quadrotor drone control model based on the combination of proximal strategy optimization algorithm and deep reinforcement learning provided in this application.

[0038] Figure 7 This is a logic block diagram of the PPO-PID position control logic in the UAV control model provided in this application.

[0039] Figure 8This is a logical block diagram of the PPO-SIFT position control logic in the drone control model provided in this application.

[0040] Fig. 9 Schematic diagram of trajectory tracking of the UAV actuator under normal and fault conditions.

[0041] Fig.10 This is a schematic diagram of algorithm tracking of the PPO-PID algorithm when the actuator is in normal condition.

[0042] Fig.11 Schematic diagram of algorithm tracking of PPO-SIFT algorithm in the case of actuator failure.

[0043] Fig.12 This is a schematic diagram of the arm length change position tracking obtained through the UAV arm length change test.

[0044] Fig.13 This is a comparison chart of the arm length change postures of the UAV obtained through the arm length change test.

[0045] Fig.14 Comparison of NDI / PPO-PID and square cumulative error for arm length change.

[0046] Fig.15 This is a comparison chart of the position tracking test under load change obtained by the UAV through the model load change test.

[0047] Fig.16 Comparison chart of NDI / PPO-SIFT square cumulative error for model load changes.

[0048] Fig.17 This is the position tracking test diagram and attitude change diagram measured when LOE occurs in an actuator of the UAV.

[0049] Fig.18 These are the position tracking test diagrams and attitude change diagrams measured when LOE occurs on the two actuators of the UAV.

[0050] Fig.19 The anti-interference performance comparison chart of PPO-PID algorithm and PPOSIFT algorithm under gust interference conditions in random directions is tested.

[0051] Fig. 20 A schematic diagram of the structure of an embodiment of a drone control device based on deep reinforcement learning provided in this application.

[0052] Fig.21 A schematic diagram of the structure of an embodiment of a drone control terminal based on deep reinforcement learning provided in this application. DETAILED DESCRIPTION

[0053] The present application provides a drone control method, device, terminal and medium based on deep reinforcement learning, which are used to solve the technical problem that the existing fault-tolerant control mechanism is difficult to maintain effective stability when facing the situation of complete failure of the actuator.

[0054] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] Quadrotors are classified into two types: X-type and cross-type. This embodiment takes the X-type drone as an example to illustrate the working principle and technical solution, and the following explanations are based on this.

[0056] First, a detailed description of an embodiment of a drone control method based on deep reinforcement learning provided by this application is as follows:

[0057] See also Figure 1 , this embodiment provides a drone control method based on deep reinforcement learning, including:

[0058] Step 101: Obtain real-time operation data of the drone;

[0059] Among them, the real-time operation data specifically includes: actual trajectory, rotation speed and movement speed;

[0060] Step 102: When an actuator of the UAV fails, a UAV control model based on deep reinforcement learning is used, and the cumulative value of the deviation between the actual trajectory and the preset expected trajectory is used as the position error. The position error, rotation speed, motion speed and the preset coordinate system rotation matrix are used as state variables. According to the current state variables and in combination with the preset proximal strategy optimization function, the action variables of each effective actuator of the UAV are updated, and reinforcement learning is performed on the UAV control model according to the preset reward and penalty function.

[0061] Step 103, calculate the lift parameters of each rotor in the UAV according to the action variables, and then obtain the control parameters of the UAV through calculation of the UAV dynamics model according to the lift parameters, so as to use the control parameters for controlling the UAV. When the deviation between the current actual trajectory of the UAV and the expected trajectory does not meet the preset accuracy condition, return to step 101, execute the process of steps 101 to 103 again, and redetermine the control parameters until the conditions are met.

[0062] It should be noted that the power unit of the quadcopter is a brushless DC motor symmetrically fixed at the end of the frame. The rotation direction of motors 1 and 3 is counterclockwise, while the rotation direction of motors 2 and 4 is clockwise. This can offset the reverse torque to avoid a spin effect.

[0063] Although a quadcopter is an underactuated system with four inputs and six outputs, it can achieve attitude changes and achieve higher-level goals such as position control by adjusting the speed of the four motors. A quadcopter has four basic motion modes, such as: (1) Vertical motion: When the four rotors provide the same lift, the quadcopter maintains a stable attitude. When the total lift is equal to the total gravity, the drone hovers. If the total lift is greater than the total gravity, the drone moves upward, otherwise it moves downward. (2) Roll motion: When the speeds of motors 1 and 2, and motors 3 and 4 are made consistent and produce a speed difference, the drone can move left and right. (3) Pitch motion: When the speeds of motors 1 and 4, and motors 2 and 3 are made consistent and produce a speed difference, the drone can move forward and backward. (4) Yaw motion: When the speeds of motors 1 and 3, and motors 2 and 4 are made consistent and produce a speed difference, the drone can yaw clockwise or counterclockwise.

[0064] In order to more conveniently describe the motion state of the UAV, this paper first establishes the following two sets of coordinate systems: (1) Inertial coordinate system: The earth coordinate system is selected , taking the starting position of the quadrotor as the origin The axis points to the initial flight direction of the aircraft. The axis is vertical to the ground and upward. Axis direction and , The axis direction follows the right-hand rule. (2) Body coordinate system: This system origin Located at the center of gravity of the quadrotor, The positive direction of the axis points to the center of the first rotor. The positive direction of the axis is perpendicular to the quadrotor plane and points outward. The positive direction of the axis , The axis direction follows the right-hand rule. The Euler angles (roll angle, pitch angle, and yaw angle) used to describe the aircraft attitude are expressed as , respectively represent the quadrotor axis, axis, The rotation angle of the axis corresponds to the rotation speed The position coordinates of the quadrotor's center of mass in the inertial coordinate system are expressed as , the corresponding movement speed is expressed as , where the relationship between the inertial coordinate system and the body coordinate system is as follows Figure 3 shown.

[0065] According to the method provided in this embodiment, it is first necessary to obtain the real-time operation data of the drone. At the same time, it is necessary to detect whether each actuator in the drone, that is, the rotor motor, is operating normally. If it is detected that an actuator fails, control is performed through a drone control model based on deep reinforcement learning. In the scheme of this embodiment, the cumulative value of the deviation between the actual trajectory and the preset expected trajectory is used as the position error, and the position error, rotation speed, movement speed and preset coordinate system rotation matrix are used as state variables. According to the current state variables, combined with the preset proximal strategy optimization function, the action variables of each effective actuator of the drone are updated, and the action variables act on the environment. Combined with the reward and penalty function, the reward variable and the state variable at the next moment are generated and fed back to the intelligent agent to complete a round of learning cycle.

[0066] Among them, the reward function plays the role of the main guiding signal for agent training in reinforcement learning. The existing reward functions of quadcopter reinforcement learning control are mostly single reward functions, which may not be able to learn all the elements of the task and fall into a local optimal solution. An excellent reward function can decompose the task hierarchy into multiple subtasks, allowing the agent to learn more flexible and adaptive strategies. This embodiment proposes a combined reward function that includes continuous rewards and sparse rewards, which can effectively reduce the ineffective exploration of the agent. The reward and penalty functions mentioned in this embodiment mainly include the following items:

[0067] 1) Position reward: In order to enable the drone to track the desired trajectory, the position reward is defined as a function related to the position error. The smaller the position error, the greater the reward, and the larger the position error, the smaller the reward. are all positive constant parameters, Indicates the position bonus when the rotor is intact. It represents the position reward in the fault-tolerant case. In order to make it easier to converge in the fault-tolerant case, a reward item is added in the z direction. It is hoped that the drone can get as close to the target point as possible in the z direction. The specific expression is as follows:

[0068] (1)

[0069] (2)

[0070] 2) Attitude reward: In order to keep the attitude of the drone stable when the rotor is intact, it is hoped that the pitch angle and roll angle will be kept near 0 as much as possible, without excessive oscillation and unstable attitude, so as to increase the stability of the drone near the target point. is a positive constant parameter, which is expressed as follows:

[0071] (3)

[0072] Angular velocity bonus: Under normal conditions of the actuator, It means that the quadcopter is expected to maintain a stable attitude near the target point, and the angular velocities of the three axes can converge to near 0. In the fault-tolerant case, Expressed as and The sum of the rolling angle and the pitch angle means that when the angular velocity of the rolling angle or the pitch angle is greater than 10rad / s, a magnitude of When the yaw rate is greater than 30 rad / s, a penalty of The penalty is to stabilize the angular velocity of each axis within a rough range without being unstable. , , These are all normal parameters, as shown below:

[0073] (4)

[0074] Crash Penalty: When a quadcopter crashes, it will receive a The penalty will encourage the drone to find a strategy that does not crash, as shown below:

[0075] (5)

[0076] 5) Energy Reward: In order to simulate the actual flight of a quadcopter drone and save more energy, It is defined as the sum of the squares of the four motor action values ​​multiplied by The coefficient can motivate the UAV to minimize energy consumption. Indicates the action values ​​of the four motors at the current moment, specifically as follows:

[0077] (6)

[0078] 6) Rewards for continuous action changes

[0079] Continuous Action Change Reward: In actual flight, if the rotor action changes frequently, it will lead to unstable attitude and increase the possibility of crash. It is defined as the sum of the squares of the differences between the motor action values ​​before and after multiplied by The coefficient is expressed as follows:

[0080] (7)

[0081] Different combined reward functions are proposed for two different scenarios: normal situation and fault-tolerant situation. Meaning is the control time interval, r total represents the total reward function under normal circumstances, r fault Represents the total reward function in the fault-tolerant case:

[0082] (8)

[0083] More specifically, the basic framework of deep reinforcement learning is as follows Figure 2 As shown in the figure, the subject of reinforcement learning is called an agent. When the agent obtains the current state information in the environment, it will make a decision based on the state and output an action. When this decision is executed, the environment will output the next state and the reward brought by the current decision. The Markov property describes that in a random process, given the past information, its future state depends only on the current state, not on the past state. Therefore, the mathematical basis of the interaction process between the agent and the environment can be modeled as a Markov decision process, usually composed of a five-tuple To represent an MDP, its meanings are: (1) state space, the set S of possible states; (2) action space, the set A of all possible actions; (3) transition probability, the probability of executing an action in the current state s Get the probability p of the next state s'; (4) reward function, perform actions under different states s The immediate reward r obtained after obtaining the next state s'; (5) Discount factor , which can make a decreasing trade-off for future rewards.

[0084] The goal of reinforcement learning is to obtain a policy function , can select an action from the action space according to the state at each moment. The strategies can be divided into two types: (1) Random strategy, input a state , output a random probability density function , and then the agent samples the probability distribution to obtain actions, where and They represent the observed values ​​of action and state at time t respectively; (2) Deterministic strategy: for the input state s, the action a output by the agent is a deterministic value:

[0085] (9)

[0086] Generally speaking, reinforcement learning generally uses random strategies, and state transfer functions are two sources of randomness in reinforcement learning. The reward for the moment is , the discounted return after time t is:

[0087] (10)

[0088] In the formula, from the state at time t Start until the end state , is the sum of rewards at all times, where is the attenuation factor, The reasons for discounting the reward at the final moment are: (1) to avoid infinite rewards in some MDPs with loops, and to ensure convergence after introducing discounts; (2) because the established model cannot accurately evaluate future reality, a discount is made for future uncertainty; (3) it is more desirable to obtain immediate rewards rather than rewards later.

[0089] Among them, the expectation of return is called the value function, which is divided into two types: action value function and the state value function .in is an evaluation of the state of the environment, reflecting the execution of the strategy under different states s The expected cumulative return is:

[0090] (11)

[0091] Where E[.] represents the expectation. The Bellman expectation equation can be used to describe the recursive relationship between state values. Consider the agent in state Take action to move to the next state , can be obtained The recursive form is as follows:

[0092] (12)

[0093] It means that at time t, the state is s and the strategy is hour The average value of .

[0094] Action-Value Function Indicates that the policy is followed in state s The expected cumulative reward that can be obtained by executing the action is similar to The recursive form is as follows:

[0095] (13)

[0096] Indicates that at time t, the state is s and the strategy is When taking action a, the discounted return The average value of .

[0097] Optimal Action-Value Function With the optimal strategy Directly related, we can exclude the influence of strategy and directly evaluate the quality of current actions and status:

[0098] (14)

[0099] Optimal strategy for:

[0100] (15)

[0101] According to Bellman's optimal equation, the optimal strategy can be obtained The state value of must be equal to the expected reward of performing the best action in that state:

[0102] (16)

[0103] Using Neural Networks to Approximate Policy Functions in Deep Reinforcement Learning ,Strategic Network, , represents the parameters of the neural network. The better, The larger the The mean should also be larger, so there is a strategy objective function:

[0104] (17)

[0105] Indicates that the strategy is , given the neural network parameters are The mean state value at .

[0106] Therefore, strategy learning can be described as We hope to get a better strategy by optimizing the network parameters. We can use gradient ascent to solve the maximization problem:

[0107] (18)

[0108] The current network parameters are , the updated network parameters are is the learning rate. The policy gradient is as follows:

[0109] (19)

[0110] The policy gradient theorem can be expressed as:

[0111] (20)

[0112] The value function and policy gradient correspond to value learning and policy learning respectively. The former mainly uses the temporal difference algorithm for learning, which has the advantages of real-time update, fast learning speed, and low variance compared to the Monte Carlo algorithm. The core idea of ​​the gradient algorithm is to update the estimate of the value function by observing the difference between the current reward and the expected future reward of the experience sample, including multiple variants such as the Sarsa algorithm with the same strategy and the Q-Learning algorithm with different strategies.

[0113] For a single data, define the loss function of the following value function:

[0114] (twenty one)

[0115] The gradient of the value function is:

[0116] (twenty two)

[0117] The basic TD algorithm update rule can be expressed as:

[0118] (twenty three)

[0119] The policy gradient algorithm directly uses the neural network to approximate the policy and iteratively calculates the parameterized network. Compared with the value function method, it can handle high-dimensional and continuous action spaces better, is less susceptible to local optimal solutions, and has better convergence. The simplified training process of the policy gradient algorithm is to first use the policy network to control the agent to sample trajectories, then calculate the cumulative reward, and then calculate the stochastic gradient and parameter update. The parameter update rule can be expressed as follows according to (10), (18), (19), (20):

[0120] (twenty four)

[0121] Regarding the proximal policy optimization algorithm, i.e., the PPO algorithm, mentioned in step 102 of this embodiment for optimizing the drone control model, since the policy gradient algorithm is a same-policy algorithm, it will cause the previous sampled data to be inaccurate after updating the network parameters, and it will take a lot of time to resample. In order to circumvent the above problem, the proximal policy optimization algorithm introduces the concept of importance sampling. After applying importance sampling, the actual samples used to estimate the expected value under a new distribution come from another distribution, which can greatly improve the learning efficiency.

[0122] Assume that the new policy distribution is , the old strategy distribution is , the policy gradient after importance sampling is: , the corresponding objective function is:

[0123] (25)

[0124] More specifically, the PPO algorithm used in this embodiment mainly includes two variants, proximal policy optimization penalty and proximal policy optimization clipping. An adaptive KL divergence is introduced in the PPO-penalty algorithm, making it an unconstrained optimization problem. The coefficients before the KL divergence are continuously updated during the iteration process. The KL divergence is a measure used to measure the difference between two probability distributions. Assume that there are probability mass functions P(x) and Q(x) of two probability distributions, and the KL divergence D KL (P||Q) represents the information difference introduced when using the probability distribution Q(x) to approximate the true distribution. The objective function of the PPO-penalty algorithm is as follows:

[0125] (26)

[0126] make , The update rule is as follows: If ,So ;if ,So ;otherwise .in, It is a hyperparameter set in advance to limit the gap between the learning strategy and the previous round of strategy.

[0127] A clipping term is introduced in the PPO-clip algorithm to constrain the update amplitude of the policy to prevent the estimated variance from being too large, causing the policy network to produce erroneous results. The objective function of the PPO-clip algorithm is as follows:

[0128] (27)

[0129] in is a hyperparameter. The min function selects the smaller value between the first and second items. The clip function outputs , if the first term is greater than the third term, then output , and the rest of the time remains unchanged. By clipping, the ratio Update in the desired direction. When the advantage function is positive, the expected ratio is close to , on the contrary close to , and the proximal policy optimization algorithm used in this embodiment is preferably PPO-clip. PPO-clip is shown in Table 1:

[0130]

[0131] Since external disturbances and noise are difficult to model and do not conform to the Markov characteristics, they are not taken into account during the reinforcement learning training process. Instead, the quadcopter is modeled as a simple rigid body model that is only subject to gravity and rotor thrust. During the training process, the current policy network is first used. Interact with the environment to collect a batch of sample trajectories and store them in the replay buffer. When the minimum update data volume is reached, the probability distribution ratio of the new and old strategies is calculated and the advantage value function of each state in the sample is estimated using the evaluation network. Then the current strategy network parameters are copied to , respectively using the advantage function and the importance sampling ratio to calculate the objective function of the policy network and the batch advantage function to calculate the loss function of the evaluation network and update it using the gradient descent algorithm and The pseudo code of the training algorithm is shown in Table 2, and the schematic diagram is shown in Figure 4 shown.

[0132]

[0133] Since neural networks have good generalization performance and can be applied to a variety of purposes, this embodiment uses the same structure in both the strategy network and the evaluation network, as shown in the figure. Figure 5 and Figure 6 In order to ensure the control performance and fast training speed of the intelligent agent on the quadcopter, the neural network in this embodiment uses two fully connected hidden layers, each with 64 neurons and uses the hyperbolic tangent function tanh as the activation function. The input of both neural networks is 18-dimensional state variables The output of the evaluation network is a one-dimensional state value function estimate used to evaluate the quality of the policy network's execution of actions under the input state. ,The output of the policy network is distributed through Gaussian The four motor action values ​​sampled The actual lift of each rotor needs to be normalized to be , It represents the ratio of the thrust of the drone to its own weight, which is a positive constant. , the relationship is when When there is no motor lift, indicates full lift.

[0134] At the same time, since the original proximal policy optimization algorithm will introduce variance due to the randomness of samples interacting with the environment, and the imprecision of the model will introduce systematic deviations in the estimation of the true action value, it will eventually lead to inaccurate evaluation network and have a negative impact on the final control strategy. This endogenous influence is not transferred by changes in the reward function or the exploration time, so in order to eliminate the steady-state error problem in the control of the quadcopter based on the original proximal policy optimization algorithm in the previous section, this embodiment proposes two improved algorithms. In order not to increase the training burden and network convergence difficulty, this embodiment uses the native proximal policy optimization algorithm to train a position controller under normal conditions or a fault-tolerant controller under actuator failure conditions as a module and the improved module as a new controller.

[0135] Among them, regarding the control under normal conditions, this embodiment adopts the PID feedback control method. Proportional-differential-integral (PID) control is a classic feedback control method that can adjust the control output according to the state error to achieve the expected value. It is divided into three components, namely proportional term, differential term and integral term. Under normal circumstances, it can be observed that there is an obvious steady-state error in the z-axis. First, according to the position error, an expected acceleration can be obtained. ,in They are all 3 by 3 diagonal matrices, which are proportional gain, differential gain and integral gain respectively:

[0136] (28)

[0137] According to the nonlinear dynamics model, the corrected thrust in the presence of steady-state error is:

[0138] (29)

[0139] From the moment distribution, we can get , that is, the thrust of the new four rotors. The schematic diagram of the PPO-PID position controller algorithm is as follows Figure 7 shown.

[0140] like Figure 7 As shown in Figure 2, when the actuators of the UAV are normal, the desired trajectory is first given. , expected trajectory Received by the PID controller and reinforcement learning controller, the UAV will take the actual trajectory required in the state space , rotation speed ,Movement speed , rotation matrix Pass it to the reinforcement learning position controller to get the thrust of the four rotors , then the UAV will take the actual trajectory is passed to the PID controller, which obtains the corrected thrust according to (28) and (29): Output to UAV to complete the entire control process.

[0141] Regarding the control under the condition of actuator failure, it is easy to observe that the controller of the native PPO algorithm has steady-state errors on all three axes under the condition of actuator failure, so the previous improved algorithm is not very applicable. Drawing on the idea of ​​integration in PID control, this embodiment proposes a fault-tolerant control that combines PPO with state integral SIFT, that is, the position error term in the system state is converted into After accumulation, we get To replace , that is, the system state after integration . and The relationship is as follows:

[0142] (30)

[0143] When the system has a steady-state error and the action output by the policy network based on the system state cannot further move the quadcopter to the target position, It will continue to accumulate and increase, driving the strategy network to generate larger action values ​​so that the quadcopter can converge to the target value and increase the control accuracy. Figure 8 shown.

[0144] More specifically, regarding the quadcopter UAV dynamics model mentioned in this embodiment, in order to simplify the quadcopter modeling process, this paper assumes that the quadcopter UAV is a uniform and symmetrical rigid body that does not deform during flight, and ignores the gyroscopic effect of the rotor. Since the thrust generated by the rotor is proportional to the square of the motor speed, it can be obtained that the thrust generated by a single actuator is ,i=1,2,3,4, where b is the lift coefficient, is the rotor speed, and the thrust matrix is ​​expressed as The total lift of the four rotors in this system is , ,According to Newton's second law, the translation equation of the quadrotor drone is:

[0145] (31)

[0146] in, The actual trajectory The second-order differential of is the total mass of the quadrotor drone, and the gravity matrix is , .Will Expands to:

[0147] (32)

[0148] Total external torque of quadrotor drone as follows:

[0149] (33)

[0150] in are the axial moments, The rolling and pitching moments that provide lift to the rotor, is the yaw moment caused by the different rotor speeds, h and w are the distances from the motor center to O respectively. b Y b , O b X b The length of the shaft, d is the drag coefficient. vector The relationship between M and F is ,in Assign the matrix to the moment:

[0151] (34)

[0152] According to the angular momentum theorem, the relationship between the rotation equation of the quadrotor drone and the total external torque can be deduced as follows:

[0153] (35)

[0154] in, Represents cross product for:

[0155] (36)

[0156] in is the moment of inertia matrix of the quadrotor on each axis. The complete nonlinear differential equation of the quadrotor UAV can be derived from formulas (31) and (35) as follows:

[0157] (37)

[0158] In order to verify the effectiveness of the improved algorithm and compare the control performance improvement of the native reinforcement learning algorithm, the PPO-PID algorithm was tested to track the fixed point from the origin. T and PPO-SIFT algorithm track the fixed point [0,1,2]T from the origin, Fig. 9They are three-dimensional schematic diagrams of tracking fixed points when the actuator is normal and when the actuator fails. The algorithm provided in this application is compared with the existing nonlinear dynamic inversion (NDI) control algorithm.

[0159] 1) PPO-PID actuator position tracking test under normal conditions: The algorithm tracking diagram of the PPO-PID algorithm under normal conditions of the actuator is as follows: Fig.10 As shown, it can be found that both the PPO-PID algorithm and the NDI algorithm can converge to the target point, but the native PPO algorithm has a steady-state error on the z-axis. After the improvement of the algorithm by the present invention, the overshoot of PPO-PID is smaller than that of the native PPO algorithm and the NDI algorithm, and it has better dynamic performance. In the position tracking effect, it can be compared that the overshoot of the PPO-PID algorithm, the PPO algorithm, and the NDI algorithm on the axis are 0.1m, 0.2m, and 0.4m, respectively, and the adjustment time on the z-axis is 2.2s, 3.2s, and 4.0s, respectively. In the attitude effect diagram, it can be found that the PPO-PID algorithm can adjust to a stable state faster than the NDI algorithm in the pitch channel and the yaw channel, and the jitter is smoother. Since reinforcement learning does not limit the yaw angle, it can still be found that it will stabilize near 2rad, while NDI can be adjusted to any angle by the PID algorithm, here is 0rad. To sum up, the PPO-PID algorithm can stabilize at a given target point in a relatively short time without any steady-state error. The dynamic performance also verifies the effectiveness of the improved algorithm.

[0160] 2) PPO-SIFT actuator fault position tracking test: The algorithm tracking diagram of the PPO-SIFT algorithm in the case of actuator failure is as follows Fig.11 As shown in the figure, it can be found that both the PPO-SIFT algorithm and the NDI algorithm can converge to the target point, but the native PPO algorithm has a steady-state error on the z axis and fluctuations in the x and y directions. Although from the comparison of attitude effects, it can be found that the NDI algorithm can stabilize the roll angular velocity and pitch angular velocity of the quadrotor near a certain value, while the roll angular velocity and pitch angular velocity of the PPO-SIFT and native algorithms have large fluctuations. However, from the analysis of the position tracking effect, after the improvement of the algorithm, the overshoot of PPO-SIFT on the x-axis is smaller than that of the native algorithm, and it has better dynamic performance on the z-axis. In summary, PPO-SIFT can stabilize the quadrotor at the target point and significantly improve the dynamic performance of the native algorithm, verifying the effectiveness of the improved algorithm.

[0161] 3) Robustness test: In order to compare the robustness of the PPO-PID and PPO-SIFT algorithms with the NDI algorithm, the following three experiments were conducted: the first was a model size change test comparing the PPO-PID algorithm with the NDI algorithm under normal actuator conditions; the second was a model load change test comparing the PPO-SIFT algorithm with the NDI algorithm under actuator failure conditions; and the third was a test of the anti-interference performance of partial rotor failure and wind disturbance. The test results are as follows:

[0162] 1. Test of arm length change:

[0163] Quadcopter UAV motor to shaft , The distance is 0.0325m. In this embodiment, five equal parts are selected in the range of 0.03-0.11m as different motor-to-axis distances to test the PPO-PID algorithm for arm length changes. During the model test, the model load and the motor gravity thrust ratio do not change. Since the motor mass is relatively light, the change in the moment of inertia is ignored, and only the motor arm length is changed. The target point tracked by the PPO-PID algorithm is [0,0,2] T , while the target point tracked by the NDI algorithm is [0,1,2] T .

[0164] The location diagram is as follows Fig.12 As shown in the figure, the posture comparison is as follows Fig.13 As shown. In the position diagram, it can be observed that NDI can maintain good control performance in the first three tests, but in the fourth experiment, the drone showed obvious oscillation near the y-axis of the target point, and in the fifth experiment, it can be observed that the drone was completely out of control around 3.5s, with obvious fluctuations on the x and y axes, and the z axis showed that the drone crashed. According to the position response curve of PPO-PID, it can be observed that the neural network controller can stably cope with changes in model size and can show good control performance at all arm lengths. In the attitude comparison diagram, it can be seen that in some cases in the first three experiments, the attitude control performance of NDI is better than that of PPO-PID, but the fourth experiment shows that the jitter is on the verge of divergence compared to the PPO-PID algorithm, and the fifth experiment diverges.

[0165] The schematic diagram of the square cumulative error of the model size is as follows Fig.14As shown in the figure, the meaning of the square cumulative error is the integration of the Euclidean distance between the current position of the drone and the target point under all time steps. Since the two algorithms track different target points, the square cumulative error can more intuitively reflect whether the control effect is more accurate and faster. The corresponding information can also be obtained through the cumulative error, that is, the first three times can get a stable control effect, but the fourth time there is oscillation, and the fifth time is directly divergent. The PPO-PID algorithm performed well in the five tests, and the cumulative error of the first four tests was also smaller than that of NDI. Although the PID control algorithm used in the NDI controller is a control algorithm that adjusts parameters through trial and error, due to its inverse dynamics characteristics, it requires a higher model measurement accuracy. When the deviation is large, the torque distribution will result in a completely wrong control amount, while the neural network controller shows a higher stability.

[0166] 2. Model load change test: Install the load at the center of gravity of the machine body, and change the load by 0%-40%. Only the impact on the mass is considered, while ignoring the changes in other parameters. The target point tracked by the PPO-SIFT algorithm is [0,0,2] T , and the target point tracked by the NDI algorithm is [0, 1, 2] T , where the position tracking diagram of PPO-SIFT and NDI under load changes is shown in Fig.15 As shown, the schematic diagram of square cumulative error is as follows Fig.16 As shown in the figure. From the tracking diagram, we can see that the NDI algorithm can stably track the target point in the first three experiments, but in the fourth experiment, it quickly fell after reaching the target point, and crashed directly in the fifth experiment. Although the PPO-SIFT algorithm overshoots in lateral tracking in the first three experiments, it generally shows good tracking performance in all five experiments.

[0167] From the cumulative error diagram, it can be found that the NDI algorithm showed a downward trend in the first three experiments and the PPO-SIFT algorithm showed a downward trend in the five experiments. This is because when the load gradually increases, the thrust of the rotor will be increasingly used to bear the additional gravity, and the remaining thrust that can be used to adjust the flight attitude will gradually decrease, which will reduce the overshoot of the drone. However, in the fourth and fifth experiments, the NDI algorithm could not control the drone, which led to a sharp increase in the cumulative error. The above shows that when the load increases, the control performance of the PPO-SIFT algorithm and the NDI algorithm will gradually weaken. Under certain conditions, the NDI algorithm can even show better performance, but the PPO-SIFT algorithm shows stable control performance under a wider range of mass changes.

[0168] 4) Anti-interference test: In the real environment of a quadcopter, there will be many external and internal interferences, such as wind disturbance, sensor failure, actuator failure, etc. This embodiment also tests the rotor partial failure capability of the PPO-PID algorithm by using a non-complete failure failure (loss of effect, LOE) failure model. When a LOE failure occurs, the actuator will produce a proportionally reduced thrust. The test of an actuator with LOE is as follows: Fig.17 As shown in the figure, the test of LOE of two actuators is as follows Fig.18 The anti-interference performance of the PPO-PID algorithm and the PPOSIFT algorithm under the condition of gust interference in random directions is also tested, as shown in Fig.19 As shown, the target points tracked in the above tests are all [0,1,2] T .

[0169] When the No. 3 motor fails 40% due to LOE, it can be observed that the NDI algorithm has large fluctuations on the x-axis and y-axis, and the roll angle and pitch angle are also in an unstable state. The failure of the No. 3 motor causes the asymmetric anti-torque to cause the quadrotor to rotate. However, the PPO-PID algorithm can be observed to have slight jitter in the roll angle and pitch angle under LOE. In addition, it shows good dynamic performance in position tracking and can also remain stable in the yaw angle. If the No. 3 motor and the No. 1 motor on the opposite side are simultaneously subjected to 40% LOE, it can be observed that the NDI algorithm has less shaking in position than one LOE, but the yaw angle velocity is faster because of more failures. Although the PPO-PID algorithm also has jitter on the x-axis, and the pitch angle and roll angle also have fluctuations within a small range, it can still maintain stability without steady-state error and yaw angle on the y-axis and z-axis. Compared with the sensitivity of the NDI algorithm, the dynamic performance of the PPO-PID algorithm is much stronger than that of the NDI algorithm.

[0170] When the drone is in a 3m / s wind field in a random direction, it can be observed that PPO-SIFT deviates from the target point for a short period of time, and then it takes about 2s to adjust back to the target point, while the PPO-PID algorithm is basically not affected by wind disturbance and can control the drone at the target point very stably. In summary, the PPO-PID / PPO-SIFT algorithm has strong anti-interference and anti-failure capabilities, and the overall anti-interference control performance is much better than the NDI algorithm.

[0171] The above is a detailed description of an embodiment of a drone control method based on deep reinforcement learning provided by the present application. The following is a detailed description of an embodiment of a drone control device based on deep reinforcement learning provided by the present application.

[0172] See also Fig. 20, this embodiment provides a drone control device based on deep reinforcement learning, including:

[0173] The operation data acquisition unit 201 is used to acquire the real-time operation data of the UAV, wherein the real-time operation data specifically includes: actual trajectory, rotation speed and movement speed;

[0174] The reinforcement learning optimization unit 202 is used for, when an actuator of the UAV fails, using a UAV control model based on deep reinforcement learning, taking the accumulated value of the deviation between the actual trajectory and the preset expected trajectory as the position error, taking the position error, the rotation speed, the movement speed and the preset coordinate system rotation matrix as the state variables, updating the action variables of each effective actuator of the UAV according to the current state variables in combination with the preset proximal strategy optimization function, and performing reinforcement learning on the UAV control model according to the preset reward and penalty function;

[0175] The control parameter determination unit 203 is used to calculate the lift parameters of each rotor in the UAV according to the action variables, and then obtain the control parameters of the UAV through calculation of the UAV dynamics model based on the lift parameters, so as to use the control parameters for controlling the UAV. When the deviation between the current actual trajectory of the UAV and the expected trajectory does not meet the preset accuracy condition, the optimization is performed again based on the current real-time operation data and the UAV control model.

[0176] like Fig.21 As shown, this embodiment also provides a drone control terminal based on deep reinforcement learning, the implementation types of the terminal include but are not limited to: a personal computer, a server and an embedded intelligent terminal, the main components of the terminal include: a memory 33 and a processor 31, and the memory 33 and the processor 31 can be connected via a communication bus 34;

[0177] The memory 33 is used to store program codes, and the program codes correspond to a drone control method based on deep reinforcement learning provided in the above embodiment;

[0178] The processor 31 is used to read and execute program codes.

[0179] The fourth aspect of the present application provides a computer-readable storage medium, in which program code is stored. When the program code is executed by a processor, a drone control method based on deep reinforcement learning as provided in the above embodiment can be implemented.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the terminals, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0181] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0182] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0183] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0184] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0185] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A drone control method based on deep reinforcement learning, characterized in that: include: Acquire real-time operation data of the drone, wherein the real-time operation data specifically includes: actual trajectory, rotation speed and movement speed; When an actuator of the drone fails, a drone control model based on deep reinforcement learning is used, with the accumulated value of the deviation between the actual trajectory and the preset expected trajectory as the position error, and the position error, the rotation speed, the movement speed and the preset coordinate system rotation matrix as state variables, so as to update the policy parameters of the proximal policy optimization function and the action variables of each effective actuator of the drone according to the current state variables in combination with the preset proximal policy optimization function, and perform reinforcement learning on the drone control model according to the preset reward and penalty function; The lift parameters of each rotor in the UAV are calculated according to the action variables, and then the control parameters of the UAV are obtained through calculation of the UAV dynamics model according to the lift parameters, so that the control parameters are used for controlling the UAV.

2. The method for controlling a drone based on deep reinforcement learning according to claim 1, characterized in that: The reward-penalty function specifically includes: a position reward sub-function, a posture reward sub-function, an angular velocity reward sub-function and a crash penalty sub-function.

3. The method for controlling a drone based on deep reinforcement learning according to claim 2, characterized in that: When the UAV has an actuator failure, the reward penalty function also includes: an action continuous change reward sub-function.

4. The method for controlling a drone based on deep reinforcement learning according to claim 1, characterized in that: The updating of the policy parameters of the proximal policy optimization function and the action variables of each effective actuator of the drone according to the current state variable and in combination with the preset proximal policy optimization function specifically includes: According to the current state variables, combined with the importance sampling ratio calculation formula in the preset proximal strategy optimization function, the probability distribution ratio of the current strategy parameters to the previous round of strategy parameters is calculated; According to the probability distribution ratio, combined with the objective function in the proximal policy optimization function, the policy parameters of the proximal policy optimization function and the action variables of each effective actuator of the drone are updated with the output value of the objective function.

5. The method for controlling a drone based on deep reinforcement learning according to claim 1, characterized in that: Also includes: Recording cache data obtained by multiple rounds of calculations of the drone control model through a preset cache unit, wherein the cache data includes: state variables at time t, action variables at time t, reward variables at time t+1, and state variables at time t+1; When the amount of the cached data reaches a preset threshold, the policy parameters of the proximal policy optimization function are updated in batches according to the cached data, and reinforcement learning is performed on the drone control model according to a preset reward and penalty function.

6. The method for controlling a drone based on deep reinforcement learning according to claim 2, characterized in that: Also includes: When the UAV has no actuator failure, the deviation between the actual trajectory and the preset expected trajectory is used as the position error through PID feedback control, so as to calculate the expected acceleration of the UAV according to the position error; According to the expected acceleration, in combination with a preset lift correction formula, a lift control correction amount is obtained, so as to control the lift correction amount according to the lift control correction amount.

7. The method for controlling a drone based on deep reinforcement learning according to claim 6, characterized in that: When the drone has no actuator failure, the reward penalty function also includes: an energy reward sub-function.

8. A drone control device based on deep reinforcement learning, characterized in that: include: An operation data acquisition unit, used to acquire real-time operation data of the UAV, wherein the real-time operation data specifically includes: actual trajectory, rotation speed and movement speed; A reinforcement learning optimization unit is used for, when an actuator of the UAV fails, using a UAV control model based on deep reinforcement learning, taking the accumulated value of the deviation between the actual trajectory and the preset expected trajectory as the position error, taking the position error, the rotation speed, the movement speed and the preset coordinate system rotation matrix as state variables, updating the action variables of each effective actuator of the UAV according to the current state variables in combination with a preset proximal strategy optimization function, and performing reinforcement learning on the UAV control model according to a preset reward and penalty function; The control parameter determination unit is used to calculate the lift parameters of each rotor in the UAV according to the action variables, and then obtain the control parameters of the UAV through calculation of the UAV dynamics model according to the lift parameters, so as to use the control parameters for controlling the UAV.

9. A drone control terminal based on deep reinforcement learning, characterized in that: include: Memory and processor; The memory is used to store program codes, and the program codes correspond to a drone control method based on deep reinforcement learning as described in any one of claims 1 to 7; The processor is used for reading and executing the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and when the program code is executed by a processor, a drone control method based on deep reinforcement learning as described in any one of claims 1 to 7 can be implemented.

Images