Four-rotor unmanned aerial vehicle hanging anti-swing control method based on PPO-Clip

Abstract

The invention belongs to the field of deep reinforcement learning and automatic control, and particularly relates to a four-rotor unmanned aerial vehicle hanging anti-swing control method based on PPO-Clip. The method comprises the following steps: 1) establishing a mathematical model of a suspended unmanned aerial vehicle system based on Newton-Euler and Euler-Lagrange methods; 2) according to the system model, designing a Backstepping-PID controller to carry out pose control on the unmanned aerial vehicle; and 3) constructing a deep reinforcement learning framework based on a near-end strategy optimization algorithm (PPO-Clip), wherein the deep reinforcement learning framework comprises the processes of sampling of an unmanned aerial vehicle experience sequence, reward function design, intelligent agent training and the like. According to the invention, by designing the four-rotor unmanned aerial vehicle suspension anti-swing control system based on the PPO-Clip algorithm, a reasonable reward function is designed to guide the unmanned aerial vehicle to adjust the position and attitude of the unmanned aerial vehicle, so that the suspension unmanned aerial vehicle can track an expected trajectory with high precision under the condition of ensuring that the load stops or converges in a short time; therefore, the trajectory tracking performance, the load anti-swing performance, the anti-interference performance and the like of the four-rotor hanging unmanned aerial vehicle are improved.

Description

Technical Field

[0001] The present invention relates to the fields of reinforcement learning and automatic control, and in particular to a method for controlling the flight of a suspended unmanned aerial vehicle and anti-swaying control of a load. Background Art

[0002] Rotor-mounted UAVs are widely used in logistics, agricultural plant protection, disaster relief and other fields due to their flexibility, low risk and freedom from geographical restrictions. However, the dynamic coupling characteristics between the UAV and the mounted payload increase the control difficulty of the system, which not only leads to a decrease in the position control accuracy of the UAV, but also its load swing will seriously threaten the flight safety of the UAV. Therefore, how to improve the stability and safety of the mounted UAV system is one of the important research directions at home and abroad.

[0003] Most of the traditional anti-sway control methods currently used in suspended UAV flight rely on precise mathematical model parameters. When facing different tracking targets or being disturbed in complex flight environments, the payload will swing greatly, seriously affecting the trajectory tracking performance and stability of the UAV. At this time, traditional control methods may need to frequently adjust parameters, resulting in a decrease in control performance. Reinforcement learning is widely used in the field of UAV control because it can achieve efficient and accurate autonomous decision-making and control. However, most traditional reinforcement learning algorithms are on-policy, which is not efficient in sample utilization. There may also be large-scale policy updates during training, resulting in reduced UAV stability. How to use reinforcement learning to enable suspended UAVs to quickly adapt to tracking targets and complex environments while suppressing payload swing control methods is rarely reported. Summary of the invention

[0004] The purpose of the present invention is to provide a method for controlling the anti-sway of a hanging quad-rotor UAV. The actor-critic neural network is trained by the PPO-Clip algorithm, and the updating amplitude of the reinforcement learning strategy is limited by the clipping function to avoid large fluctuations in the strategy during the training process, so that the UAV can adjust its position more smoothly. Finally, the control action output by the optimization strategy network is used to correct the current position and posture of the UAV. This method can effectively improve the stability of the hanging UAV and suppress the swing of the hanging load.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A PPO-Clip-based quadcopter UAV suspension anti-sway control technology is characterized by designing a Backstepping-PID controller to adjust the position and attitude of the UAV to provide stable training conditions for reinforcement learning. A suspension UAV control framework based on the PPO-Clip algorithm is designed, the probability ratio of the new and old strategies is introduced to measure the strategy change, and the clipping function is used to limit the amplitude of the strategy update. A quadcopter UAV suspension load anti-sway control system based on PPO-Clip is constructed, including the following steps:

[0007] S1: The position and attitude of the UAV are described by establishing an inertial coordinate system and a body coordinate system. On this basis, the influence of the rope mass and air resistance on the UAV hanging system is ignored, and a mathematical model of the quadrotor UAV hanging system is established based on Newton-Euler and Euler-Lagrange.

[0008] S2: According to the nonlinear, under-actuated and strongly coupled characteristics of quadrotor drones, the outer loop position control adopts the Backstepping controller and the inner loop attitude control adopts the PID controller.

[0009] S3: Use the PPO-Clip algorithm to train the neural network and continuously update the optimization strategy network parameters θ and value network parameters φ, and correct the input trajectory of the hanging drone according to the optimized strategy.

[0010] S4: Set the desired position and desired yaw angle of the UAV, and apply interference to the payload during the flight to verify the anti-interference ability of the method.

[0011] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0012] After the algorithm trajectory correction, the UAV tracking trajectory is smoother and the tracking speed is faster. It can quickly suppress the load swing during the UAV take-off phase, or make the load quickly converge to the spherical pendulum state when tracking an arc trajectory. When the load is disturbed, the UAV can adjust its position autonomously to reduce the load swing frequency and the impact of the load swing on the stability of the UAV. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0014] Figure 1 This is the block diagram of the UAV suspension anti-sway control system based on the PPO-Clip algorithm;

[0015] Figure 2 This is the control block diagram of the suspended drone based on reinforcement learning PPO-Clip;

[0016] Figure 3It is a training reward comparison chart;

[0017] Figure 4 It is the fixed hovering drone position;

[0018] Figure 5 is the fixed-point hovering load position;

[0019] Figure 6 It is the change of load swing angle before and after the algorithm correction;

[0020] Figure 7 It is the spiral three-dimensional trajectory of the drone;

[0021] Figure 8 is the position tracking error before spiral trajectory correction;

[0022] Fig. 9 is the spiral trajectory PPO position tracking error;

[0023] Fig.10 is the spiral trajectory PPO-Clip position tracking error;

[0024] Fig.11 is the swing trajectory of the spiral trajectory load relative to the UAV;

[0025] Fig.12 is the load swing angle α trajectory;

[0026] Fig.13 is the rate curve of the swing angle α;

[0027] Fig.14 is the three-dimensional trajectory of the drone;

[0028] Fig.15 is the drone location;

[0029] Fig.16 It is the UAV motor speed control diagram;

[0030] Fig.17 is the trajectory of the load swing angle α. DETAILED DESCRIPTION

[0031] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0032] Any feature disclosed in this specification (including any additional claims, abstract and drawings), unless otherwise stated, may be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0033] The method of this embodiment is: to establish a nonlinear system model of a quadrotor hanging UAV based on the Newton-Euler and Euler-Lagrange methods, to control the posture of the hanging UAV by designing a Backstepping-PID controller, to guide the UAV to make positive decisions by using the reward and punishment mechanism of reinforcement learning, and to design a quadrotor hanging UAV control system based on the PPO-Clip algorithm.

[0034] First, in order to design the flight control system of the quadrotor drone, an eight-degree-of-freedom nonlinear mathematical model of the suspended drone was established based on the Newton-Euler and Euler-Lagrange methods as shown below:

[0035]

[0036] Where M is the mass of the drone, m is the mass of the payload, g is the acceleration due to gravity, and I xx ,I yy ,I zz The drones are circling O b X b , O b Y b and O b Z b The moment of inertia, F i (i=1,2,3,4) is the lift generated by the rotor, M i (i=1,2,3,4) is the torque on the z-axis of the drone.

[0037] In designing the backstepping controller, let the drone position P = (x Q ,y Q ,z Q ) T ,speed Combined with the nonlinear mathematical model of the UAV, its acceleration is expressed by the state space equation:

[0038]

[0039] Where R = (S ψ S φ +S θ C ψ C φ ,-C ψ S φ +S θ S ψ C φ ,-C θ C ψ ) T ,

[0040]

[0041] Then the state space equation of the UAV position control system is:

[0042]

[0043] Using the recursive idea and Lyapunov's theorem, the overall position control law can be deduced as:

[0044]

[0045] make ψ d = 0, then the inner loop attitude reference value U1,θ can be obtained by reverse solution d , The inner loop PID controller outputs control quantities U2, U3, and U4 based on the reference quantity and the actual posture feedback quantity.

[0046]

[0047] Where k pφ,pθ,pψ ,k iφ,iθ,iψ and k dφ,dθ,dψ The designed UAV suspension anti-sway control system is an adjustable proportional, integral and differential gain coefficient. Figure 1 .

[0048] The state of the suspended UAV system is represented by the vector S = (p Q ,p L ) T Indicates that e x ,e y ,e z is the error between the actual position and the expected position of the UAV in the inertial coordinate system, is the speed of the drone, is the distance between the actual position of the load and the target position x in the body coordinate system Lref ,y Lref ,z Lref The error, is the speed of the load, action A = (F1', F2', F3', F4') is the lift of the four rotors required for the drone to correct its trajectory, and the transition of the system state can be expressed by the state transfer function as s t+1 =P(s t+1 ∣s t ,a t ).

[0049] The UAV experience sequence τ={s1,a1,s2,a2,…,s T ,a T}, collect state, action and reward data, record the strategy as π, and its strategy parameter as θ. Then the probability of action occurring in this sequence is p θ (τ) is:

[0050]

[0051] Where p θ (a t |s t ) means that in a given state s t Take action a t The probability value depends on the mathematical model of the UAV suspension system. In order to ensure that the UAV can effectively track the expected trajectory, the reward function r' is designed as follows:

[0052]

[0053] In the formula For the swing of the load, the reward function is designed to give rewards and penalties by constraining the position and speed of the load. That is, rewards are given when the load is at the target position and the speed is approximately zero, and penalties are given in other cases. The reward function r" is designed as follows:

[0054]

[0055] In the formula λ1, λ2 are both positive numbers to be determined, ζ L , Q is any minimum value, then the linear combination of the reward function is:

[0056] r t =h T [r'(t) r”(t)] T

[0057] Where h = [h1 h2] T , the tracking and anti-sway performance of the drone can be adjusted by adjusting the weights h1 and h2. The expected reward obtained by the agent is:

[0058]

[0059] Total Reward The expected reward gradient is:

[0060]

[0061] Where A θ (s t ,a t ) is the advantage function, and the generalized advantage estimation (GAE) is used to calculate the advantage estimate. The time difference error δ t :

[0062] δ t =r t +γV φ (s t+1 )-V φ (s t )

[0063] Where V φ (s t ) is the current state value function, which can be expressed as:

[0064]

[0065] Calculate the advantage estimate A θ (s t ,a t ):

[0066]

[0067] Where λ is the attenuation parameter used to smooth the advantage estimate, γ is the discount factor, and its value is between (0,1). The current policy data is stored in the experience playback area and output through the old policy neural network in subsequent training. The old policy is recorded as π θ , whose parameter is θ' and the action probability distribution is p θ' (τ), according to the important sampling formula:

[0068]

[0069] The available strategies π and π θ The mathematical relationship expected by the function is:

[0070]

[0071] Then the expected reward gradient is:

[0072]

[0073] According to the logarithmic gradient identity formula:

[0074]

[0075] The objective function can be inferred and the shear function can be introduced into the objective function Where ε is the clipping parameter, the objective function of PPO-Clip is:

[0076]

[0077] In order to enable the drone to make better decisions in the environment and obtain higher cumulative rewards, it is also necessary to optimize the strategy by continuously updating the strategy parameters θ and the value function parameters φ. The strategy loss gradient g θ for:

[0078]

[0079] Update the policy parameters θ using the gradient ascent method:

[0080] θ k+1 =θ k +α θ g

[0081] Where α θ is the learning rate, the value function loss L VF (φ) is:

[0082]

[0083] By minimizing the loss function L VF (φ) is used to update the network parameters of the value function. The control block diagram of the suspended UAV based on the PPO-Clip algorithm is shown in the following figure: Figure 2 shown.

[0084] In any state s, the drone can find a correction action a according to the policy network to make the system transition from the current state to the next state, and iterate the process repeatedly until the target position or the upper limit of the number of steps is reached. The action probability is expressed as:

[0085] p θ (a|s)=softmax(f(s;θ))

[0086] Where f(s; θ) is the output of the neural network model. The softmax function converts the output into an action probability distribution. The current position of the drone is corrected by changing the lift of the four rotors of the drone. The corrected trajectory of the drone is:

[0087]

[0088] Where G x ,G y ,G z is the position correction of the UAV, x Q ,y Q ,z Q It is the trajectory of the drone after being corrected by the algorithm.

[0089] In order to verify the effectiveness of the PPO-Clip algorithm on the tracking performance of the suspended UAV and the anti-sway performance of the payload, a comparative analysis was made on the tracking accuracy and anti-sway performance of the payload before and after the algorithm was modified and with the traditional PPO algorithm in fixed-point hovering, tracking spiral trajectories, and payload interference experiments.

[0090] Table 1 Model parameters

[0091] Table 1 Model parameters

[0092]

[0093] The basic parameters of the PPO-Clip algorithm are configured as follows

[0094] Table 2 PPO-Clip algorithm parameters

[0095] Table 2PPO-Clip Algorithm parameter

[0096]

[0097]

[0098] Experiment 1: Assume that the drone needs to hover at the target position, set the initial position to (0,0,0), the drone target position to (1,1,1.5), and the simulation time T = 50s. Figure 3 The reward values ​​of PPO-Clip and PPO in training converge to 3.5×10 2 However, the PPO-Clip reward oscillation is smaller, and the average reward after convergence is 13.46% higher than that of PPO, indicating that PPO-Clip is updated more smoothly than the traditional PPO strategy and the accumulated reward is higher.

[0099] Depend on Figure 4 It can be seen that before the algorithm correction, the drone has a certain amount of overshoot in the x, y, and z channels. After the algorithm trajectory correction, the overshoot and adjustment time of the system are reduced. The overshoot of PPO-Clip in the x and y channels is 0, and the adjustment time is 7.4s and 9s faster than that of PPO respectively. Figure 5 It can be obtained that the vibration amplitude of the load position corrected by the algorithm decays rapidly, and the load position x after the PPO-Clip algorithm correction L ,y L It converges to zero in about 12 seconds, with only about 8 oscillations, z L It converges to zero after about 6 seconds; Figure 6After correction by the PPO-Clip algorithm, the maximum load swing angle only reaches 11.49°, and before correction, the maximum load swing angle is 27.18°. Although PPO has only one large swing, the swing amplitude is as high as 23.11°, and there is still a small angle oscillation after 20 seconds.

[0100] Experiment 2: Assume the initial position (0,0,0), simulation time T = 100s, and the tracking trajectory of the drone is [x d =0.5sin(0.2t),y d =0.5cos(0.2t)-0.5,z d =0.1t]. Figure 7-10 It can be seen that the maximum error of the drone position reached 0.38m when the trajectory correction was not performed. After the algorithm correction, the maximum error of PPO was 0.29m, and the maximum error of PPO-Clip was 0.23m. Since the drone will continue to generate inertial force relative to the suspension point when tracking the spiral trajectory, the load will eventually converge to the spherical pendulum state, and its swing angle will be constant at a certain angle. Figure 11-13 It can be seen that after correction by the PPO-Clip algorithm, the load has a smaller swing amplitude and a lower oscillation frequency than before and after PPO correction, and the load can converge to the spherical pendulum state faster. When the algorithm is not corrected, the swing angle still has a small oscillation after 20 seconds; in terms of swing angle rate, PPO-Clip has a smaller oscillation amplitude and a faster attenuation speed than before correction and PPO, indicating that the UAV is more stable during flight.

[0101] Experiment 3: Assume the initial position (0,0,0), simulation time T = 100s, and expected trajectory [x d ,y d ,z d ]=[5(t+1),0,2], at 30s, the UAV hanging load is moved along the body coordinate system X b ,Y b Interference pulse is applied in the axial direction To simulate the external disturbance of the load.

[0102] Depend on Figure 14-15 It can be seen that after the PPO-Clip algorithm is used to correct the oscillation of the drone in the y channel, it is significantly reduced and converges at 43.32s, which is 2.32s and 4.60s faster than before the PPO and correction, respectively. Fig.16 It can be seen that the maximum motor speed difference before correction reached 140.93, PPO was 94.046, and PPO-Clip was only 66.353, which shows that after the algorithm was corrected, the drone was more stable when subjected to load interference. Fig.17It can be seen that the load swing angle after PPO-Clip algorithm correction is only 0.156 rad, and the oscillation frequency is lower, with only about 4 oscillations. The PPO is 0.183 rad, and it reaches 0.312 rad before correction, with about 7 oscillations. In summary, the anti-swing control strategy of the PPO-Clip algorithm can quickly suppress the load swing by adjusting the position of the UAV when the load is disturbed, and effectively track the desired trajectory after suppressing the load swing.

[0103] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A PPO-Clip-based anti-sway control method for a quad-rotor UAV, characterized in that: The UAV flight control system is designed based on Backstepping-PID control technology. The actor-critic neural network is trained by the PPO-Clip algorithm, and the clip function in the algorithm is used to limit the update amplitude of the reinforcement learning strategy. Finally, the lift of the four rotors of the UAV is adjusted to correct the input trajectory of the UAV to form a closed-loop flight control system, which improves the trajectory tracking performance of the UAV while suppressing the swing of the suspended payload.

2. The method for controlling the hanging anti-sway of a quad-rotor UAV based on PPO-Clip according to claim 1, characterized in that: Based on the Newton-Euler and Euler-Lagrange methods, an eight-degree-of-freedom nonlinear system model of a quadrotor hanging UAV is established, and a closed-loop control system with attitude as the inner loop and anti-sway and position as the outer loop is designed.

3. The method for controlling the hanging anti-sway of a quad-rotor UAV based on PPO-Clip according to claim 2, characterized in that: The modeling of the four-rotor hanging UAV system is as follows: S1:O I X I Y I Z I is the inertial coordinate system; O b X b Y b Z b As the body coordinate system, the 8 degrees of freedom of the suspended drone are introduced, namely (x, y, z, φ, θ, ψ, x L ,y L ), where (x, y, z) is the position of the UAV in the inertial coordinate system; (φ, θ, ψ) is the attitude angle of the UAV in the inertial coordinate system; (x L ,y L ) is the coordinate of X in the body coordinate system b Y b The projection distance of the two axes; S2: Project the inertial coordinate system onto the body coordinate system in the rotation order of zyx, and its rotation matrix is: where C θ , C φ , C ψ , S θ , S φ , S ψ are cosθ, cosφ, cosψ, sinθ, sinφ, sinψ; S3: Use the Newton-Euler method to obtain the load in the body coordinate system Z b Axis projection distance z L The relationship with the rope length L is: S4: According to Newton's second law, the tension in the rope is: S5: Because the load is connected to the center of mass of the drone through a rope, the position change of the load will not affect the attitude of the drone, so the Euler motion equation remains unchanged. The mathematical model of the quadrotor hanging drone is: Where M is the mass of the drone, m is the mass of the payload, g is the acceleration due to gravity, and I xx ,I yy ,I zz The drones are circling O b X b , O b Y b and O b Z b The moment of inertia, F i (i=1,2,3,4) is the lift generated by the rotor. M i (i=1,2,3,4) is the moment on the z-axis of the drone; S6: Use the Lagrangian method to establish the kinematic model of the load. The kinetic energy of the load is: S7: Potential energy of load: V=mg(z+z L ) S8: Load position x L and L The Lagrange equation is: S9: Load position x L ,y L The equation of motion is:

4. The method for controlling the hanging anti-sway of a quad-rotor UAV based on PPO-Clip according to claim 3 is characterized in that: Design a Backstepping-PID controller, establish a reinforcement learning framework for a hanging UAV based on the PPO-Clip algorithm, observe the state of the UAV, train the neural network and update the parameters of the policy network and the value function network, and design a closed-loop flight control system for a quadcopter hanging UAV based on the PPO-Clip algorithm. The design steps are as follows: S1: Let the drone position P = (x Q ,y Q ,z Q ) T ,speed Then the state space equation of the UAV position control system is: Where R = (S ψ S φ +S θ C ψ C φ ,-C ψ S φ +S θ S ψ C φ ,-C θ C ψ ) T , S2: Using the recursive thinking and Lyapunov theorem, the overall position control law can be derived as follows: Where e1 = P d -P, e2=V d -V, k1, k2 are undetermined parameters, let Then the inner loop attitude reference U1,θ can be obtained by reverse solution d , The inner loop PID controller outputs the control quantities U2, U3, and U4 according to the reference quantity and the actual posture feedback quantity: Where k pφ,pθ,pψ ,k iφ,iθ,iψ and k dφ,dθ,dψ are adjustable proportional, integral and differential gain coefficients; S3: The motion process of the UAV suspension system is described as an uncertain Markov decision process (MDP) with a continuous state space and a discrete action space. The MDP can be represented by a set of vectors (S, A, P, R, γ). First, a sample is established. The empirical sequence τ = {s1, a1, s2, a2,, s T ,a T }, collect the state, action and other data of the drone, record it as strategy π, and its strategy parameter is θ. Then the probability of action occurring in this sequence is p θ (τ) is: Where p θ (a t |s t ) means that in a given state s t Take action a t The probability of , whose probability value depends on the mathematical model of the UAV hanging system; S4: Design a reward function r' for the tracking performance of the drone. When the error between the actual position and the expected position of the drone is less than a minimum value, a reward is given, otherwise a penalty is given. For the swing of the payload, rewards and penalties are given by constraining the position and speed of the payload, that is, rewards are given when the payload is at the target position and the speed is approximately zero, and penalties are given in other cases. The reward function is designed as r", and the linear combination of the reward function is: r t =h T [r'(t)r”(t)] T Where h = [h1h2] T , the tracking and anti-sway performance of the UAV can be adjusted by adjusting the weights h1 and h2; S5: Use the reward function and the expected cumulative reward to solve the time difference error δ t , whose expression is: δ t =r t +γV φ (s t+1 )-V φ (s t ) From this we calculate the advantage estimate A θ (s t ,a t ), whose expression is: Where λ is the attenuation parameter; S6: The PPO-Clip algorithm uses the clipping function to control the policy update amplitude to prevent the policy update from being too large and thus causing unstable training. The clipping function is introduced into the objective function. Where ε is the clipping parameter, the objective function expression of PPO-Clip is as follows: S7: Update the policy parameter θ using the gradient ascent method. The update formula is: i k+1 =θ k +a θ g θ Where α θ is the learning rate, the value function loss L VF (φ) is By minimizing the loss function L VF (φ) is used to update the value function network parameters. In any state s, the drone can find an action a according to the policy network to make the system transition from the current state to the next state, and repeatedly iterate the above steps until it reaches the target position or the upper limit of the number of steps.

5. A PPO-Clip-based four-rotor UAV suspension anti-sway control method according to claim 4, characterized in that: The Backstepping-PID controller is used to adjust the posture of the UAV, and then the input trajectory is corrected according to the correction strategy optimized by the PPO-Clip algorithm. The steps are as follows: S1: Set the desired trajectory and desired yaw angle of the UAV, and form a closed-loop flight control system of the UAV through Backstepping-PID control; S2: Subtract the position and attitude of the drone output by the quadrotor drone suspension system from the expected value, use the deviation as the input of the controller, and use the Backstepping-PID controller to control the position and attitude of the drone; S3: The PPO-Clip algorithm establishes samples by collecting the position and speed information of the drone and the payload output by the quadrotor drone suspension system, trains the neural network through the reinforcement learning algorithm, and outputs the lift of the four rotors through the strategy network to correct the current trajectory of the drone, thereby improving the tracking performance of the suspended drone and the anti-sway performance of the payload.

Images