Unmanned aerial vehicle model prediction control method, system and device based on Actor-Critic and artificial potential field and medium

Abstract

The invention discloses an unmanned aerial vehicle model prediction control method, system and device based on Actor-Critic and an artificial potential field, and a medium. A network of multiple static multi-obstacle environments and dynamic multi-obstacle environments, a kinetic model of a four-rotor unmanned aerial vehicle, an award value based on the artificial potential field, and an unmanned aerial vehicle model prediction control algorithm based on Actor-Critic and the artificial potential field is constructed in sequence; training an algorithm model according to the set network parameters, and selecting an optimal weight file; inputting the optimal weight file and a network of an unmanned aerial vehicle model prediction control algorithm based on Actor-Critic and an artificial potential field into a plurality of static multi-obstacle environments and dynamic multi-obstacle environments for testing; obtaining an unmanned aerial vehicle flight path diagram, a reward value r, a path length L, a path maximum torsion degree LS, a path global torsion degree GS, and an index result of whether collision occurs or not and threat; the system, the equipment and the medium are used for implementing the method. The method can effectively plan the path and timely avoid the obstacle, and has the advantages of high real-time performance, strong robustness, timely obstacle avoidance and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle control, and in particular relates to a method, system, device and medium for predictive control of unmanned aerial vehicle models based on Actor-Critic and artificial potential fields. Background Art

[0002] With the advancement of UAV technology, UAVs are increasingly used in military, civil and commercial fields. However, traditional remote control methods or predetermined trajectory control methods are often difficult to cope with complex and changing environmental conditions, so more intelligent and autonomous control methods are needed. The combination of reinforcement learning and control algorithms for UAV flight trajectory planning and prediction tracking is one of the current development trends in the field of UAV flight control.

[0003] Reinforcement learning algorithms have made significant progress and applications in the fields of robot control, autonomous driving, aircraft control, and power system dispatch. The Actor-Critic algorithm is an important algorithm in reinforcement learning, which is usually used to solve problems in continuous action spaces and is suitable for UAV flight trajectory planning and prediction tracking in complex dynamic environments. With the further optimization of control algorithms based on reinforcement learning and the expansion of their application fields, related algorithms will continue to play an important role in the future.

[0004] At present, the model predictive control method of quadrotor unmanned aerial vehicles (hereinafter referred to as: drones) based on reinforcement learning has a significant effect in the flight control of drones. In (Barros GM, Colombini E L. Using soft actor-critic for low-level UAV control[J].arXiv preprint arXiv:2010.02293,2020.), model-free reinforcement learning has been successfully used to control drones without prior knowledge of the robot model, and a framework is proposed to train the Soft Actor-Critic (SAC) algorithm to control quadrotors at low altitude in the target task. In (Chatterjee D, Roy R, Sengupta A. Comparison of Reinforcement Learning controller with a classical controller for an UAV [C] / / 2023 Second International Conference on Electrical, Electronics, Information and Communication Technologies (ICEEICT). IEEE, 2023: 01-05.), the performance of model predictive control (MPC) and controller trained in RL for unmanned aerial vehicles (UAVs) is compared. The simulation results on the actual nonlinear UAV model show that the performance of the RL controller is close to or even better than that of MPC. In (Romero A, Song Y, Scaramuzza D. Actor-critic model predictive control [J]. arXiv preprint arXiv: 2306.09852, 2023.), differentiable MPC is added to the Actor-Critic network framework of reinforcement learning, combining the short-term prediction optimization capability of MPC with the exploratory and end-to-end training characteristics of reinforcement learning, unifying the advantages of model-based control and reinforcement learning.

[0005] Defects and shortcomings of existing related inventions:

[0006] 1. Due to the modeling errors of drones and the complexity of the working environment, the quadrotor drone controller cannot use only one set of control parameters to control the quadrotor drone in different scenarios.

[0007] 2. Model predictive control requires a large number of prediction models and control strategies, so its computational complexity is high, resulting in poor real-time performance.

[0008] 3. In general, the reward function in reinforcement learning only obtains positive rewards when the drone reaches the target. This method is inefficient in obtaining reward feedback to help the drone explore the environment. Therefore, the convergence speed of network updates is slow, and it is difficult for the drone to learn the optimal strategy from exploration. Summary of the invention

[0009] In order to overcome the deficiencies of the prior art, the present invention aims to provide a method, system, device and medium for predictive control of unmanned aerial vehicle models based on Actor-Critic and artificial potential field.

[0010] First, artificial potential fields are introduced to enhance the perception and path planning capabilities of drones to the environment, and the rewards in the Actor-Critic algorithm are redesigned by modeling the environment as a virtual potential field; secondly, the short-term prediction optimization of model predictive control (MPC) is combined with the exploratory and end-to-end training characteristics of the Actor-Critic algorithm. The strategy generated by it effectively manages short-term strategies through the MPC based on the Actor network, and effectively manages long-term predictions for the evaluation network Critic, unifying the advantages of artificial potential fields, model-based MPC, and model-free Actor-Critic algorithms. In the artificial potential field, the drone, as the control object, gradually learns to plan a path with high safety, short distance, and short flight time in advance according to the virtual potential field through interaction with the environment. Finally, experiments are conducted in multiple static multi-obstacle environments and dynamic multi-obstacle environments; the present invention has good performance indicators in terms of cumulative reward r, flight path length L, path global distortion GS, collision and threat index, that is, the present invention has good performance in path planning and prediction tracking.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] A model predictive control method for unmanned aerial vehicle based on Actor-Critic and artificial potential field includes the following steps:

[0013] Step 1: Construct multiple static multi-obstacle environments and dynamic multi-obstacle environments;

[0014] Step 2: Construct the dynamic model of the quadrotor drone;

[0015] Step 3: Construct a reward value based on an artificial potential field;

[0016] Step 4: Construct a network of UAV model predictive control algorithm based on Actor-Critic and artificial potential field;

[0017] Step 5: Set network parameters;

[0018] Step 6: According to the network parameters set in step 5, the reward value based on the artificial potential field constructed in step 3 is input into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in step 4, and the dynamic model of the quadcopter UAV constructed in step 2 is used as the control object. Training is performed in a variety of static multi-obstacle environments and dynamic multi-obstacle environments constructed in step 1. After each round of training, the training weight file is output; by comparing with the previous training results, the training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file;

[0019] Step 7: Input the optimal weight file obtained in step 6 and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in step 4 into the various static multi-obstacle environments and dynamic multi-obstacle environments constructed in step 1 for testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs, and threat index results.

[0020] The implementation method of step 1 includes:

[0021] Step 101: Set up multiple static multi-obstacle environments: set the center position (x, y, z) and radius R of the sphere, the center position (x, y, z), radius R and height H of the bottom circle of the cylinder, and the center position (x, y, z), radius R and height H of the bottom circle of the cone;

[0022] Step 102: Set up a variety of dynamic multi-obstacle environments: set the center position (x, y, z) of the sphere, the radius R, the obstacle movement trajectory and the speed v.

[0023] The specific implementation method of step 2 is:

[0024] Define geographic coordinate system O I (X I ,Y I ,Z I ) and the drone body coordinate system O B (X B ,Y B ,Z B ), the drone is in the geographic coordinate system O I The position vector in is defined as ζ:

[0025]

[0026] The velocity vector ν of the drone is defined as:

[0027]

[0028] The attitude vector of the drone in space is defined as η:

[0029]

[0030] The rolling angle φ in formula (3) represents the rolling angle of the UAV along the geographic coordinate system X I The rotation angle of the drone along the geographic coordinate system Y I The rotation angle of the drone along the geographic coordinate system Z I The rotation angle of , the Euler angle angular velocity vector ω is:

[0031]

[0032] In formula (4), p, q, and r are the X-axis coordinates of the UAV in the body coordinate system. B , Y B , Z B The angular velocity of the axis rotation, the control quantity state space is defined as:

[0033] U=[U1 U2 U3 U4] T (5)

[0034] In formula (5), It is the sum of the thrusts generated by the four rotors, representing the total thrust of the quadcopter along the coordinate axis Z of the aircraft system. B displacement movement; is the rolling moment, which represents the rolling moment of the quadrotor drone along the X B a rotational tumbling motion; is the pitch moment, representing the Y direction of the quadrotor drone B Rotational pitch motion; is the yaw moment, representing the yaw moment along Z B Rotational yaw motion; K is the lift coefficient; K d is the anti-torque coefficient; l is the distance from the center of each rotor of the quadrotor to its center of mass; ω1, ω2, ω3, ω4 are the rotation speeds of the four rotors respectively;

[0035] The state space and state vector of the dynamic model of the quadrotor drone are:

[0036]

[0037] In formula (6), the state vector of the quadrotor drone is Therefore, the state of the quadrotor drone at time k is X(k)∈X; k x , k y , k z It is obtained by decomposing the air resistance coefficient of the UAV in the inertial reference frame; k φ , k θ, k ψ They are the total air resistance torque coefficient at X I , Y I , Z I Component of the axis; I x , I y , I z For quad-rotor drone along the aircraft system X B , Y B , Z B Moment of inertia of the shaft; J RP is the moment of inertia of the propeller along its rotation axis; Ω = -ω1 + ω2 - ω3 + ω4 is the algebraic sum of the motor speeds.

[0038] The specific implementation method of step three is:

[0039] The reward function sets the following four rewards: (1) reward r1 for the target to attract the drone; (2) reward r2 for the obstacle to repel the drone; (3) collision penalty r3; (4) reward r4 for reaching the target;

[0040] (1) Rewards for attracting drones

[0041] The attraction of the target to the UAV is proportional to the distance between the control object and the target point; as shown in formula (9), the reward r1 caused by the attraction of the target to the UAV in the artificial potential field is simplified as follows:

[0042] D attract =|qq g | (7)

[0043] D 0_g =|q0-q g | (8)

[0044]

[0045] In formula (7), D attract is the distance between the UAV and the target point, q is the position of the UAV, and q g is the target position; D in formula (8) 0_g is the distance between the initial position and the target position, q0 is the starting position; in formula (9), represents the distance between the UAV and the target at time k; represents the distance between the UAV and the target at time k+1; represents the change in the position of the drone between time k and k+1. When When , it means the drone position has not changed. When , it means the drone is moving away from the target;

[0046] (2) Rewards for obstacles that repel drones

[0047] When the drone enters the repulsive force range D of the obstacle, the reward r2 caused by the obstacle's repulsion of the drone is set by simplifying the artificial potential field;

[0048] D repel =|qq o | (10)

[0049]

[0050] When the UAV is in the repulsive field range D of the obstacle, the obstacle will produce a certain repulsive force on the UAV, making the UAV stay away from the obstacle and preventing the UAV from colliding with the obstacle. In formula (10), D repel is the distance between the UAV and the obstacle, q is the position of the UAV, and q o is the obstacle position; in formula (11), represents the distance between the UAV and the obstacle at time k; represents the distance between the drone and the obstacle at time k+1; It represents the distance change of the drone caused by obstacle rejection between time k and k+1. When When , it means the drone position has not changed. When the UAV is moving away from obstacles;

[0051] (3) Collision Penalty

[0052] When the drone collides with an obstacle, the drone will receive a collision penalty r3;

[0053]

[0054] In formula (12), the collision reward is equal to the distance between the drone and the obstacle center at the next moment: Subtract the obstacle radius R divided by the obstacle radius R minus 1;

[0055] (4) Rewards for reaching the goal

[0056] When the drone reaches the target point, the environment will give the drone a positive reward r4;

[0057]

[0058] In formula (13), when the distance between the drone and the target point is equal to 0, the reward given to the drone by the environment is 3; when the distance between the drone and the target point is not equal to 0, the reward given to the drone by the environment is 0;

[0059] The reward of artificial potential field processing is defined as:

[0060] APF-Reward=λ1·r1+λ2·r2+λ3·r3+λ4·r4 (14)

[0061] In formula (14), APF-Reward is the reward value processed by the artificial potential field; λ1, λ2, λ3, and λ4 represent the weights of the four rewards respectively.

[0062] The specific implementation method of step 4 is:

[0063] The purpose of UAV control is trajectory tracking control, so the output of the UAV system is defined as the position ζ of the UAV in the geographic coordinate system and the heading angle ψ during the flight. Therefore, the output Y of the UAV system is:

[0064]

[0065] In formula (15), the output of the UAV system includes the position x, y, z of the UAV in the geographic coordinate system and the heading angle ψ during the flight;

[0066] The UAV flight target state R(k) is set in a static multi-obstacle environment / dynamic multi-obstacle environment. The UAV measures the UAV state X(k) in real time during flight, and feeds the measured UAV state X(k) back to the input end to subtract it from the target state R(k) to obtain the deviation E(k)=R(k)-X(k). The deviation E(k) generates the output control quantity U(k) through the model predictive control module to drive the UAV to the target position to eliminate the deviation E(k). According to equations (1), (2), (3) and (4), the state of the UAV system is defined as Define the control output of the UAV system as U = [U1 U2 U3 U4] T , define the output of the UAV system as Y = [xy zψ] T ; Discretize the continuous state space model, and the discretized linear state space model of the UAV is:

[0067]

[0068] In formula (16), k represents the current time step, A represents the weight coefficient matrix of the system state; B represents the weight coefficient matrix of the controller output; X(k) represents the UAV system state at time k; U(k) represents the controller output at time k; X(k+1|k) represents the UAV system state at time k+1 predicted at time k; that is, the system state quantity X in the prediction interval N is obtained: k,N and the control quantity U k,N They are:

[0069] Xk,N =[X(k|k) X(k+1|k) … X(k+N|k)] T (17)

[0070] U k,N =[U(k|k) U(k+1|k) … U(k+N-1|k)] T (18)

[0071] In formula (17), X(k|k) is the state of the UAV at time k, X(k+1|k) is the state of the UAV at time k+1 predicted at time k, and X(k+N|k) is the state of the UAV at time k+N predicted at time k; In formula (18), U(k|k) is the control output of the UAV at time k, U(k+1|k) is the control output of the UAV at time k+1 predicted at time k, and U(k+N-1|k) is the control output of the UAV at time k+N-1 predicted at time k;

[0072] Substituting equations (17) and (18) into equation (16), we can obtain the predicted state X k,N for:

[0073] X k,N =M·X(k)+C·U k,N (19)

[0074] In formula (19), X(k) is the state of the UAV at time k, M = [IAA 2 …A N ] T ,

[0075]

[0076] Since the main control goal of the UAV is to track the expected trajectory, the expected trajectory R in the prediction interval N is defined. k,N for:

[0077] R k,N =[R(k|k) R(k+1|k) … R(k+N|k)] (20)

[0078] In formula (20), R(k+1) is the expected state at time k+1, R(k+2) is the expected state at time k+2, and R(k+N) is the expected state at time k+N;

[0079] Therefore, the trajectory error X in the prediction interval N is k,N for:

[0080] E k,N=[[R(k|k)-X(k|k)] [R(k+1|k)-X(k+1|k)] … [R(k+N|k)-X(k+N|k)]] (21)

[0081] In formula (21), [R(k|k)-X(k|k)] is the state error of the UAV system at time k, [R(k+1|k)-X(k+1|k)] is the state error of the UAV system at time k+1, and [R(k+N|k)-X(k+N|k)] is the state error of the UAV system at time k+N;

[0082] Define the objective function J:

[0083]

[0084] In formula (22), Q and R are the weight coefficient matrices of error and control quantity respectively;

[0085] Substituting equation (18) and equation (21) into equation (22), we can simplify equation (22) to obtain:

[0086]

[0087] J=E(k) T ·G·E(k)+2E(k) T ·F·U k,N +U k,N T ·H·U k,N (twenty four)

[0088] In formula (23), In formula (24),

[0089] Using the Actor-Critic algorithm, introducing the Actor strategy network and the Critic evaluation network, the UAV control strategy was improved, including:

[0090] Return G at time k k The expression is:

[0091]

[0092] In formula (25), R k+i is the reward at time k+i, and γ is the loss factor;

[0093] If the evaluation network Critic is an evaluation network of the state value state value, and the input is the state; the loss function calculation formula of the evaluation network Critic adopts the mean square error loss function, that is, the mean of the cumulative square value of the TD error value, and the expression is as follows:

[0094]

[0095] In formula (26), L critic is the loss function when the evaluation network Critic is the evaluation network of the state value, V(s k ) is the state s at time k k The corresponding value, V(s k+1 ) is the state s at time k+1 k+1 The corresponding value;

[0096] If the evaluation network Critic is an evaluation network of action value actionvalue, that is, the input is state and action, then the loss function of the evaluation network Critic is as follows:

[0097]

[0098] In formula (27), L critic is the loss function when the evaluation network Critic is the evaluation network of action value actionvalue, Q(s k ,a k ) is the k moment, the controlled object is at s k When taking action k When, the reward is obtained;

[0099] The optimization goals of the Actor network are as follows:

[0100]

[0101] In formula (28), π * represents the optimal strategy. Since the meaning of this formula is to increase the probability of action selection when the TD error value is greater than 0, and to reduce the probability of action selection when the TD error value is less than 0, the goal is to minimize the loss function -L actor ;

[0102] Therefore, the value function Vπ(s) of the evaluation network Critic has parameter θ q The updated formula is:

[0103]

[0104] In formula (29), α1 is the descending step size, is the descending gradient;

[0105] Therefore, the strategy function of the Actor network is θ (s) Parameter θ π Updated formula:

[0106]

[0107] In formula (30), α2 is the descending step size, is the descending gradient;

[0108] Minimize the objective function J; transform the problem of solving the optimal control quantity into the following quadratic optimization problem:

[0109]

[0110] In formula (31), U(k) min and U(k) max are the minimum and maximum control values ​​respectively; the Actor network outputs the real-time optimal F, H weight coefficient matrix, and the evaluation network Critic evaluates the strategy through TDerror; then, solve the quadratic optimization problem to obtain the optimal control value sequence U within the prediction interval N k,N , then select the first item U(k+1|k) in the sequence as the output of the MPC controller at the current time k, and perform rolling optimization and feedback correction to control the quadrotor drone to track the trajectory.

[0111] The specific implementation method of step seven is:

[0112] The calculation of experimental data such as path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs and threat index is as follows:

[0113] (1) Path length L

[0114]

[0115] In formula (32), S k Indicates the distance the drone moves from time k to time k+1, which is equal to the distance between the drone and the target at time k The distance between the drone and the target at time k+1 The absolute value of the difference;

[0116]

[0117] In formula (33), the path length L is equal to S k Iterative addition, N is the number of algorithm iterations;

[0118] (2) Maximum path distortion LS

[0119] The distortion τ of the UAV flight curve at time k k for:

[0120]

[0121] In formula (34), T k is the tangent vector of the UAV flight curve at time k, N k is the normal vector of the UAV flight curve at time k, B k is the cross product of the normal vectors of the UAV flight curve at time k (also called the double normal vector of the hyperbola), B′ k is the derivative of the double normal vector of the UAV flight curve at time k;

[0122] LS=max{τ 0 ,τ 1 ,...,τ N} (35)

[0123] (3) Path global distortion GS

[0124]

[0125] The distortion on the path is integrated to obtain the path global distortion measure GS, where τ(s) is the distortion of point s on the path, and a and b are the starting point and end point of the UAV flight path;

[0126] (4) Whether a collision occurred

[0127]

[0128] In formula (37), the distance between the UAV and the obstacle at time k is When the distance between the drone and the obstacle is less than or equal to the obstacle radius R, the drone collides with the obstacle; when the distance between the drone and the obstacle at time k is When it is larger than the obstacle radius R, the drone does not collide with the obstacle;

[0129] (5) Threat Index

[0130]

[0131] In formula (38), when When the drone collides with an obstacle, the threat index is equal to ∞; When the drone is within the obstacle rejection range but no collision occurs, the threat index is equal to ν k is the speed of the drone at time k, is the distance between the UAV and the obstacle center at time k; When the drone does not enter the obstacle rejection range, the threat index is 0.

[0132] The present invention also provides a UAV model predictive control system based on Actor-Critic and artificial potential field, comprising:

[0133] Environment construction module: used to construct a variety of static multi-obstacle environments and dynamic multi-obstacle environments;

[0134] Quad-rotor UAV dynamics model construction module: used to build the dynamics model of the quad-rotor UAV;

[0135] Artificial potential field-based reward value construction module: used to construct the reward value based on the artificial potential field;

[0136] Network construction module: used to construct the network of UAV model predictive control method based on Actor-Critic and artificial potential field;

[0137] Network training module: According to the set network parameters, the reward value based on the artificial potential field is input into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field. The dynamic model of the quadcopter is used as the control object, and training is carried out in a variety of static multi-obstacle environments and dynamic multi-obstacle environments. After each round of training, the training weight file is output; the training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file;

[0138] Test result output module: used to input the optimal weight file and the network of the constructed UAV model predictive control algorithm based on Actor-Critic and artificial potential field into a variety of constructed static multi-obstacle environments and dynamic multi-obstacle environments for testing, and obtain the UAV flight path map, reward value r, path length L, path maximum distortion LS, path global distortion GS, whether a collision occurs and threat index results.

[0139] The present invention also provides a UAV model predictive control device based on Actor-Critic and artificial potential field, comprising:

[0140] Memory: a computer program for storing the above-mentioned UAV model predictive control method based on Actor-Critic and artificial potential field, which is a computer-readable device;

[0141] Processor: used to implement the Actor-Critic and artificial potential field based UAV model predictive control method when executing the computer program.

[0142] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the unmanned aerial vehicle model predictive control method based on Actor-Critic and artificial potential field.

[0143] Compared with the prior art, the present invention has the following beneficial effects:

[0144] 1. Under the framework of the Actor-Critic algorithm, the present invention innovatively uses the evaluation network Critic to evaluate the pros and cons of the current control strategy (i.e., the MPC weight coefficient matrix) in real time, and then controls the Actor network to dynamically adjust and output the optimal MPC weight coefficient matrices F and H. Since both the Actor network and the evaluation network Critic are implemented through deep neural networks, they have powerful nonlinear mapping and generalization capabilities, and can process complex input data (such as drone status information, environmental perception data, etc.) and output high-precision control instructions. This end-to-end optimization method not only simplifies the complex optimization process in the traditional MPC algorithm, but also significantly improves the computational efficiency and real-time performance of the system.

[0145] 2. The present invention deeply optimizes and reconstructs the core component of Model Predictive Control (MPC) - the objective function (Formula (22)). Specifically, the complex model described by Formula (22) is significantly simplified. This innovative improvement aims to solve the high computational complexity and huge computational workload faced by high-speed dynamic systems such as drones. It not only reduces the number of numerical iterations in the calculation process, but also reduces the demand for computing resources, thereby significantly improving the calculation speed of the system.

[0146] 3. The present invention integrates the artificial potential field method into the Actor-Critic algorithm framework to optimize the reward mechanism and obtains the reward APF-Reward processed by the artificial potential field method. This innovation effectively accelerates the convergence speed of the Actor-Critic network update, improves the learning efficiency and decision-making ability of the UAV in the constructed static multi-obstacle environment\dynamic multi-obstacle environment, and enables the UAV to quickly transition from the initial exploration stage to the advanced stage of mastering and executing the optimal flight strategy.

[0147] In summary, the present invention unifies the advantages of artificial potential field, model-based model predictive control and model-free Actor-Critic algorithm by introducing rewards for artificial potential field processing in the Actor-Critic algorithm and combining it with the UAV model predictive control method. The network is trained and predicted using multiple designed static multi-obstacle environments and dynamic multi-obstacle environments, which can effectively plan paths and avoid obstacles in time. It has the advantages of high real-time performance, strong robustness and timely obstacle avoidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0148] Figure 1 It is a flowchart of an algorithm according to an embodiment of the present invention.

[0149] Figure 2 It is the coordinate definition of the drone modeling in the embodiment of the present invention.

[0150] Figure 3 It is a force analysis diagram of the improved artificial potential field method according to an embodiment of the present invention.

[0151] Figure 4 4 is a block diagram of a model predictive control system according to an embodiment of the present invention.

[0152] Figure 5 This is a simulation diagram of a UAV in a static multi-obstacle environment using traditional technology (Actor-Critic MPC algorithm). The cyan point is the starting point, the red point is the target point, and the red track is the flight path of the UAV. Figure 5 (a) is the flight result of a UAV using traditional technology (Actor-Critic MPC algorithm) in static multi-obstacle environment No. 1. Figure 5 (b) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the static multi-obstacle environment No. 2. Figure 5 (c) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the static multi-obstacle environment No. 3. Figure 5 (d) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the static multi-obstacle environment No. 4. Figure 5 (e) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the static multi-obstacle environment No. 5. Figure 5 (f) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the static multi-obstacle environment No. 6.

[0153] Figure 6This is a simulation diagram of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a static multi-obstacle environment. The cyan point is the starting point, the red point is the target point, and the red track is the flight path of the UAV. Figure 6 (a) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a static multi-obstacle environment No. 1. Figure 6 (b) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a static multi-obstacle environment No. 2. Figure 6 (c) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a static multi-obstacle environment No. 3. Figure 6 (d) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a static multi-obstacle environment No. 4. Figure 6 (e) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a static multi-obstacle environment No. 5. Figure 6 (f) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a static multi-obstacle environment No. 6.

[0154] Figure 7 This is the cumulative reward curve of the strategy network in 6 dynamic multi-obstacle environments of the embodiment of the present invention (Actor-Critic APF MPC algorithm). The present invention (Actor-Critic APF MPC algorithm) is a blue curve, and the traditional technology (Actor-Critic MPC algorithm) is an orange curve. Among them, Figure 7 (a) is a comparison curve of the cumulative rewards of the UAVs trained in the No. 1 dynamic multi-obstacle environment using the traditional technology (Actor-Critic MPC algorithm) and the invention (Actor-Critic APF MPC algorithm). Figure 7 (b) is a comparison curve of the cumulative rewards of the UAVs trained in the dynamic multi-obstacle environment No. 2 using the traditional technology (Actor-Critic MPC algorithm) and the invention (Actor-Critic APF MPC algorithm). Figure 7 (c) is a comparison curve of the cumulative rewards of the UAVs trained in the dynamic multi-obstacle environment No. 3 using the traditional technology (Actor-Critic MPC algorithm) and the invention (Actor-Critic APFMPC algorithm). Figure 7(d) is a comparison curve of the cumulative rewards of the UAVs trained in the No. 4 dynamic multi-obstacle environment using the traditional technology (Actor-Critic MPC algorithm) and the present invention (Actor-Critic APF MPC algorithm). Figure 7 (e) is a comparison curve of the cumulative rewards of the UAVs trained in the dynamic multi-obstacle environment No. 5 using the traditional technology (Actor-Critic MPC algorithm) and the invention (Actor-Critic APF MPC algorithm). Figure 7 (f) is a comparison curve of the cumulative rewards of the UAVs trained in the dynamic multi-obstacle environment No. 6 using the traditional technology (Actor-Critic MPC algorithm) and the present invention (Actor-Critic APF MPC algorithm).

[0155] Figure 8 This is a simulation diagram of a UAV in a dynamic multi-obstacle environment using traditional technology (Actor-Critic MPC algorithm). The green triangle is the UAV, the cyan point is the starting point, the red point is the target point, the red track is the UAV flight path, and the black track is the obstacle movement track. Figure 8 (a) is the flight result of a UAV using traditional technology (Actor-Critic MPC algorithm) in dynamic multi-obstacle environment No. 1. Figure 8 (b) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the dynamic multi-obstacle environment No. 2. Figure 8 (c) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the dynamic multi-obstacle environment No. 3. Figure 8 (d) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the dynamic multi-obstacle environment No. 4. Figure 8 (e) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the dynamic multi-obstacle environment No. 5. Figure 8 (f) is the flight result of the UAV using traditional technology (Actor-Critic MPC algorithm) in the dynamic multi-obstacle environment No. 6.

[0156] Fig. 9 This is a simulation diagram of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a dynamic multi-obstacle environment. The green triangle is the UAV, the cyan point is the starting point, the red point is the target point, the red track is the flight path of the UAV, and the black track is the obstacle motion track. Fig. 9(a) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a dynamic multi-obstacle environment No. 1. Fig. 9 (b) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a dynamic multi-obstacle environment No. 2. Fig. 9 (c) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a dynamic multi-obstacle environment No. 3. Fig. 9 (d) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a dynamic multi-obstacle environment No. 4. Fig. 9 (e) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a dynamic multi-obstacle environment No. 5. Fig. 9 (f) is a diagram showing the flight results of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a dynamic multi-obstacle environment No. 6. DETAILED DESCRIPTION

[0157] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0158] In order to solve the problems in the prior art, the present invention takes advantage of the high learning efficiency and excellent performance of reinforcement learning and model predictive control, proposes a UAV model predictive control method based on Actor-Critic and artificial potential field, and makes the following improvements:

[0159] 1. The evaluation network Critic in the Actor-Critic algorithm evaluates the quality of the current strategy, so that the Actor network outputs the real-time optimal MPC weight coefficient matrix to control the drone to fly along the predicted trajectory.

[0160] 2. Simplify the objective function of model predictive control, reduce the amount of calculation and complexity, and improve the system calculation speed to achieve efficient real-time control of the UAV.

[0161] 3. By introducing rewards processed by artificial potential field method in Actor-Critic algorithm, the convergence speed of Actor-Critic network update is improved, so that the drone can quickly learn the optimal strategy from exploration. The details are as follows:

[0162] A model predictive control method for unmanned aerial vehicle based on Actor-Critic and artificial potential field includes the following steps:

[0163] Step 1: Construct a variety of static multi-obstacle environments and dynamic multi-obstacle environments to provide a test environment for drone simulation;

[0164] Step 2: Construct the dynamic model of the quadrotor drone;

[0165] Step 3: Construct a reward value based on an artificial potential field;

[0166] Step 4: Construct a network of UAV model predictive control algorithm based on Actor-Critic and artificial potential field;

[0167] Step 5: Set network parameters;

[0168] Step 6: According to the network parameters set in step 5, the reward value based on the artificial potential field constructed in step 3 is input into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in step 4, and the dynamic model of the quadcopter UAV constructed in step 2 is used as the control object. Training is performed in a variety of static multi-obstacle environments and dynamic multi-obstacle environments constructed in step 1. After each round of training, the training weight file is output; by comparing with the previous training results, the training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file;

[0169] Step 7: Input the optimal weight file obtained in step 6 and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in step 4 into the various static multi-obstacle environments and dynamic multi-obstacle environments constructed in step 1 for experimental testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs, and threat index results.

[0170] The algorithm flow chart of the present invention is as follows Figure 1 As shown in the figure, the UAV introduces an artificial potential field method in the constructed static multi-obstacle environment / dynamic multi-obstacle environment to deal with the path planning problem of the UAV flying in a complex environment, and performs effective obstacle avoidance and path planning. The path generated by the artificial potential field is combined with the real-time trajectory of the UAV as a reference trajectory and used as the input of the control system. The reward is set using the idea of ​​the artificial potential field to improve the convergence speed and accuracy of the system. The UAV explores and learns in the constructed static multi-obstacle environment / dynamic multi-obstacle environment. The control system obtains the state X(k) at time k through the state observer, combines it with the expected state R(k), and obtains E(k), which is passed as input to the model predictive controller. X(k) is passed as input to the evaluation network Critic and Actor network of the Actor-Critic algorithm. The evaluation network Critic outputs TDerror to evaluate the current strategy to improve the strategy and drive learning. The Actor network outputs the weight coefficient matrix F of the model predictive controller k , H k, calculate the control quantity sequence U within the prediction interval N k,N Then, the first item U(k+1|k) in the sequence is selected as the control quantity at time k+1, and rolling optimization and feedback correction are performed to control the UAV to perform path planning and predictive tracking, which effectively improves the robustness and adaptability of the system.

[0171] The implementation method of step 1 includes:

[0172] Step 101: Set up multiple static multi-obstacle environments: set the center position (x, y, z) and radius R of the sphere, the center position (x, y, z), radius R and height H of the bottom circle of the cylinder, and the center position (x, y, z), radius R and height H of the bottom circle of the cone;

[0173] Step 102: Set up a variety of dynamic multi-obstacle environments: set the center position (x, y, z) of the sphere, the radius R, the obstacle movement trajectory and the speed v.

[0174] The specific implementation method of step 2 is:

[0175] like Figure 2 As shown, define the geographic coordinate system O I (X I ,Y I ,Z I ) and the drone body coordinate system O B (X B ,Y B ,Z B ), the drone is in the geographic coordinate system O I The position vector in is defined as ζ:

[0176]

[0177] The velocity vector ν of the drone is defined as:

[0178]

[0179] The attitude vector of the drone in space is defined as η:

[0180]

[0181] The rolling angle φ in formula (3) represents the rolling angle of the UAV along the geographic coordinate system X I The rotation angle of the drone along the geographic coordinate system Y I The rotation angle of the drone along the geographic coordinate system Z I The rotation angle of , the Euler angle angular velocity vector ω is:

[0182]

[0183] In formula (4), p, q, and r are the X-axis coordinates of the UAV in the body coordinate system. B , Y B , Z B The angular velocity of the axis rotation, the control quantity state space is defined as:

[0184] U=[U1 U2 U3 U4] T (5)

[0185] In formula (5), It is the sum of the thrusts generated by the four rotors, representing the total thrust of the quadcopter along the coordinate axis Z of the aircraft system. B displacement movement; is the rolling moment, which represents the rolling moment of the quadrotor drone along the X B a rotational tumbling motion; is the pitch moment, representing the Y direction of the quadrotor drone B Rotational pitch motion; is the yaw moment, representing the yaw moment along Z B Rotational yaw motion; K is the lift coefficient; K d is the anti-torque coefficient; l is the distance from the center of each rotor of the quadrotor to its center of mass; ω1, ω2, ω3, ω4 are the rotation speeds of the four rotors respectively;

[0186] The state space and state vector of the dynamic model of the quadrotor drone are:

[0187]

[0188] In formula (6), the state vector of the quadrotor drone is Therefore, the state of the quadrotor drone at time k is X(k)∈X; k x , k y , k z It is obtained by decomposing the air resistance coefficient of the UAV in the inertial reference frame; k φ , k θ , k ψ They are the total air resistance torque coefficient at X I , Y I , Z I Component of the axis; I x , I y , I z For quad-rotor drone along the aircraft system X B , Y B , Z B Moment of inertia of the shaft; J RP is the moment of inertia of the propeller along its rotation axis; Ω = -ω1 + ω2 - ω3 + ω4 is the algebraic sum of the motor speeds.

[0189] The specific implementation method of step three is:

[0190] The present invention solves the problem of local optimum and unreachable target by improving the obstacle repulsion potential field function. The force analysis is as follows: Figure 3 By adding an adjustment factor to the obstacle repulsion field model of the traditional artificial potential field method, the repulsion and attraction are reduced to zero at the same time only when the UAV reaches the target point, thus solving the problems of local optimality and unreachable target.

[0191] The reward of the Actor-Critic algorithm in the present invention is the feedback signal obtained by the drone in the environment exploration, which is used to evaluate the actions taken by the controlled object. A simple method is to set a general reward, that is, the drone can only get a positive reward when the task is completed. However, this method is inefficient in collecting useful empirical data to help the controlled object explore the environment. Therefore, the convergence speed of the network update is slow, and the drone cannot learn the optimal strategy from the exploration. The present invention introduces the idea of ​​the artificial potential field method (obstacles and targets exert repulsion and attraction on the controlled object respectively) into the design of the reward function. The reward function setting of the present invention takes into account the following four rewards: (1) reward r1 for the target attracting the drone; (2) reward r2 for the obstacle repelling the drone; (3) collision penalty r3; (4) reward r4 for reaching the target;

[0192] (1) Rewards for attracting drones

[0193] The attraction of the target to the UAV is proportional to the distance between the control object and the target point; as shown in formula (9), the reward r1 caused by the attraction of the target to the UAV in the artificial potential field is simplified as follows:

[0194] D attract =|qq g | (7)

[0195] D 0_g =|q0-q g | (8)

[0196]

[0197] In formula (7), D attract is the distance between the UAV and the target point, q is the position of the UAV, and q g is the target position; D in formula (8) 0_g is the distance between the initial position and the target position, q0 is the starting position; in formula (9), represents the distance between the UAV and the target at time k; represents the distance between the UAV and the target at time k+1; represents the change in the position of the drone between time k and k+1. When When , it means the drone position has not changed. When , it means the drone is moving away from the target;

[0198] (2) Rewards for obstacles that repel drones

[0199] When the drone enters the repulsive force range D of the obstacle, similar to the reward set for the target to attract the drone, the present invention sets the reward r2 caused by the obstacle's repulsion of the drone by simplifying the artificial potential field;

[0200] D repel =|qq o | (10)

[0201]

[0202] When the UAV is in the repulsive field range D of the obstacle, the obstacle will produce a certain repulsive force on the UAV, making the UAV stay away from the obstacle and preventing the UAV from colliding with the obstacle. In formula (10), D repel is the distance between the UAV and the obstacle, q is the position of the UAV, and q o is the obstacle position; in formula (11), represents the distance between the UAV and the obstacle at time k; represents the distance between the drone and the obstacle at time k+1; It represents the distance change of the drone caused by obstacle rejection between time k and k+1. When When , it means the drone position has not changed. When the UAV is moving away from obstacles;

[0203] (3) Collision Penalty

[0204] When the drone collides with an obstacle, the drone will receive a collision penalty r3;

[0205]

[0206] In formula (12), the collision reward is equal to the distance between the drone and the obstacle center at the next moment: Subtract the obstacle radius R divided by the obstacle radius R minus 1;

[0207] (4) Rewards for reaching the goal

[0208] When the drone reaches the target point, the environment will give the drone a positive reward r4;

[0209]

[0210] In formula (13), when the distance between the drone and the target point is equal to 0, the reward given to the drone by the environment is 3; when the distance between the drone and the target point is not equal to 0, the reward given to the drone by the environment is 0;

[0211] In summary, the reward of artificial potential field processing can be defined as:

[0212] APF-Reward=λ1·r1+λ2·r2+λ3·r3+λ4·r4 (14)

[0213] In formula (14), APF-Reward is the reward value processed by the artificial potential field; λ1, λ2, λ3, and λ4 represent the weights of the four rewards respectively.

[0214] The specific implementation method of step 4 is:

[0215] The purpose of UAV control is trajectory tracking control, so the output of the UAV system is defined as the position ζ of the UAV in the geographic coordinate system and the heading angle ψ during the flight. Therefore, the output Y of the UAV system is:

[0216]

[0217] In formula (15), the output of the UAV system includes the position x, y, z of the UAV in the geographic coordinate system and the heading angle ψ during the flight;

[0218] like Figure 4 As shown in the figure, the UAV flight target state R(k) is set in a static multi-obstacle environment / dynamic multi-obstacle environment. The UAV measures the UAV state X(k) in real time during the flight, and feeds the measured UAV state X(k) back to the input end to subtract it from the target state R(k) to obtain the deviation E(k)=R(k)-X(k). The deviation E(k) generates the output control quantity U(k) through the model predictive control module to drive the UAV to fly to the target position to eliminate the deviation E(k). According to formula (1), formula (2), formula (3) and formula (4), the state of the UAV system is defined as Define the control output of the UAV system as U = [U1 U2 U3 U4] T , define the output of the UAV system as Y = [xy zψ] T ; Model predictive control is a digital control method, so the continuous state space model needs to be discretized first; the discretized linear state space model of the UAV is:

[0219] X(k+1|k)=A·X(k)+B·U(k) (16)

[0220] In formula (16), k represents the current time step, A represents the weight coefficient matrix of the system state; B represents the weight coefficient matrix of the controller output; X(k) represents the UAV system state at time k; U(k) represents the controller output at time k; X(k+1|k) represents the UAV system state at time k+1 predicted at time k; thus, the system state quantity X in the prediction interval N can be obtained: k,N and the control quantity U k,N They are:

[0221] X k,N =[X(k|k) X(k+1|k) … X(k+N|k)] T (17)

[0222] U k,N =[U(k|k) U(k+1|k) … U(k+N-1|k)] T (18)

[0223] In formula (17), X(k|k) is the state of the UAV at time k, X(k+1|k) is the state of the UAV at time k+1 predicted at time k, and X(k+N|k) is the state of the UAV at time k+N predicted at time k; In formula (18), U(k|k) is the control output of the UAV at time k, U(k+1|k) is the control output of the UAV at time k+1 predicted at time k, and U(k+N-1|k) is the control output of the UAV at time k+N-1 predicted at time k;

[0224] Substituting equation (17) and equation (18) into equation (16), we can obtain the predicted state X k,N for:

[0225] X k,N =M·X(k)+C·U k,N (19)

[0226] In formula (19), X(k) is the state of the UAV at time k, M = [IAA 2 …A N ] T ,

[0227]

[0228] Since the main control goal of the UAV is to track the expected trajectory, the expected trajectory R in the prediction interval N is defined. k,N for:

[0229] R k,N =[R(k|k) R(k+1|k) … R(k+N|k)] (20)

[0230] In formula (20), R(k+1) is the expected state at time k+1, R(k+2) is the expected state at time k+2, and R(k+N) is the expected state at time k+N;

[0231] Therefore, the trajectory error X in the prediction interval N is k,N for:

[0232] E k,N =[[R(k|k)-X(k|k)] [R(k+1|k)-X(k+1|k)] … [R(k+N|k)-X(k+N|k)]] (21)

[0233] In formula (21), [R(k|k)-X(k|k)] is the state error of the UAV system at time k, [R(k+1|k)-X(k+1|k)] is the state error of the UAV system at time k+1, and [R(k+N|k)-X(k+N|k)] is the state error of the UAV system at time k+N;

[0234] Define the objective function J:

[0235]

[0236] In formula (22), Q and R are the weight coefficient matrices of error and control quantity, respectively. In simulation and practical applications, adjusting Q and R can make the UAV focus on trajectory tracking or save energy, laying a solid foundation for achieving good experimental results.

[0237] Substituting equation (18) and equation (21) into equation (22), we can simplify equation (22) to obtain:

[0238]

[0239] J=E(k) T ·G·E(k)+2E(k) T ·F·U k,N +U k,N T ·H·U k,N (twenty four)

[0240] In formula (23), In formula (24),

[0241] The present invention uses a UAV control algorithm that combines the Actor-Critic algorithm, the artificial potential field method, and the model predictive control method to perform obstacle avoidance and trajectory planning for the UAV, aiming to significantly improve the obstacle avoidance capability and trajectory planning efficiency of the UAV in the constructed static multi-obstacle environment\dynamic multi-obstacle environment. By integrating these advanced methods, the adaptability of the UAV in the face of dynamic environmental changes can be enhanced.

[0242] The Actor-Critic algorithm is an existing algorithm that combines the ideas of Policy Base and Value Base. Actor is implemented using the policy gradient algorithm, and Critic is implemented using temporal difference. Actor is the policy function π θ (s), usually implemented with a neural network, the input is the current state, and the output is an action. The training goal of the network is to maximize the expected cumulative return. Critic is the value function V π (s), the network can estimate the value function of the current strategy, that is, it can evaluate the quality of the Actor (strategy function). The original Actor (strategy gradient method) uses the expectation of cumulative returns as the training basis. In this way, π can only be updated after the end of the round. θ Parameters of (s).

[0243] Return G at time k k The expression is:

[0244]

[0245] In formula (25), R k+i is the reward at time k+i, and γ is the loss factor;

[0246] If the evaluation network Critic is an evaluation network of the state value state value, and the input is the state; the loss function calculation formula of the evaluation network Critic adopts the mean square error loss function, that is, the mean of the cumulative square value of the TD error value, and the expression is as follows:

[0247]

[0248] In formula (26), L critic is the loss function when the evaluation network Critic is the evaluation network of the state value, V(s k ) is the state s at time k k The corresponding value, V(s k+1 ) is the state s at time k+1 k+1 The corresponding value;

[0249] If the evaluation network Critic is an evaluation network of action value actionvalue, that is, the input is state and action, then the loss function of the evaluation network Critic is as follows:

[0250]

[0251] In formula (27), L critic is the loss function when the evaluation network Critic is the evaluation network of action value actionvalue, Q(s k ,a k ) is the k moment, the controlled object is at s k When taking action k When, the reward is obtained;

[0252] The optimization goals of the Actor network can be as follows:

[0253]

[0254] In formula (28), π * represents the optimal strategy. Since the meaning of this formula is to increase the probability of action selection when the TD error value is greater than 0, and to reduce the probability of action selection when the TD error value is less than 0, the goal is to minimize the loss function -L actor ;

[0255] Therefore, the value function Vπ(s) of the evaluation network Critic has parameter θ q The updated formula is:

[0256]

[0257] In formula (29), α1 is the descending step size, is the descending gradient;

[0258] Therefore, the strategy function of the Actor network is θ (s) Parameter θ π Updated formula:

[0259]

[0260] In formula (30), α2 is the descending step size, is the descending gradient;

[0261] In order to control the drone, it is necessary to minimize the objective function J; therefore, the problem of solving the optimal control quantity is transformed into the following quadratic optimization problem:

[0262]

[0263] In formula (31), U(k) min and U(k)max are the minimum and maximum control values ​​respectively; the Actor network outputs the real-time optimal F, H weight coefficient matrix, and the evaluation network Critic evaluates the strategy through TDerror; then, solve the quadratic optimization problem and obtain the optimal control value sequence U within the prediction interval N k,N , then select the first item U(k+1|k) in the sequence as the output of the MPC controller at the current time k, and perform rolling optimization and feedback correction to control the quadrotor drone to track the trajectory.

[0264] In step 5, the following settings are set: training round: episode = 1000, step size = 0.2, batch size batch_size ≥ 24, learning rate ≤ 3 × 10 -4 .

[0265] The specific implementation method of step seven is:

[0266] The calculation of experimental data such as path length L, maximum path distortion LS, global path distortion GS, collision and threat index in the simulation experiment is as follows:

[0267] (1) Path length L

[0268]

[0269] In formula (32), S k Indicates the distance the drone moves from time k to time k+1, which is equal to the distance between the drone and the target at time k The distance between the drone and the target at time k+1 The absolute value of the difference;

[0270]

[0271] In formula (33), the path length L is equal to S k The iterative addition is performed, and N is the number of algorithm iterations.

[0272] (2) Maximum path distortion LS

[0273] The distortion τ of the UAV flight curve at time k k for:

[0274]

[0275] In formula (34), T k is the tangent vector of the UAV flight curve at time k, N k is the normal vector of the UAV flight curve at time k, B k is the cross product of the normal vectors of the UAV flight curve at time k (also called the double normal vector of the hyperbola), B′ kis the derivative of the double normal vector of the UAV flight curve at time k;

[0276] LS=max{τ 0 ,τ 1 ,...,τ N} (35)

[0277] (3) Path global distortion GS

[0278]

[0279] The distortion on the path is integrated to obtain the path global distortion measure GS, where τ(s) is the distortion of point s on the path, and a and b are the starting point and end point of the UAV flight path;

[0280] (4) Whether a collision occurred

[0281]

[0282] In formula (37), the distance between the UAV and the obstacle at time k is When the distance between the drone and the obstacle is less than or equal to the obstacle radius R, the drone collides with the obstacle; when the distance between the drone and the obstacle at time k is When it is larger than the obstacle radius R, the drone does not collide with the obstacle;

[0283] (5) Threat Index

[0284]

[0285] In formula (38), when When the drone collides with an obstacle, the threat index is equal to ∞; When the drone is within the obstacle rejection range but no collision occurs, the threat index is equal to ν k is the speed of the drone at time k, is the distance between the UAV and the obstacle center at time k; When the drone does not enter the obstacle rejection range, the threat index is 0.

[0286] Example

[0287] like Figure 1 As shown, the algorithm in the present invention, a UAV model predictive control method based on Actor-Critic and artificial potential field, comprises the following steps:

[0288] Step 1: The UAV introduces an artificial potential field method in the constructed static multi-obstacle environment / dynamic multi-obstacle environment to deal with the path planning problem in complex environments, and performs effective obstacle avoidance and path planning;

[0289] Step 2: Construct a dynamic model of quadcopter flight in a variety of static multi-obstacle environments / dynamic multi-obstacle environments and define the geographic coordinate system O I (X I ,Y I ,Z I ), UAV body coordinate system O B (X B ,Y B ,Z B ), the UAV is in the geographic coordinate system O I The position vector ζ, the velocity vector ν of the UAV, the attitude vector η of the UAV in space, the angular velocity vector ω of the Euler angle, the state quantity X, the control quantity U and the output quantity Y;

[0290] Step 3: Construct a reward value based on an artificial potential field. The present invention introduces the idea of ​​the artificial potential field method (obstacles and targets exert repulsion and attraction on the controlled object respectively) into the design of the reward function. The reward function setting of the present invention considers the following four rewards: (1) reward r1 for the target to attract the drone; (2) reward r2 for the obstacle to repel the drone; (3) collision penalty r3; (4) reward r4 for reaching the target;

[0291] Step 4: The present invention first defines the objective function J=E(k) of the model predictive control method: T ·G·E(k)+2E(k) T ·F·U k,N +U k,N T ·H·U k,N , construct the Actor network and the evaluation network Critic of the Actor-Critic algorithm, where Actor is the strategy function π θ (s), the input is the current state, the output is an action; Critic is the value function V π (s), the network estimates the value function of the current strategy and evaluates the quality of the Actor network; in order to control the drone, it is necessary to minimize the objective function J, so the problem of solving the optimal control amount is transformed into the following quadratic optimization problem:

[0292]

[0293] Step 5: The network parameters of the present invention are set as follows: training round: episode = 1000, step size = 0.2, batch size batch_size ≥ 24, learning rate ≤ 3 × 10 -4 ;

[0294] Step 6: According to the network parameters set in step 5, the reward value based on the artificial potential field constructed in step 3 is input into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in step 4, and the dynamic model of the quadcopter UAV constructed in step 2 is used as the control object. Training is performed in a variety of static multi-obstacle environments and dynamic multi-obstacle environments constructed in step 1. After each round of training, the training weight file is output; by comparing with the previous training results, the training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file;

[0295] Step 7: Input the optimal weight file obtained in step 6 and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in step 4 into the various static multi-obstacle environments and dynamic multi-obstacle environments constructed in step 1 for experimental testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs, and threat index results.

[0296] like Figure 2 As shown, the present invention provides a method for defining a coordinate system of a quad-rotor UAV.

[0297] The quadrotor drone (hereinafter referred to as drone) is a nonlinear, multivariable, strongly coupled underactuated control system. It is relatively complicated to directly model its dynamics. In order to simplify the complexity of the model, the following assumptions are made for the research object:

[0298] (1) The quadrotor is a uniform and symmetrical rigid body;

[0299] (2) The mass and moment of inertia of the quadrotor do not change;

[0300] (3) The geometric center of the quadrotor coincides with its center of gravity;

[0301] (4) A quadrotor is subject only to gravity and propeller pull.

[0302] Based on the Newton-Euler method, the dynamic characteristics of the UAV are analyzed and an accurate UAV model is established to understand and analyze the system characteristics of the UAV and use it as a simulation model for simulation experiments. Secondly, the dynamic model is reasonably simplified and used as a prediction model for the model predictive control method.

[0303] like Figure 3 As shown, the present invention provides a method for improving artificial potential field.

[0304] (1) Gravitational potential field

[0305] The gravitational potential field is mainly related to the distance between the drone and the target point. The greater the distance, the greater the potential energy value of the drone; the smaller the distance, the smaller the potential energy value of the drone, so the function of the gravitational potential field is:

[0306]

[0307] In formula (40), η is the proportional gain coefficient, ρ(q,q g ) is a vector, representing the position q of the drone and the position q of the target point g Euclidean distance between |qq g |, the vector direction is from the drone position to the target point position. The corresponding gravitational force F attract (q) is the negative gradient of the gravitational field, representing the gravitational potential field function U attract (q) changes fastest in the direction.

[0308]

[0309] (2) Repulsive potential field

[0310] Improve the obstacle repulsion potential field function to solve the problem of local optimality and unreachable target. The force analysis is as follows: Figure 3 As shown. Adding the adjustment factor ρ to the obstacle repulsion field model of the traditional artificial potential field method n (q,q g ), so that the repulsion and gravitation will be reduced to zero at the same time only when the UAV reaches the target point, thus solving the problems of local optimum and unreachable target.

[0311] The improved repulsive potential field function is:

[0312]

[0313] In formula (42), k is the proportional coefficient, ρ(q,q o ) is a vector, the direction is from the obstacle to the drone, and the size is the Euclidean distance between the drone and the obstacle. o |; D is a constant, indicating the maximum impact range of obstacles on the drone; n is an optional positive constant.

[0314] The corresponding repulsive force is the negative gradient force of the repulsive potential field:

[0315]

[0316] In formula (43), F repel1 The direction of the obstacle is toward the drone; F repel2The direction of the drone points to the target point. The distance between the drone and the target point is added to the improved repulsive field function, so that the gravity and repulsion of the drone are reduced to a certain extent when it flies to the target. Only when the drone reaches the target point, the gravity and repulsion are reduced to zero at the same time, that is, the target point becomes the minimum point of potential energy value, thus solving the problem of local optimum and unreachable target.

[0317] (3) Resultant potential field

[0318] According to the gravitational field function and repulsive field function defined above, the composite field of the entire operating space can be obtained. The magnitude of the resultant potential field of the UAV is the sum of the repulsive potential field and the gravitational potential field to which the UAV is subjected. Therefore, the total function of the resultant potential field is:

[0319] U(q)=U attract (q)+U repel (q) (44)

[0320] The combined force is

[0321]

[0322] The direction of the resultant force determines the flight direction of the drone, and the magnitude of the resultant force determines the flight acceleration of the drone.

[0323] The effect of the present invention is further described below in conjunction with simulation experiments:

[0324] 1. Simulation experiment conditions

[0325] The hardware platform of the simulation experiment of the present invention is: CPU: Intel I5-13490F, main frequency is 2.5GHz, 16G running memory; GPU: RTX4060Ti.

[0326] The software platform of the simulation experiment platform of the present invention is: Windows 11 operating system and PyCharm, PyTorch2.01, and CUDA 12.4.

[0327] 2. Simulation steps

[0328] Step 1: According to the set network parameters, the constructed reward value based on artificial potential field is introduced into the UAV model predictive control network based on Actor-Critic and artificial potential field. The constructed quadrotor UAV dynamics model is taken as the control object, and training is carried out in a variety of static multi-obstacle environments and dynamic multi-obstacle environments. After each round of training, the training weight file is output; by comparing with the previous training results, the training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file.

[0329] Step 2: Input the optimal weight file and UAV dynamics model obtained in step 1 into the constructed static multi-obstacle environment / dynamic multi-obstacle environment for experimental testing to obtain the UAV flight path map, reward value r, path length L, path maximum distortion LS, path global distortion GS, whether a collision occurs, and threat index and other results.

[0330] 3. Simulation content and results analysis

[0331] The simulation experiment of the present invention verifies the effectiveness of the UAV model predictive control method based on Actor-Critic and artificial potential field in 6 static multi-obstacle environments and 6 dynamic multi-obstacle environments.

[0332] (1) Static multi-obstacle environment experiment

[0333] In the static multi-obstacle environment experiment, the reward (APF-Reward) or the traditional reward (Reward) introduced into the Actor-Critic based UAV model predictive control method was compared to form the present invention (Actor-Critic APF MPC algorithm) and the traditional technology (Actor-Critic MPC algorithm). Due to the randomness of reinforcement learning training, different random number seeds are set for multiple trainings during the experiment. Figure 5 The path planning effect of controlling UAV with traditional technology (Actor-Critic MPC algorithm) in a static multi-obstacle environment.

[0334] See also Figure 6 By introducing the reward value processed by artificial potential field in the present invention (Actor-Critic APF MPC algorithm), the convergence speed of the algorithm can be accelerated, the local optimal solution can be avoided, the learning direction can be guided, and the sparse reward problem can be solved.

[0335] In order to compare the improvement of the present invention (Actor-Critic_APF_MPC algorithm) over the traditional technology (Actor-CriticMPC algorithm), the reward value r, flight path length (L), path maximum distortion LS, path global distortion GS, whether a collision occurs and other indicators are used to evaluate the flight trajectory in the simulation experiment, and the above two algorithms are applied to Figure 5 The indicators in each environment are recorded and the results are recorded in Table 1. Table 1: Comparison of indicators between the present invention (AC APF MPC algorithm) and the traditional technology (AC MPC algorithm) in a static multi-obstacle environment

[0336]

[0337] As shown in Table 1, the UAV using the present invention (Actor-Critic APF MPC algorithm) is significantly superior to the UAV using the traditional technology (Actor-Critic MPC algorithm) in terms of four indicators: reward value r, maximum path distortion LS, global path distortion GS, and collision. The reward value r is a comprehensive indicator for evaluating path performance. Experimental data show that the reward value of the UAV using the present invention (Actor-CriticAPF MPC algorithm) is superior to that of the UAV using the traditional technology (Actor-Critic MPC algorithm) in the above six environments. The maximum path distortion LS and the global path distortion GS represent the global stability and smoothness of the UAV during flight. Experimental data show that the UAV using the present invention (Actor-Critic APF MPC algorithm) is superior to the UAV using the traditional technology (Actor-Critic MPC algorithm) in terms of LS and GS in the above six environments. The L of the drone using the present invention (Actor-Critic APF MPC algorithm) and the drone using the traditional technology (Actor-Critic MPC algorithm) in the above 6 environments are similar, but in Env_2 and Env_4, the drone using the traditional technology (Actor-Critic MPC algorithm) collides with obstacles in order to obtain the shortest path, while the drone using the present invention (Actor-Critic APF MPC algorithm) does not collide with obstacles.

[0338] (2) Dynamic multi-obstacle environment

[0339] In the dynamic multi-obstacle environment, the experiment compared the reward (APF-Reward) and the general reward (Reward) introduced into the Actor-Critic based UAV model predictive control method, forming the present invention (Actor-Critic APF MPC algorithm) and the traditional technology (Actor-Critic MPC algorithm). Due to the randomness of reinforcement learning training, different random number seeds are set for multiple trainings during the experiment. Figure 7 Cumulative reward return curves for 6 dynamic multi-obstacle environments.

[0340] like Figure 7 As shown, both curves have convergence characteristics, which illustrates the effectiveness of the present invention (Actor-Critic APF MPC algorithm) and the traditional technology (Actor-Critic MPC algorithm). The cumulative reward of the present invention (Actor-Critic APF MPC algorithm) in 6 different dynamic multi-obstacle environments is greater than that of the traditional technology (Actor-Critic MPC algorithm), and has higher stability. Figure 8The path planning effect of a UAV using traditional technology (Actor-Critic MPC algorithm) in a dynamic multi-obstacle environment. Fig. 9 The figure shows the path planning effect of the UAV using the Actor-Critic APF MPC algorithm of the present invention in a dynamic multi-obstacle environment.

[0341] In order to compare the improvement of the present invention (Actor-Critic APF MPC algorithm) over the traditional technology (Actor-Critic MPC algorithm) in a dynamic multi-obstacle environment. In the simulation experiment, the reward value r, flight path length L, path maximum distortion LS, path global distortion GS, whether a collision occurs and threat index are used to evaluate the flight trajectory, and the above two algorithms are applied to the above six different static multi-obstacle environments. The indicators in each environment are recorded, and the results are recorded in Table 2.

[0342] Table 2: Comparison of indicators between the present invention (AC APF MPC algorithm) and the traditional technology (AC MPC algorithm) in a dynamic multi-obstacle environment

[0343]

[0344]

[0345] As shown in Table 2, in a dynamic multi-obstacle environment, the UAV using the present invention (Actor-Critic APF MPC algorithm) that introduces artificial potential field processing reward values ​​is significantly superior to the UAV using the traditional technology (Actor-Critic MPC algorithm) using general reward values ​​in five indicators, namely reward value r, flight path length L, path global distortion GS, collision and threat index. The experimental data show that the UAV using the present invention (Actor-Critic APF MPC algorithm) is superior to the UAV using the traditional technology (Actor-Critic MPC algorithm) in r in the above six environments, indicating that the former has better training effect and convergence speed. L represents the efficiency of the UAV flight. The experimental data show that the UAV using the present invention (Actor-Critic APF MPC algorithm) is superior to the UAV using the traditional technology (Actor-Critic MPC algorithm) in r in the six environments. In Env-2, Env-5 and Env-6, the drone using the traditional technology (Actor-Critic MPC algorithm) collides with obstacles, resulting in a threat index of ∞, and LS and GS are better than the drone using the present invention (Actor-Critic APF MPC algorithm). The threat index is a measure of the degree to which the drone is threatened by obstacles. The experimental data show that the threat index of the drone using the present invention (Actor-Critic APF MPC algorithm) is smaller than that of the drone using the traditional technology (Actor-Critic MPC algorithm).

[0346] The present invention proposes a model predictive control method for unmanned aerial vehicles based on actor-critic and artificial potential field. By introducing the reward processed by the artificial potential field into actor-critic, the convergence speed and accuracy of the algorithm are improved. The model predictive control combines the path generated by the artificial potential field as a reference trajectory with the real-time trajectory of the unmanned aerial vehicle as the system input. The short-term optimization of the model predictive control is combined with the exploratory and end-to-end training characteristics of actor-critic. The present invention unifies the advantages of artificial potential field, model-based model predictive control, and model-free actor-critic algorithm. By training in a large number of dynamic multi-obstacle environments, the unmanned aerial vehicle gradually learns to plan the path in advance in the constructed artificial potential field and avoid obstacles. In the experimental stage, the unmanned aerial vehicle using the present invention (actor-critic APF MPC algorithm) and the unmanned aerial vehicle using the traditional technology (actor-critic MPC algorithm) are compared, and the trained model is tested in multiple static multi-obstacle environments and dynamic multi-obstacle environments. Experiments show that the UAV using the present invention (Actor-Critic APF MPC algorithm) has good performance in terms of cumulative reward r, flight path length L, path global distortion GS, collision and threat index, etc. This strongly proves that after the present invention is applied to the UAV, it has good online planning capabilities, can perceive the existence of obstacles in advance, and avoid them in time.

Claims

1. A model predictive control method for unmanned aerial vehicle based on Actor-Critic and artificial potential field, characterized in that: The following steps are involved: Step 1: Construct multiple static multi-obstacle environments and dynamic multi-obstacle environments; Step 2: Construct the dynamic model of the quadrotor drone; Step 3: Construct a reward value based on an artificial potential field; Step 4: Construct a network of UAV model predictive control algorithm based on Actor-Critic and artificial potential field; Step 5: Set network parameters; Step 6: According to the network parameters set in step 5, the reward value based on the artificial potential field constructed in step 3 is input into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in step 4, and the dynamic model of the quadcopter UAV constructed in step 2 is used as the control object. Training is performed in a variety of static multi-obstacle environments and dynamic multi-obstacle environments constructed in step 1. After each round of training, the training weight file is output; by comparing with the previous training results, the training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file; Step 7: Input the optimal weight file obtained in step 6 and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in step 4 into the various static multi-obstacle environments and dynamic multi-obstacle environments constructed in step 1 for testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs, and threat index results.

2. The method for predictive control of unmanned aerial vehicle model based on Actor-Critic and artificial potential field according to claim 1, characterized in that: The implementation method of step 1 includes: Step 101: Set up multiple static multi-obstacle environments: set the center position (x, y, z) and radius R of the sphere, the center position (x, y, z), radius R and height H of the bottom circle of the cylinder, and the center position (x, y, z), radius R and height H of the bottom circle of the cone; Step 102: Set up a variety of dynamic multi-obstacle environments: set the center position (x, y, z) of the sphere, the radius R, the obstacle movement trajectory and the speed v.

3. The method for predictive control of unmanned aerial vehicle model based on Actor-Critic and artificial potential field according to claim 1, characterized in that: The specific implementation method of step 2 is: Define geographic coordinate system O I (X I ,Y I ,Z I ) and the drone body coordinate system O B (X B ,Y B ,Z B ), the drone is in the geographic coordinate system O I The position vector in is defined as ζ: The velocity vector ν of the drone is defined as: The attitude vector of the drone in space is defined as η: The rolling angle φ in formula (3) represents the rolling angle of the UAV along the geographic coordinate system X I The rotation angle of the drone along the geographic coordinate system Y I The rotation angle of the drone along the geographic coordinate system Z I The rotation angle of , the Euler angle angular velocity vector ω is: In formula (4), p, q, and r are the X-axis coordinates of the UAV in the body coordinate system. B , Y B , Z B The angular velocity of the axis rotation, the control quantity state space is defined as: U=[U1 U2 U3 U4] T (5) In formula (5), It is the sum of the thrusts generated by the four rotors, representing the total thrust of the quadcopter along the coordinate axis Z of the aircraft system. B displacement movement; is the rolling moment, which represents the rolling moment of the quadcopter along the X B a rotational tumbling motion; is the pitch moment, representing the Y direction of the quadrotor drone B Rotational pitch motion; is the yaw moment, representing the yaw moment along Z B Rotational yaw motion; K is the lift coefficient; K d is the anti-torque coefficient; l is the distance from the center of each rotor of the quadrotor to its center of mass; ω1, ω2, ω3, ω4 are the rotation speeds of the four rotors respectively; The state space and state vector of the dynamic model of the quadrotor drone are: In formula (6), the state vector of the quadrotor drone is Therefore, the state of the quadrotor drone at time k is X(k)∈X; k x , k y , k z It is obtained by decomposing the air resistance coefficient of the UAV in the inertial reference frame; k φ , k θ , k ψ They are the total air resistance torque coefficient at X I , Y I , Z I Component of the axis; I x , I y , I z For quad-rotor drone along the aircraft system X B , Y B , Z B Moment of inertia of the shaft; J RP is the moment of inertia of the propeller along its rotation axis; Ω = -ω1 + ω2 - ω3 + ω4 is the algebraic sum of the motor speeds.

4. The method for predictive control of unmanned aerial vehicle model based on Actor-Critic and artificial potential field according to claim 1, characterized in that: The specific implementation method of step three is: The reward function sets the following four rewards: (1) reward r1 for the target to attract the drone; (2) reward r2 for the obstacle to repel the drone; (3) collision penalty r3; (4) reward r4 for reaching the target; (1) Rewards for attracting drones The attraction of the target to the UAV is proportional to the distance between the control object and the target point; as shown in formula (9), the reward r1 caused by the attraction of the target to the UAV in the artificial potential field is simplified as follows: D attract =|q-q g | (7) D 0_g =|q0-q g | (8) In formula (7), D attract is the distance between the UAV and the target point, q is the position of the UAV, and q g is the target position; D in formula (8) 0_g is the distance between the initial position and the target position, q0 is the starting position; in formula (9), represents the distance between the UAV and the target at time k; represents the distance between the UAV and the target at time k+1; represents the change in the position of the drone between time k and k+1. When When , it means the drone position has not changed. When , it means the drone is moving away from the target; (2) Rewards for obstacles that repel drones When the drone enters the repulsive force range D of the obstacle, the reward r2 caused by the obstacle's repulsion of the drone is set by simplifying the artificial potential field; D repel =|q-q o | (10) When the UAV is in the repulsive field range D of the obstacle, the obstacle will produce a certain repulsive force on the UAV, making the UAV stay away from the obstacle and preventing the UAV from colliding with the obstacle. In formula (10), D repel is the distance between the UAV and the obstacle, q is the position of the UAV, and q o is the obstacle position; in formula (11), represents the distance between the UAV and the obstacle at time k; represents the distance between the drone and the obstacle at time k+1; It represents the distance change of the drone caused by obstacle rejection between time k and k+1. When When , it means the drone position has not changed. When the UAV is moving away from obstacles; (3) Collision Penalty When the drone collides with an obstacle, the drone will receive a collision penalty r3; In formula (12), the collision reward is equal to the distance between the drone and the obstacle center at the next moment: Subtract the obstacle radius R divided by the obstacle radius R minus 1; (4) Rewards for reaching the goal When the drone reaches the target point, the environment will give the drone a positive reward r4; In formula (13), when the distance between the drone and the target point is equal to 0, the reward given to the drone by the environment is 3; when the distance between the drone and the target point is not equal to 0, the reward given to the drone by the environment is 0; The reward of artificial potential field processing is defined as: APF-Reward=λ1·r1+λ2·r2+λ3·r3+λ4·r4 (14) In formula (14), APF-Reward is the reward value processed by the artificial potential field; λ1, λ2, λ3, and λ4 represent the weights of the four rewards respectively.

5. The method for predictive control of unmanned aerial vehicle model based on Actor-Critic and artificial potential field according to claim 1, characterized in that: The specific implementation method of step 4 is: The purpose of UAV control is trajectory tracking control, so the output of the UAV system is defined as the position ζ of the UAV in the geographic coordinate system and the heading angle ψ during the flight. Therefore, the output Y of the UAV system is: In formula (15), the output of the UAV system includes the position x, y, z of the UAV in the geographic coordinate system and the heading angle ψ during the flight; The UAV flight target state R(k) is set in a static multi-obstacle environment / dynamic multi-obstacle environment. The UAV measures the UAV state X(k) in real time during flight, and feeds the measured UAV state X(k) back to the input end to subtract it from the target state R(k) to obtain the deviation E(k)=R(k)-X(k). The deviation E(k) generates the output control quantity U(k) through the model predictive control module to drive the UAV to the target position to eliminate the deviation E(k). According to equations (1), (2), (3) and (4), the state of the UAV system is defined as Define the control output of the UAV system as U = [U1 U2 U3 U4] T , define the output of the UAV system as Y = [xyz ψ] T ; Discretize the continuous state space model, and the discretized linear state space model of the UAV is: X(k+1|k)=A·X(k)+B·U(k) (16) In formula (16), k represents the current time step, A represents the weight coefficient matrix of the system state; B represents the weight coefficient matrix of the controller output; X(k) represents the UAV system state at time k; U(k) represents the controller output at time k; X(k+1k) represents the UAV system state at time k+1 predicted at time k; that is, the system state quantity X in the prediction interval N is obtained: k,N and the control quantity U k,N They are: X k,N =[X(k|k) X(k+1|k)…X(k+N|k)] T (17) YOU k,N =[U(k|k) U(k+1|k)…U(k+N-1|k)] T (18) In formula (17), X(k|k) is the state of the UAV at time k, X(k+1|k) is the state of the UAV at time k+1 predicted at time k, and X(k+N|k) is the state of the UAV at time k+N predicted at time k; In formula (18), U(k|k) is the control output of the UAV at time k, U(k+1|k) is the control output of the UAV at time k+1 predicted at time k, and U(k+N-1|k) is the control output of the UAV at time k+N-1 predicted at time k; Substituting equation (17) and equation (18) into equation (16), we can obtain the predicted state X k,N for: X k,N =M·X(k)+C·U k,N (19) In formula (19), X(k) is the state of the UAV at time k, M = [IAA 2 …A N ] T , Since the main control goal of the UAV is to track the expected trajectory, the expected trajectory R in the prediction interval N is defined. k,N for: R k,N =[R(k|k) R(k+1|k)…R(k+N|k)] (20) In formula (20), R(k+1) is the expected state at time k+1, R(k+2) is the expected state at time k+2, and R(k+N) is the expected state at time k+N; Therefore, the trajectory error X in the prediction interval N is k,N for: E k,N =[[R(k|k)-X(k|k)] [R(k+1|k)-X(k+1|k)]…[R(k+N|k)-X(k+N|k)]] (21) In formula (21), [R(k|k)-X(k|k)] is the state error of the UAV system at time k, [R(k+1|k)-X(k+1|k)] is the state error of the UAV system at time k+1, and [R(k+N|k)-X(k+N|k)] is the state error of the UAV system at time k+N; Define the objective function J: In formula (22), Q and R are the weight coefficient matrices of error and control quantity respectively; Substituting equation (18) and equation (21) into equation (22), we can simplify equation (22) to obtain: J=E(k) T ·G·E(k)+2E(k) T ·F·U k,N +U k,N T ·H·U k,N (24) In formula (23), In formula (24), Using the Actor-Critic algorithm, introducing the Actor strategy network and the Critic evaluation network, the UAV control strategy was improved, including: Return G at time k k The expression is: In formula (25), R k+i is the reward at time k+i, and γ is the loss factor; If the evaluation network Critic is an evaluation network of the state value state value, and the input is the state; the loss function calculation formula of the evaluation network Critic adopts the mean square error loss function, that is, the mean of the cumulative square value of the TD error value, and the expression is as follows: In formula (26), L critic is the loss function when the evaluation network Critic is the evaluation network of the state value, V(s k ) is the state s at time k k The corresponding value, V(s k+1 ) is the state s at time k+1 k+1 The corresponding value; If the evaluation network Critic is an evaluation network of action value actionvalue, that is, the input is state and action, then the loss function of the evaluation network Critic is as follows: In formula (27), L critic is the loss function when the evaluation network Critic is the evaluation network of action value actionvalue, Q(s k ,a k ) is the k moment, the controlled object is at s k When taking action k When, the reward is obtained; The optimization goals of the Actor network are as follows: In formula (28), π * represents the optimal strategy. Since the meaning of this formula is to increase the probability of action selection when the TD error value is greater than 0, and to reduce the probability of action selection when the TD error value is less than 0, the goal is to minimize the loss function -L actor ; Therefore, the value function Vπ(s) of the evaluation network Critic has parameter θ q The updated formula is: In formula (29), α1 is the descending step size, is the descending gradient; Therefore, the strategy function of the Actor network is θ (s) Parameter θ π Updated formula: In formula (30), α2 is the descending step size, is the descending gradient; Minimize the objective function J; transform the problem of solving the optimal control quantity into the following quadratic optimization problem: In formula (31), U(k) min and U(k) max are the minimum and maximum control values ​​respectively; the Actor network outputs the real-time optimal F, H weight coefficient matrix, and the evaluation network Critic evaluates the strategy through TDerror; then, solve the quadratic optimization problem and obtain the optimal control value sequence U within the prediction interval N k,N , then select the first item U(k+1|k) in the sequence as the output of the MPC controller at the current time k, and perform rolling optimization and feedback correction to control the quadrotor drone to track the trajectory.

6. The method for predictive control of unmanned aerial vehicle model based on Actor-Critic and artificial potential field according to claim 1, characterized in that: The specific implementation method of step seven is: The calculation of experimental data such as path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs and threat index is as follows: (1) Path length L In formula (32), S k Indicates the distance the drone moves from time k to time k+1, which is equal to the distance between the drone and the target at time k The distance between the drone and the target at time k+1 The absolute value of the difference; In formula (33), the path length L is equal to S k Iterative addition, N is the number of algorithm iterations; (2) Maximum path distortion LS The distortion τ of the UAV flight curve at time k k for: In formula (34), T k is the tangent vector of the UAV flight curve at time k, N k is the normal vector of the UAV flight curve at time k, B k is the cross product of the normal vectors of the UAV flight curve at time k (also called the double normal vector of the hyperbola), B′ k is the derivative of the double normal vector of the UAV flight curve at time k; LS=max{τ 0 ,t 1 ,…,t N } (35) (3) Path global distortion GS The distortion on the path is integrated to obtain the path global distortion measure GS, where τ(s) is the distortion of point s on the path, and a and b are the starting point and end point of the UAV flight path; (4) Whether a collision occurred In formula (37), the distance between the UAV and the obstacle at time k is When the distance between the drone and the obstacle is less than or equal to the obstacle radius R, the drone collides with the obstacle; when the distance between the drone and the obstacle at time k is When it is larger than the obstacle radius R, the drone does not collide with the obstacle; (5) Threat Index In formula (38), when When the drone collides with an obstacle, the threat index is equal to ∞; When the drone is within the obstacle rejection range but no collision occurs, the threat index is equal to ν k is the speed of the drone at time k, is the distance between the UAV and the obstacle center at time k; When the drone does not enter the obstacle rejection range, the threat index is 0.

7. A UAV model predictive control system based on Actor-Critic and artificial potential field based on the method according to any one of claims 1 to 6, characterized in that: include: Environment construction module: used to construct a variety of static multi-obstacle environments and dynamic multi-obstacle environments; Quad-rotor UAV dynamics model construction module: used to build a quad-rotor UAV dynamics model; Artificial potential field-based reward value construction module: used to construct the reward value based on the artificial potential field; Network construction module: used to construct the network of UAV model predictive control method based on Actor-Critic and artificial potential field; Network training module: According to the set network parameters, the reward value based on the artificial potential field is input into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field. The dynamic model of the constructed quadcopter UAV is used as the control object, and training is carried out in a variety of static multi-obstacle environments and dynamic multi-obstacle environments. After each round of training, the training weight file is output; Select the training weight file corresponding to the training result with the highest reward value and the shortest path as the optimal weight file; Test result output module: used to input the optimal weight file and the network of the constructed UAV model predictive control algorithm based on Actor-Critic and artificial potential field into a variety of constructed static multi-obstacle environments and dynamic multi-obstacle environments for testing, and obtain the UAV flight path map, reward value r, path length L, path maximum distortion LS, path global distortion GS, whether a collision occurs and threat index results.

8. A UAV model predictive control device based on Actor-Critic and artificial potential field, characterized in that: include: Memory: a computer program storing a method for predictive control of a UAV model based on Actor-Critic and artificial potential field as described in any one of claims 1 to 6, which is a computer-readable device; Processor: used to implement the UAV model predictive control method based on Actor-Critic and artificial potential field as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the unmanned aerial vehicle model predictive control method based on Actor-Critic and artificial potential field as described in any one of claims 1 to 6.

Images