A predictive control method, system, device, and medium for unmanned aerial vehicles (UAVs) based on Actor-Critic and artificial potential fields.

Abstract

A method, system, device, and medium for UAV model predictive control based on Actor-Critic and artificial potential fields are disclosed. The method involves sequentially constructing networks for various static and dynamic multi-obstacle environments, a quadcopter UAV's dynamic model, a reward value based on the artificial potential field, and a UAV model predictive control algorithm based on Actor-Critic and artificial potential fields. The algorithm model is trained according to the set network parameters, and an optimal weight file is selected. The optimal weight file and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential fields are input into various static and dynamic multi-obstacle environments for testing, yielding UAV flight path diagrams, reward value r, path length L, maximum path distortion LS, global path distortion GS, collision occurrence, and threat index results. The system, device, and medium are used to implement the method. This invention can effectively plan paths and avoid obstacles in a timely manner, possessing advantages such as high real-time performance, strong robustness, and timely obstacle avoidance.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) control technology, and specifically relates to a UAV model predictive control method, system, device, and medium based on Actor-Critic and artificial potential field. Background Technology

[0002] With the advancement of drone technology, drones are increasingly being used in military, civilian, and commercial fields. However, traditional remote control methods or predetermined trajectory control methods often struggle to cope with complex and ever-changing environmental conditions, thus necessitating more intelligent and autonomous control methods. Combining reinforcement learning with control algorithms for drone flight trajectory planning and predictive tracking is one of the current development trends in the field of drone flight control.

[0003] Reinforcement learning algorithms have seen significant development and application in fields such as robot control, autonomous driving, aircraft control, and power system scheduling. The Actor-Critic algorithm is an important algorithm in reinforcement learning, typically used to solve problems in continuous action spaces, and is suitable for UAV flight trajectory planning, prediction, and tracking in complex dynamic environments. With further optimization of reinforcement learning-based control algorithms and the expansion of their application areas, these algorithms will continue to play a vital role in the future.

[0004] Currently, model predictive control methods for quadrotor unmanned aerial vehicles (UAVs) based on reinforcement learning have shown significant effectiveness in UAV flight control. In (Barros GM, Colombini E L. Using soft actor-critic for low-level UAV control[J]. arXiv preprint arXiv:2010.02293,2020.), model-free reinforcement learning has been successfully used to control UAVs without prior knowledge of the robot model. A framework for training the Soft Actor-Critic (SAC) algorithm was proposed to perform low-altitude control of quadrotor aircraft in target missions. In (Chatterjee D, Roy R, Sengupta A. Comparison of Reinforcement Learning controller with a classical controller for an UAV[C] / / 2023 Second International Conference on Electrical, Electronics, Information and Communication Technologies (ICEEICT). IEEE, 2023:01-05.), the performance of Model Predictive Control (MPC) and a controller trained on RL for unmanned aerial vehicles (UAVs) is compared. Simulation results on a real nonlinear UAV model show that the performance of the RL controller is close to or even better than that of MPC. In (Romero A, Song Y, Scaramuzza D. Actor-critic model predictive control[J]. arXiv preprint arXiv:2306.09852,2023.), a differentiable MPC is added to the Actor-Critic network framework of reinforcement learning, combining the short-term predictive optimization capability of MPC with the exploratory and end-to-end training characteristics of reinforcement learning, thus unifying the advantages of model-based control and reinforcement learning.

[0005] The defects and shortcomings of existing related inventions:

[0006] 1. Due to modeling errors and the complexity of the working environment, quadcopter drone controllers cannot use only one set of control parameters to control quadcopter drones in different scenarios.

[0007] 2. Model predictive control requires a large number of predictive models and control strategies, resulting in high computational complexity and poor real-time performance.

[0008] 3. In reinforcement learning, the general reward function only provides a positive reward when the drone reaches the target. This method is inefficient in obtaining reward feedback to help the drone explore the environment. Therefore, the convergence speed of network updates is slow, and it is difficult for the drone to learn the optimal strategy from exploration. Summary of the Invention

[0009] To overcome the shortcomings of the prior art, the present invention aims to provide a UAV model predictive control method, system, device, and medium based on Actor-Critic and artificial potential field.

[0010] First, an artificial potential field is introduced to enhance the UAV's environmental perception and path planning capabilities. The reward in the Actor-Critic algorithm is redesigned by modeling the environment as a virtual potential field. Second, the short-term prediction optimization of Model Predictive Control (MPC) is combined with the exploratory and end-to-end training characteristics of the Actor-Critic algorithm. The resulting policy effectively manages short-term policies through MPC based on the Actor network and effectively manages long-term predictions through the Critic evaluation network, unifying the advantages of the artificial potential field with model-based MPC and model-free Actor-Critic algorithms. In the artificial potential field, the UAV, as the controlled object, gradually learns to plan a safe, short-distance, and short-flight-time path in advance based on the virtual potential field through interaction with the environment. Finally, experiments were conducted in multiple static and dynamic multi-obstacle environments. This invention demonstrates good performance in terms of cumulative reward r, flight path length L, global path distortion GS, collision probability, and threat index, indicating excellent performance in path planning and predictive tracking.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A predictive control method for unmanned aerial vehicles (UAVs) based on Actor-Critic and artificial potential fields includes the following steps:

[0013] Step 1: Construct various static and dynamic multi-obstacle environments;

[0014] Step 2: Construct a dynamic model of the quadcopter drone;

[0015] Step 3: Construct a reward value based on an artificial potential field;

[0016] Step 4: Construct a network for a UAV model predictive control algorithm based on Actor-Critic and artificial potential fields;

[0017] Step 5: Set network parameters;

[0018] Step Six: Based on the network parameters set in Step Five, input the reward value based on the artificial potential field constructed in Step Three into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in Step Four. Use the dynamic model of the quadcopter UAV constructed in Step Two as the controlled object, and train it in various static and dynamic multi-obstacle environments constructed in Step One. Output the training weight file after each round of training. By comparing with the previous training results, select the training weight file corresponding to the training result with the highest reward value and the shortest path as the optimal weight file.

[0019] Step 7: Input the optimal weight file obtained in Step 6 and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in Step 4 into various static and dynamic multi-obstacle environments constructed in Step 1 for testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs and threat index results.

[0020] The implementation method of step one includes:

[0021] Step 101: Set up multiple static multi-obstacle environments: Set the center position (x,y,z) and radius R of the sphere, the center position (x,y,z), radius R and height H of the bottom circle of the cylinder, and the center position (x,y,z), radius R and height H of the bottom circle of the cone;

[0022] Step 102: Set up multiple dynamic multi-obstacle environments: Set the center position (x, y, z) of the sphere, radius R, and the trajectory and velocity v of the obstacles.

[0023] The specific implementation method of step two is as follows:

[0024] Define geographic coordinate system O I (X I ,Y I Z I ) and the UAV body coordinate system O B (X B ,Y B Z B The drone is in the geographic coordinate system O I The position vector in the vector is defined as ζ:

[0025]

[0026] The velocity vector ν of the UAV is defined as:

[0027]

[0028] The attitude vector of the UAV in space is defined as η:

[0029]

[0030] In equation (3), the roll angle φ represents the distance the UAV travels along the geographic coordinate system X. I The rotation angle; the pitch angle θ represents the UAV's rotation along the geographic coordinate system Y. I The rotation angle; the heading angle ψ represents the UAV's rotation along the geographic coordinate system Z. I The rotation angle, Euler angle, and angular velocity vector ω are:

[0031]

[0032] In equation (4), p, q, and r represent the distances along the X-axis of the UAV in the body coordinate system, respectively. B Y B Z B The angular velocity of the axis rotation, the control variable state space is defined as:

[0033] U = [U1 U2 U3 U4] T (5)

[0034] In equation (5), The sum of the thrust generated by the four rotors represents the thrust of the quadcopter UAV along the Z-axis of the machine coordinate system. B Displacement motion; The rolling torque represents the torque of the quadcopter drone along the X-axis. B Rotating and tumbling motion; The pitch moment represents the pitch moment of the quadcopter UAV along the Y-axis. B Rotational pitch motion; The yaw moment represents the torque along the Z-axis. B Rotational yaw motion; K is the lift coefficient; K d ω1, ω2, ω3, and ω4 are the anti-torque coefficients; l is the distance from the center of each rotor to its center of mass; ω1, ω2, ω3, and ω4 are the rotational speeds of the four rotors, respectively.

[0035] The state space and state vector of the dynamic model of the quadcopter UAV are as follows:

[0036]

[0037] In equation (6), the state vector of the quadcopter UAV is Therefore, the state of the quadcopter UAV at time k is X(k)∈X; x k y k z It is obtained by decomposing the air drag coefficient of the UAV in an inertial reference frame; k φ k θk ψ The total air resistance torque coefficients are respectively in X I Y I Z I Components of the axis; I x I y I z For quadcopter UAVs along the aircraft system X B Y B Z B Moment of inertia of the shaft; J RP ω is the moment of inertia of the propeller along its axis of rotation; Ω = -ω1 + ω2 - ω3 + ω4 is the algebraic sum of the motor speeds.

[0038] The specific implementation method of step three is as follows:

[0039] The reward function is set with the following four rewards: (1) reward r1 for attracting the drone to the target; (2) reward r2 for repelling the drone from the obstacle; (3) collision penalty r3; (4) reward r4 for reaching the target.

[0040] (1) Rewards for attracting drones to the target

[0041] The attraction of the target to the UAV is proportional to the distance between the controlled object and the target point; as shown in equation (9), the reward r1 caused by the attraction of the target to the UAV in the artificial potential field is simplified:

[0042] D attract =|qq g | (7)

[0043] D 0_g =|q0-q g | (8)

[0044]

[0045] In equation (7), D attract Let q be the distance between the drone and the target point, and q be the drone's position. g The target location; D in equation (8) 0_g Let q0 be the distance between the initial position and the target position, and q0 be the starting position; in equation (9), This represents the distance between the UAV and the target at time k; This represents the distance between the drone and the target at time k+1; This represents the change in the drone's position between time k and k+1. At that time, it indicates that the drone is approaching the target. When, it indicates that the drone's position has not changed. At that time, it indicates that the drone is moving away from the target;

[0046] (2) Rewards for drones that are repelled by obstacles

[0047] When the drone enters the repulsive force range D of the obstacle, the reward r2 caused by the obstacle's repulsion of the drone is set by simplifying the artificial potential field;

[0048] D repel =|qq o | (10)

[0049]

[0050] When the drone is within the repulsive field range D of the obstacle, the obstacle exerts a certain repulsive force on the drone, causing the drone to move away from the obstacle and preventing the drone from colliding with the obstacle; in equation (10), D repel Let q be the distance between the drone and the obstacle, and q be the drone's position. o The location of the obstacle; in equation (11), This represents the distance between the drone and the obstacle at time k; This represents the distance between the drone and the obstacle at time k+1; This represents the change in distance caused by the drone's repulsion from obstacles between time k and k+1. When, it indicates that the drone is approaching the obstacle, when When, it indicates that the drone's position has not changed. This indicates that the drone is moving away from the obstacle;

[0051] (3) Collision penalty

[0052] When the drone collides with an obstacle, it will receive a collision penalty of r3.

[0053]

[0054] In equation (12), the collision reward is equal to the distance between the UAV and the center of the obstacle at the next moment. Subtract the obstacle radius R and divide by the obstacle radius R minus 1;

[0055] (4) Rewards for reaching the goal

[0056] When the drone reaches the target point, the environment will give the drone a positive reward of r4;

[0057]

[0058] In equation (13), when the distance between the drone and the target point is 0, the environment rewards the drone with 3; when the distance between the drone and the target point is not 0, the environment rewards the drone with 0.

[0059] The reward for artificial potential field processing is defined as follows:

[0060] APF-Reward=λ1·r1+λ2·r2+λ3·r3+λ4·r4 (14)

[0061] In equation (14), APF-Reward is the reward value of artificial potential field processing; λ1, λ2, λ3, and λ4 represent the weights of the four rewards, respectively.

[0062] The specific implementation method of step four is as follows:

[0063] The purpose of UAV control is trajectory tracking control; therefore, the output of the UAV system is defined as the UAV's position ζ in the geographic coordinate system and its heading angle ψ during flight. Thus, the UAV system output Y is:

[0064]

[0065] In Equation (15), the output of the UAV system includes the position x, y, z of the UAV in the geographic coordinate system and the heading angle ψ during flight.

[0066] In a static multi-obstacle environment / dynamic multi-obstacle environment, the target state R(k) of the UAV is set. During flight, the UAV measures its state X(k) in real time and feeds it back to the input. The measured UAV state X(k) is subtracted from the target state R(k) to obtain the deviation E(k) = R(k) - X(k). The deviation E(k) is used by the model predictive control module to generate the output control quantity U(k) to drive the UAV to the target position to eliminate the deviation E(k). According to equations (1), (2), (3) and (4), the state of the UAV system is defined as follows: Define the control output of the unmanned aerial vehicle (UAV) system as U = [U1 U2 U3 U4] T The output of the unmanned aerial vehicle (UAV) system is defined as Y = [xy zψ]. T Discretize the continuous state-space model; the discretized linear state-space model of the UAV is as follows:

[0067]

[0068] In equation (16), k represents the current time step, A represents the weight coefficient matrix of the system state, B represents the weight coefficient matrix of the controller output, X(k) represents the UAV system state at time k, U(k) represents the controller output at time k, and X(k+1|k) represents the UAV system state predicted at time k+1. Thus, the system state quantity X within the prediction interval N is obtained. k,N With control quantity U k,N They are respectively:

[0069] Xk,N =[X(k|k) X(k+1|k) … X(k+N|k)] T (17)

[0070] U k,N =[U(k|k) U(k+1|k) … U(k+N-1|k)] T (18)

[0071] In Equation (17), X(k|k) is the UAV state quantity at time k, X(k+1|k) is the UAV state quantity predicted at time k+1, and X(k+N|k) is the UAV state quantity predicted at time k+N; in Equation (18), U(k|k) is the UAV control output quantity at time k, U(k+1|k) is the UAV control output quantity predicted at time k+1, and U(k+N-1|k) is the UAV control output quantity predicted at time k+N-1.

[0072] Substituting equations (17) and (18) into equation (16), we obtain the predicted state X. k,N for:

[0073] X k,N =M·X(k)+C·U k,N (19)

[0074] In equation (19), X(k) is the UAV state variable at time k, and M = [IAA 2 …A N ] T ,

[0075]

[0076] Since the primary control objective of the UAV is to track a desired trajectory, the desired trajectory R within the prediction interval N is defined. k,N for:

[0077] R k,N =[R(k|k) R(k+1|k) … R(k+N|k)] (20)

[0078] In equation (20), R(k+1) is the expected state quantity at time k+1, R(k+2) is the expected state quantity at time k+2, and R(k+N) is the expected state quantity at time k+N.

[0079] Therefore, the trajectory error X within the prediction interval N k,N for:

[0080] E k,N=[[R(k|k)-X(k|k)] [R(k+1|k)-X(k+1|k)] … [R(k+N|k)-X(k+N|k)]] (21)

[0081] In equation (21), [R(k|k)-X(k|k)] is the state error of the UAV system at time k, [R(k+1|k)-X(k+1|k)] is the state error of the UAV system at time k+1, and [R(k+N|k)-X(k+N|k)] is the state error of the UAV system at time k+N.

[0082] Define the objective function J:

[0083]

[0084] In equation (22), Q and R are the weighting coefficient matrices of the error and the control quantity, respectively;

[0085] Substituting equations (18) and (21) into equation (22), and simplifying equation (22), we get:

[0086]

[0087] J = E(k) T ·G·E(k)+2E(k) T ·F·U k,N +U k,N T ·H·U k,N (twenty four)

[0088] In equation (23), In equation (24),

[0089] The Actor-Critic algorithm, by introducing an Actor policy network and a Critic evaluation network, improves the UAV control strategy, specifically including:

[0090] Return at time k k The expression is:

[0091]

[0092] In equation (25), R k+i Let γ be the reward at time k+i, and γ be the loss factor.

[0093] If the evaluation network Critic is a state value evaluation network, the input is the state; the loss function of the evaluation network Critic is calculated using the mean squared error loss function, which is the mean of the cumulative squared values ​​of the TD error values, and the expression is as follows:

[0094]

[0095] In equation (26), L critic Let V(s) be the loss function when the evaluation network Critic is an evaluation network with state values. k ) represents the state s at time k. k The corresponding value, V(s) k+1 ) represents the state s at time k+1. k+1 The corresponding value;

[0096] If the evaluation network Critic is an evaluation network that evaluates actions based on their values ​​(action values), i.e., the input is state and action, then the loss function of the evaluation network Critic is as follows:

[0097]

[0098] In equation (27), L critic Let Q(s) be the loss function when the evaluation network Critic is an evaluation network for action values. k ,a k At time k, the controlled object is in state s. k At that time, take action a k The reward received at that time;

[0099] The optimization objectives of the Actor network are as follows:

[0100]

[0101] In equation (28), π * Representing the optimal strategy, since this formula means that when the TD error value is greater than 0, the probability of choosing this action is increased, and when the TD error value is less than 0, the objective is to minimize the loss function -L. actor ;

[0102] Therefore, the value function Vπ(s) for evaluating the network Critic has parameters θ. q The updated formula is:

[0103]

[0104] In equation (29), α1 is the descent step size. For descent gradient;

[0105] Therefore, the policy function π of the Actor network θ (s) parameter θ π Updated formula:

[0106]

[0107] In equation (30), α2 is the descent step size. For descent gradient;

[0108] Minimize the objective function J; the problem of finding the optimal control quantity is transformed into the following quadratic optimization problem:

[0109]

[0110] In equation (31), U(k) min and U(k) max These represent the minimum and maximum control values, respectively. The Actor network outputs the real-time optimal F and H weight coefficient matrices, and the Critic network evaluates the policy performance using TDerror. Then, this quadratic optimization problem is solved to obtain the optimal control sequence U within the prediction interval N. k,N Then, the first item U(k+1|k) in the sequence is selected as the output of the MPC controller at the current time k, and rolling optimization and feedback correction are performed to control the quadcopter UAV to perform trajectory tracking.

[0111] The specific implementation method of step seven is as follows:

[0112] The experimental data, including path length L, maximum path twist LS, global path twist GS, collision occurrence, and threat index, are calculated as follows:

[0113] (1) Path length L

[0114]

[0115] In equation (32), S k This represents the distance the drone travels from time k to k+1, which is equal to the distance between the drone and the target at time k. Distance between the UAV and the target at time k+1 The absolute value of the difference;

[0116]

[0117] In equation (33), the path length L is equal to S. k The iterations are summed, where N is the number of algorithm iterations;

[0118] (2) Path maximum twist LS

[0119] The distortion τ of the drone's flight curve at time k k for:

[0120]

[0121] In equation (34), T k Let N be the tangent vector of the UAV's flight curve at time k. k Let B be the normal vector of the UAV's flight curve at time k. k Let B′ be the cross product of the normal vectors of the UAV's flight curve at time k (also known as the double normal vectors of the hyperbola). k Let be the derivative of the two normal vectors of the UAV's flight curve at time k;

[0122] LS=max{τ 0 ,τ 1 ,...,τ N} (35)

[0123] (3) Path global distortion GS

[0124]

[0125] Integrating the twist degree on the path, we obtain the global twist metric GS of the path. In Equation (36), τ(s) is the twist degree of point s on the path, and a and b are the start and end points of the UAV flight path.

[0126] (4) Whether a collision occurred

[0127]

[0128] In equation (37), the distance between the UAV and the obstacle at time k is... When the distance between the drone and the obstacle is less than or equal to the radius R of the obstacle, a collision occurs; when the distance between the drone and the obstacle at time k... When the radius R of the obstacle is greater than the obstacle's radius, the drone does not collide with the obstacle.

[0129] (5) Threat Index

[0130]

[0131] In equation (38), when When a drone collides with an obstacle, the threat level equals ∞; when When a drone is within the obstacle's repulsion range but does not collide with it, the threat index is equal to... ν k Let k be the speed of the drone at time k. Let k be the distance between the drone and the center of the obstacle; when The threat index is 0 when the drone is not within the range of the obstacle's repulsion effect.

[0132] This invention also provides a UAV model prediction and control system based on Actor-Critic and artificial potential field, comprising:

[0133] Environment building module: used to build various static and dynamic multi-obstacle environments;

[0134] The quadcopter drone dynamics model building module is used to build the dynamics model of a quadcopter drone.

[0135] Artificial potential field-based reward value construction module: used to construct reward values ​​based on artificial potential fields;

[0136] Network building module: Used to build networks for predictive control methods of UAV models based on Actor-Critic and artificial potential fields;

[0137] Network training module: Based on the set network parameters, the constructed reward value based on the artificial potential field is input into the network of the constructed UAV model predictive control algorithm based on Actor-Critic and artificial potential field. The constructed dynamic model of the quadcopter UAV is used as the controlled object. Training is carried out in various constructed static multi-obstacle environments and dynamic multi-obstacle environments. After each round of training, the training weight file is output. The training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file.

[0138] Test result output module: This module is used to input the optimal weight file and the constructed network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field into various static and dynamic multi-obstacle environments for testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurred, and threat index results.

[0139] This invention also provides a UAV model prediction and control device based on Actor-Critic and artificial potential field, comprising:

[0140] Memory: A computer program that stores the above-mentioned UAV model predictive control method based on Actor-Critic and artificial potential field, and is a computer-readable device;

[0141] Processor: Used to implement the aforementioned UAV model predictive control method based on Actor-Critic and artificial potential field when executing the computer program.

[0142] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned UAV model predictive control method based on Actor-Critic and artificial potential field.

[0143] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0144] 1. This invention, within the Actor-Critic algorithm framework, innovatively utilizes the evaluation network Critic to evaluate the merits of the current control strategy (i.e., the weight coefficient matrix of the MPC) in real time, thereby controlling the Actor network to dynamically adjust and output the optimal MPC weight coefficient matrices F and H. Since both the Actor network and the evaluation network Critic are implemented through deep neural networks, they possess powerful nonlinear mapping and generalization capabilities, enabling them to handle complex input data (such as UAV status information, environmental perception data, etc.) and output high-precision control commands. This end-to-end optimization approach not only simplifies the complex optimization process in traditional MPC algorithms but also significantly improves the system's computational efficiency and real-time performance.

[0145] 2. This invention deeply optimizes and reconstructs the core component of Model Predictive Control (MPC)—the objective function (Equation (22)). Specifically, it significantly simplifies the complex model described by Equation (22). This innovative improvement aims to solve the problems of high computational complexity and huge computational load faced by high-speed dynamic systems such as UAVs. It not only reduces the number of numerical iterations in the calculation process, but also reduces the demand for computing resources, thereby significantly improving the system's computational speed.

[0146] 3. This invention integrates the artificial potential field method into the Actor-Critic algorithm framework to optimize the reward mechanism, resulting in the APF-Reward processed by the artificial potential field method. This innovation effectively accelerates the convergence speed of the Actor-Critic network update, improves the learning efficiency and decision-making ability of UAVs in constructed static multi-obstacle environments and dynamic multi-obstacle environments, and enables UAVs to quickly transition from the initial exploration stage to the advanced stage of mastering and executing the optimal flight strategy.

[0147] In summary, this invention integrates the rewards of artificial potential field processing into the Actor-Critic algorithm with UAV model predictive control methods, thus unifying the advantages of artificial potential field, model-based predictive control, and model-free Actor-Critic algorithms. By training and predicting the network using multiple designed static and dynamic multi-obstacle environments, it can effectively plan paths and avoid obstacles in a timely manner, exhibiting advantages such as high real-time performance, strong robustness, and timely obstacle avoidance. Attached Figure Description

[0148] Figure 1 This is a flowchart of the algorithm according to an embodiment of the present invention.

[0149] Figure 2 This is the coordinate definition for drone modeling in this embodiment of the invention.

[0150] Figure 3 This is a force analysis diagram of the improved artificial potential field method according to an embodiment of the present invention.

[0151] Figure 4 This is a block diagram of the model prediction control system according to an embodiment of the present invention.

[0152] Figure 5 This is a simulation image of a drone operating in a static multi-obstacle environment using traditional technology (Actor-Critic MPC algorithm). The cyan dot represents the starting point, the red dot represents the target point, and the red trajectory represents the drone's flight path. Figure 5 (a) is a flight result diagram of the UAV using traditional technology (Actor-Critic MPC algorithm) in static multi-obstacle environment No. 1. Figure 5 (b) is a flight result diagram of the UAV in static multi-obstacle environment No. 2 using traditional technology (Actor-Critic MPC algorithm). Figure 5 (c) is a flight result diagram of the UAV in static multi-obstacle environment No. 3 using traditional technology (Actor-Critic MPC algorithm). Figure 5 (d) shows the flight results of the UAV in static multi-obstacle environment No. 4 using traditional technology (Actor-Critic MPC algorithm). Figure 5 (e) is a flight result diagram of the UAV using traditional technology (Actor-Critic MPC algorithm) in static multi-obstacle environment No. 5. Figure 5 (f) is a flight result diagram of the UAV using the traditional technology (Actor-Critic MPC algorithm) in static multi-obstacle environment No. 6.

[0153] Figure 6This is a simulation image of a UAV using the present invention (Actor-Critic APF MPC algorithm) in a static multi-obstacle environment. The cyan dot represents the starting point, the red dot represents the target point, and the red trajectory represents the UAV's flight path. Figure 6 (a) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in static multi-obstacle environment No. 1. Figure 6 (b) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in static multi-obstacle environment No. 2. Figure 6 (c) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in static multi-obstacle environment No. 3. Figure 6 (d) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in static multi-obstacle environment No. 4. Figure 6 (e) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in static multi-obstacle environment No. 5. Figure 6 (f) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in static multi-obstacle environment No. 6.

[0154] Figure 7 This is a cumulative reward curve of the policy network under six dynamic multi-obstacle environments using an embodiment of the present invention (Actor-Critic APF MPC algorithm). The present invention (Actor-Critic APF MPC algorithm) is represented by a blue curve, while the traditional technique (Actor-Critic MPC algorithm) is represented by an orange curve. Figure 7 (a) is a comparison curve of the cumulative reward during training in the No. 1 dynamic multi-obstacle environment, using the traditional technique (Actor-Critic MPC algorithm) and the present invention (Actor-Critic APF MPC algorithm), respectively. Figure 7 (b) is a comparison curve of the cumulative reward during training in the No. 2 dynamic multi-obstacle environment, using the traditional technique (Actor-Critic MPC algorithm) and the present invention (Actor-Critic APF MPC algorithm), respectively. Figure 7 (c) is a comparison curve of the cumulative reward during training in dynamic multi-obstacle environment No. 3 for UAVs using the traditional technique (Actor-Critic MPC algorithm) and the present invention (Actor-Critic APFMPC algorithm), respectively. Figure 7(d) is a comparison curve of the cumulative reward during training in the No. 4 dynamic multi-obstacle environment, using the traditional technique (Actor-Critic MPC algorithm) and the present invention (Actor-Critic APF MPC algorithm), respectively. Figure 7 (e) is a comparison curve of the cumulative reward during training in the No. 5 dynamic multi-obstacle environment, using the traditional technique (Actor-Critic MPC algorithm) and the present invention (Actor-Critic APF MPC algorithm), respectively. Figure 7 (f) is a comparison curve of the cumulative reward during training in dynamic multi-obstacle environment No. 6 for UAVs using the traditional technology (Actor-Critic MPC algorithm) and the present invention (Actor-Critic APF MPC algorithm).

[0155] Figure 8 This is a simulation image of a drone operating in a dynamic multi-obstacle environment using traditional technology (Actor-Critic MPC algorithm). The green triangle represents the drone, the cyan dot represents the starting point, the red dot represents the target point, the red trajectory represents the drone's flight path, and the black trajectory represents the obstacle's movement path. Figure 8 (a) is a flight result diagram of the UAV using traditional technology (Actor-Critic MPC algorithm) in dynamic multi-obstacle environment No. 1. Figure 8 (b) is a flight result diagram of the UAV using traditional technology (Actor-Critic MPC algorithm) in dynamic multi-obstacle environment No. 2. Figure 8 (c) is a flight result diagram of the UAV using traditional technology (Actor-CriticMPC algorithm) in dynamic multi-obstacle environment No. 3. Figure 8 (d) is a flight result diagram of the UAV using traditional technology (Actor-Critic MPC algorithm) in dynamic multi-obstacle environment No. 4. Figure 8 (e) is a flight result diagram of the UAV using traditional technology (Actor-Critic MPC algorithm) in dynamic multi-obstacle environment No. 5. Figure 8 (f) is a flight result diagram of the UAV using the traditional technology (Actor-Critic MPC algorithm) in dynamic multi-obstacle environment No. 6.

[0156] Figure 9 This is a simulation image of a UAV using the Actor-Critic APF MPC algorithm in a dynamic multi-obstacle environment. The green triangle represents the UAV, the cyan dot represents the starting point, the red dot represents the target point, the red trajectory represents the UAV's flight path, and the black trajectory represents the obstacle's movement trajectory. Figure 9(a) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in dynamic multi-obstacle environment No. 1. Figure 9 (b) is a flight result diagram of the UAV using the present invention (Actor-CriticAPF MPC algorithm) in dynamic multi-obstacle environment No. 2. Figure 9 (c) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in dynamic multi-obstacle environment No. 3. Figure 9 (d) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in dynamic multi-obstacle environment No. 4. Figure 9 (e) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in dynamic multi-obstacle environment No. 5. Figure 9 (f) is a flight result diagram of the UAV using the present invention (Actor-Critic APF MPC algorithm) in dynamic multi-obstacle environment No. 6. Detailed Implementation

[0157] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0158] To address the problems of existing technologies, this invention leverages the advantages of reinforcement learning and model predictive control, such as high learning efficiency and excellent performance, to propose a UAV model predictive control method based on Actor-Critic and artificial potential fields, and makes the following improvements:

[0159] 1. In the Actor-Critic algorithm, the Critic network evaluates the effectiveness of the current strategy, enabling the Actor network to output the real-time optimal MPC weight coefficient matrix, which controls the UAV to fly along the predicted trajectory.

[0160] 2. Simplify the objective function of model predictive control, reduce the amount of computation and computational complexity, and improve the system's computing speed to achieve efficient real-time control of UAVs.

[0161] 3. By introducing an artificial potential field method to process the reward in the Actor-Critic algorithm, the convergence speed of the Actor-Critic network update is improved, enabling the UAV to quickly learn the optimal policy from exploration. Specifically:

[0162] A predictive control method for unmanned aerial vehicles (UAVs) based on Actor-Critic and artificial potential fields includes the following steps:

[0163] Step 1: Construct various static and dynamic multi-obstacle environments to provide a testing environment for UAV simulation;

[0164] Step 2: Construct a dynamic model of the quadcopter drone;

[0165] Step 3: Construct a reward value based on an artificial potential field;

[0166] Step 4: Construct a network for a UAV model predictive control algorithm based on Actor-Critic and artificial potential fields;

[0167] Step 5: Set network parameters;

[0168] Step Six: Based on the network parameters set in Step Five, input the reward value based on the artificial potential field constructed in Step Three into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in Step Four. Use the dynamic model of the quadcopter UAV constructed in Step Two as the controlled object, and train it in various static and dynamic multi-obstacle environments constructed in Step One. Output the training weight file after each round of training. By comparing with the previous training results, select the training weight file corresponding to the training result with the highest reward value and the shortest path as the optimal weight file.

[0169] Step 7: Input the optimal weight file obtained in Step 6 and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in Step 4 into various static and dynamic multi-obstacle environments constructed in Step 1 for experimental testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs and threat index results.

[0170] The algorithm flowchart of this invention is as follows: Figure 1 As shown, an artificial potential field method is introduced into the constructed static / dynamic multi-obstacle environment to address the path planning problem of UAVs flying in complex environments, enabling effective obstacle avoidance and path planning. The path generated by the artificial potential field is combined with the UAV's real-time trajectory as a reference trajectory and used as input to the control system. The concept of artificial potential field is used to set rewards to improve the system's convergence speed and accuracy. The UAV explores and learns in the constructed static / dynamic multi-obstacle environment. The control system obtains the state X(k) at time k through a state observer, combines it with the desired state R(k) to obtain E(k), and uses it as input to the model predictive controller. X(k) is input to the evaluation network Critic and Actor network of the Actor-Critic algorithm. The evaluation network Critic outputs TDerror to evaluate the current policy to improve the policy and drive learning. The Actor network outputs the weight coefficient matrix F of the model predictive controller. k H kCalculate the control quantity sequence U within the prediction interval N. k,N Then, the first item U(k+1|k) in the sequence is selected as the control variable at time k+1, and rolling optimization and feedback correction are performed to control the UAV to perform path planning and predictive tracking, which effectively improves the robustness and adaptability of the system.

[0171] The implementation method of step one includes:

[0172] Step 101: Set up multiple static multi-obstacle environments: Set the center position (x,y,z) and radius R of the sphere, the center position (x,y,z), radius R and height H of the bottom circle of the cylinder, and the center position (x,y,z), radius R and height H of the bottom circle of the cone;

[0173] Step 102: Set up multiple dynamic multi-obstacle environments: Set the center position (x, y, z) of the sphere, radius R, and the trajectory and velocity v of the obstacles.

[0174] The specific implementation method of step two is as follows:

[0175] like Figure 2 As shown, the geographic coordinate system O is defined. I (X I ,Y I Z I ) and the UAV body coordinate system O B (X B ,Y B Z B The drone is in the geographic coordinate system O I The position vector in the vector is defined as ζ:

[0176]

[0177] The velocity vector ν of the UAV is defined as:

[0178]

[0179] The attitude vector of the UAV in space is defined as η:

[0180]

[0181] In equation (3), the roll angle φ represents the distance the UAV travels along the geographic coordinate system X. I The rotation angle; the pitch angle θ represents the UAV's rotation along the geographic coordinate system Y. I The rotation angle; the heading angle ψ represents the UAV's rotation along the geographic coordinate system Z. I The rotation angle, Euler angle, and angular velocity vector ω are:

[0182]

[0183] In equation (4), p, q, and r represent the distances along the X-axis of the UAV in the body coordinate system, respectively. B Y B Z B The angular velocity of the axis rotation, the control variable state space is defined as:

[0184] U = [U1 U2 U3 U4] T (5)

[0185] In equation (5), The sum of the thrust generated by the four rotors represents the thrust of the quadcopter UAV along the Z-axis of the machine coordinate system. B Displacement motion; The rolling torque represents the torque of the quadcopter drone along the X-axis. B Rotating and tumbling motion; The pitch moment represents the pitch moment of the quadcopter UAV along the Y-axis. B Rotational pitch motion; The yaw moment represents the torque along the Z-axis. B Rotational yaw motion; K is the lift coefficient; K d ω1, ω2, ω3, and ω4 are the anti-torque coefficients; l is the distance from the center of each rotor to its center of mass; ω1, ω2, ω3, and ω4 are the rotational speeds of the four rotors, respectively.

[0186] The state space and state vector of the dynamic model of the quadcopter UAV are as follows:

[0187]

[0188] In equation (6), the state vector of the quadcopter UAV is Therefore, the state of the quadcopter UAV at time k is X(k)∈X; x k y k z It is obtained by decomposing the air drag coefficient of the UAV in an inertial reference frame; k φ k θ k ψ The total air resistance torque coefficients are respectively in X I Y I Z I Components of the axis; I x I y I z For quadcopter UAVs along the aircraft system X B Y B Z B Moment of inertia of the shaft; J RP ω is the moment of inertia of the propeller along its axis of rotation; Ω = -ω1 + ω2 - ω3 + ω4 is the algebraic sum of the motor speeds.

[0189] The specific implementation method of step three is as follows:

[0190] This invention addresses the problems of local optima and target unreachability by improving the obstacle repulsive potential field function. Its force analysis is as follows: Figure 3 As shown, an adjustment factor is added to the obstacle repulsion field model of the traditional artificial potential field method, so that the repulsive and attractive forces only decrease to zero when the UAV reaches the target point, thereby solving the problems of local optima and target unreachability.

[0191] In this invention, the reward of the Actor-Critic algorithm is the feedback signal obtained by the UAV in exploring the environment, which is used to evaluate the actions taken by the controlled object. A simple method is to set a general reward, that is, the UAV can only get a positive reward when the task is completed. However, this method is inefficient in collecting useful empirical data to help the controlled object explore the environment. Therefore, the convergence speed of network updates is slow, and the UAV cannot learn the optimal strategy from exploration. This invention introduces the idea of ​​artificial potential field method (obstacles and targets exert repulsive and attractive effects on the controlled object, respectively) into the design of reward function. The reward function setting of this invention considers the following four rewards: (1) reward r1 for the target attracting the UAV; (2) reward r2 for the obstacle repelling the UAV; (3) collision penalty r3; (4) reward r4 for reaching the target;

[0192] (1) Rewards for attracting drones to the target

[0193] The attraction of the target to the UAV is proportional to the distance between the controlled object and the target point; as shown in equation (9), the reward r1 caused by the attraction of the target to the UAV in the artificial potential field is simplified:

[0194] D attract =|qq g | (7)

[0195] D 0_g =|q0-q g | (8)

[0196]

[0197] In equation (7), D attract Let q be the distance between the drone and the target point, and q be the drone's position. g The target location; D in equation (8) 0_g Let q0 be the distance between the initial position and the target position, and q0 be the starting position; in equation (9), This represents the distance between the UAV and the target at time k; This represents the distance between the drone and the target at time k+1; This represents the change in the drone's position between time k and k+1. At that time, it indicates that the drone is approaching the target. When, it indicates that the drone's position has not changed. At that time, it indicates that the drone is moving away from the target;

[0198] (2) Rewards for drones that are repelled by obstacles

[0199] When the drone enters the repulsive force range D of the obstacle, similar to the reward set for attracting the drone to the target, this invention sets the reward r2 caused by the obstacle's repulsion of the drone by simplifying the artificial potential field;

[0200] D repel =|qq o | (10)

[0201]

[0202] When the drone is within the repulsive field range D of the obstacle, the obstacle exerts a certain repulsive force on the drone, causing the drone to move away from the obstacle and preventing the drone from colliding with the obstacle; in equation (10), D repel Let q be the distance between the drone and the obstacle, and q be the drone's position. o The location of the obstacle; in equation (11), This represents the distance between the drone and the obstacle at time k; This represents the distance between the drone and the obstacle at time k+1; This represents the change in distance caused by the drone's repulsion from obstacles between time k and k+1. When, it indicates that the drone is approaching the obstacle, when When, it indicates that the drone's position has not changed. This indicates that the drone is moving away from the obstacle;

[0203] (3) Collision penalty

[0204] When the drone collides with an obstacle, it will receive a collision penalty of r3.

[0205]

[0206] In equation (12), the collision reward is equal to the distance between the UAV and the center of the obstacle at the next moment. Subtract the obstacle radius R and divide by the obstacle radius R minus 1;

[0207] (4) Rewards for reaching the goal

[0208] When the drone reaches the target point, the environment will give the drone a positive reward of r4;

[0209]

[0210] In equation (13), when the distance between the drone and the target point is 0, the environment rewards the drone with 3; when the distance between the drone and the target point is not 0, the environment rewards the drone with 0.

[0211] In summary, the reward for artificial potential field processing can be defined as:

[0212] APF-Reward=λ1·r1+λ2·r2+λ3·r3+λ4·r4 (14)

[0213] In equation (14), APF-Reward is the reward value of artificial potential field processing; λ1, λ2, λ3, and λ4 represent the weights of the four rewards, respectively.

[0214] The specific implementation method of step four is as follows:

[0215] The purpose of UAV control is trajectory tracking control; therefore, the output of the UAV system is defined as the UAV's position ζ in the geographic coordinate system and its heading angle ψ during flight. Thus, the UAV system output Y is:

[0216]

[0217] In Equation (15), the output of the UAV system includes the position x, y, z of the UAV in the geographic coordinate system and the heading angle ψ during flight.

[0218] like Figure 4 As shown, in a static multi-obstacle environment / dynamic multi-obstacle environment, the UAV flight target state R(k) is set. During flight, the UAV measures its state X(k) in real time and feeds it back to the input. The measured UAV state X(k) is subtracted from the target state R(k) to obtain the deviation E(k) = R(k) - X(k). The deviation E(k) is used by the model predictive control module to generate the output control quantity U(k) to drive the UAV to the target position to eliminate the deviation E(k). According to equations (1), (2), (3) and (4), the state of the UAV system is defined as follows: Define the control output of the unmanned aerial vehicle (UAV) system as U = [U1 U2 U3 U4] T The output of the unmanned aerial vehicle (UAV) system is defined as Y = [xy zψ]. T Model predictive control is a digital control method; therefore, for continuous state-space models, they first need to be discretized. The discretized linear state-space model of an unmanned aerial vehicle (UAV) is as follows:

[0219] X(k+1|k)=A·X(k)+B·U(k) (16)

[0220] In equation (16), k represents the current time step, A represents the weight coefficient matrix of the system state, B represents the weight coefficient matrix of the controller output, X(k) represents the UAV system state at time k, U(k) represents the controller output at time k, and X(k+1|k) represents the UAV system state predicted at time k+1. Thus, the system state quantity X within the prediction interval N can be obtained. k,N With control quantity U k,N They are respectively:

[0221] X k,N =[X(k|k) X(k+1|k) … X(k+N|k)] T (17)

[0222] U k,N =[U(k|k) U(k+1|k) … U(k+N-1|k)] T (18)

[0223] In Equation (17), X(k|k) is the UAV state quantity at time k, X(k+1|k) is the UAV state quantity predicted at time k+1, and X(k+N|k) is the UAV state quantity predicted at time k+N; in Equation (18), U(k|k) is the UAV control output quantity at time k, U(k+1|k) is the UAV control output quantity predicted at time k+1, and U(k+N-1|k) is the UAV control output quantity predicted at time k+N-1.

[0224] Substituting equations (17) and (18) into equation (16), we can obtain the predicted state X. k,N for:

[0225] X k,N =M·X(k)+C·U k,N (19)

[0226] In equation (19), X(k) is the UAV state variable at time k, and M = [IAA 2 …A N ] T ,

[0227]

[0228] Since the primary control objective of the UAV is to track a desired trajectory, the desired trajectory R within the prediction interval N is defined. k,N for:

[0229] R k,N =[R(k|k) R(k+1|k) … R(k+N|k)] (20)

[0230] In equation (20), R(k+1) is the expected state quantity at time k+1, R(k+2) is the expected state quantity at time k+2, and R(k+N) is the expected state quantity at time k+N.

[0231] Therefore, the trajectory error X within the prediction interval N k,N for:

[0232] E k,N =[[R(k|k)-X(k|k)] [R(k+1|k)-X(k+1|k)] … [R(k+N|k)-X(k+N|k)]] (21)

[0233] In equation (21), [R(k|k)-X(k|k)] is the state error of the UAV system at time k, [R(k+1|k)-X(k+1|k)] is the state error of the UAV system at time k+1, and [R(k+N|k)-X(k+N|k)] is the state error of the UAV system at time k+N.

[0234] Define the objective function J:

[0235]

[0236] In equation (22), Q and R are the weighting coefficient matrices of error and control quantity, respectively. In simulation and practical applications, adjusting Q and R can make the UAV focus on trajectory tracking or save energy, laying a solid foundation for achieving good experimental results.

[0237] Substituting equations (18) and (21) into equation (22), and simplifying equation (22), we get:

[0238]

[0239] J = E(k) T ·G·E(k)+2E(k) T ·F·U k,N +U k,N T ·H·U k,N (twenty four)

[0240] In equation (23), In equation (24),

[0241] This invention employs a UAV control algorithm that combines the Actor-Critic algorithm, the artificial potential field method, and the model predictive control method to perform obstacle avoidance and trajectory planning for UAVs. The aim is to significantly improve the obstacle avoidance capability and trajectory planning efficiency of UAVs in constructed static multi-obstacle environments and dynamic multi-obstacle environments. By integrating these advanced methods, the adaptability of UAVs in the face of dynamic environmental changes can be enhanced.

[0242] The Actor-Critic algorithm is an existing algorithm that combines the ideas of Policy Base and Value Base. The Actor is implemented using the policy gradient algorithm, while the Critic is implemented using temporal differencing. The Actor is the policy function π. θ (s), typically implemented using a neural network, takes the current state as input and outputs an action. The training objective of this network is to maximize the expected cumulative reward. Critic is the value function V. π (s) This network can estimate the value function of the current policy, that is, it can evaluate the quality of the Actor (policy function). The original Actor (policy gradient method) uses the expected cumulative reward as the basis for training, which means that π can only be updated at the end of the round. θ The parameters of (s).

[0243] Return at time k k The expression is:

[0244]

[0245] In equation (25), R k+i Let γ be the reward at time k+i, and γ be the loss factor.

[0246] If the evaluation network Critic is a state value evaluation network, the input is the state; the loss function of the evaluation network Critic is calculated using the mean squared error loss function, which is the mean of the cumulative squared values ​​of the TD error values, and the expression is as follows:

[0247]

[0248] In equation (26), L critic Let V(s) be the loss function when the evaluation network Critic is an evaluation network with state values. k ) represents the state s at time k. k The corresponding value, V(s) k+1 ) represents the state s at time k+1. k+1 The corresponding value;

[0249] If the evaluation network Critic is an evaluation network that evaluates actions based on their values ​​(action values), i.e., the input is state and action, then the loss function of the evaluation network Critic is as follows:

[0250]

[0251] In equation (27), L critic Let Q(s) be the loss function when the evaluation network Critic is an evaluation network for action values. k ,a k At time k, the controlled object is in state s. k At that time, take action a k The reward received at that time;

[0252] The optimization objective of the Actor network can be as follows:

[0253]

[0254] In equation (28), π * Representing the optimal strategy, since this formula means that when the TD error value is greater than 0, the probability of choosing this action is increased, and when the TD error value is less than 0, the objective is to minimize the loss function -L. actor ;

[0255] Therefore, the value function Vπ(s) for evaluating the network Critic has parameters θ. q The updated formula is:

[0256]

[0257] In equation (29), α1 is the descent step size. For descent gradient;

[0258] Therefore, the policy function π of the Actor network θ (s) parameter θ π Updated formula:

[0259]

[0260] In equation (30), α2 is the descent step size. For descent gradient;

[0261] To control the drone, the objective function J needs to be minimized; therefore, the problem of finding the optimal control quantity is transformed into the following quadratic optimization problem:

[0262]

[0263] In equation (31), U(k) min and U(k)max These represent the minimum and maximum control values, respectively. The Actor network outputs the real-time optimal F and H weight coefficient matrices, and the Critic network evaluates the policy performance using TDerror. Then, this quadratic optimization problem is solved to obtain the optimal control sequence U within the prediction interval N. k,N Then, the first item U(k+1|k) in the sequence is selected as the output of the MPC controller at the current time k, and rolling optimization and feedback correction are performed to control the quadcopter UAV to perform trajectory tracking.

[0264] In step five, the following settings are configured: training epochs: episode = 1000, step size = 0.2, batch size ≥ 24, learning rate ≤ 3 × 10⁻⁶. -4 .

[0265] The specific implementation method of step seven is as follows:

[0266] The calculation of experimental data such as path length L, maximum path twist LS, global path twist GS, collision occurrence, and threat index in the simulation experiment is as follows:

[0267] (1) Path length L

[0268]

[0269] In equation (32), S k This represents the distance the drone travels from time k to k+1, which is equal to the distance between the drone and the target at time k. Distance between the UAV and the target at time k+1 The absolute value of the difference;

[0270]

[0271] In equation (33), the path length L is equal to S. k The iterations are summed, where N is the number of algorithm iterations.

[0272] (2) Path maximum twist LS

[0273] The distortion τ of the drone's flight curve at time k k for:

[0274]

[0275] In equation (34), T k Let N be the tangent vector of the UAV's flight curve at time k. k Let B be the normal vector of the UAV's flight curve at time k. k Let B′ be the cross product of the normal vectors of the UAV's flight curve at time k (also known as the double normal vectors of the hyperbola). kLet be the derivative of the two normal vectors of the UAV's flight curve at time k;

[0276] LS=max{τ 0 ,τ 1 ,...,τ N} (35)

[0277] (3) Path global distortion GS

[0278]

[0279] Integrating the twist degree on the path, we obtain the global twist metric GS of the path. In Equation (36), τ(s) is the twist degree of point s on the path, and a and b are the start and end points of the UAV flight path.

[0280] (4) Whether a collision occurred

[0281]

[0282] In equation (37), the distance between the UAV and the obstacle at time k is... When the distance between the drone and the obstacle is less than or equal to the radius R of the obstacle, a collision occurs; when the distance between the drone and the obstacle at time k... When the radius R of the obstacle is greater than the obstacle's radius, the drone does not collide with the obstacle.

[0283] (5) Threat Index

[0284]

[0285] In equation (38), when When a drone collides with an obstacle, the threat level equals ∞; when When a drone is within the obstacle's repulsion range but does not collide with it, the threat index is equal to... ν k Let k be the speed of the drone at time k. Let k be the distance between the drone and the center of the obstacle; when The threat index is 0 when the drone is not within the range of the obstacle's repulsion effect.

[0286] Example

[0287] like Figure 1 As shown, the algorithm in this invention—a UAV model predictive control method based on Actor-Critic and artificial potential field—includes the following steps:

[0288] Step 1: The UAV introduces an artificial potential field method in various static and dynamic multi-obstacle environments to handle path planning problems in complex environments, enabling effective obstacle avoidance and path planning;

[0289] Step 2: Construct a dynamic model of the quadcopter UAV's flight in various static and dynamic multi-obstacle environments, and define the geographic coordinate system O. I (X I ,Y I Z I ), UAV body coordinate system O B (X B ,Y B Z B ), UAV in geographic coordinate system O I The vectors in the diagram include the position vector ζ, the velocity vector ν of the UAV, the attitude vector η of the UAV in space, the Euler angle angular rate vector ω, the state variable X, the control variable U, and the output variable Y.

[0290] Step 3: Constructing a reward value based on an artificial potential field. This invention introduces the idea of ​​the artificial potential field method (obstacles and targets exert repulsive and attractive effects on the controlled object, respectively) into the design of the reward function. The reward function setting of this invention considers the following four rewards: (1) reward r1 for the target attracting the drone; (2) reward r2 for the obstacle repelling the drone; (3) collision penalty r3; (4) reward r4 for reaching the target;

[0291] Step 4: This invention first defines the objective function J = E(k) for the model predictive control method. T ·G·E(k)+2E(k) T ·F·U k,N +U k,N T ·H·U k,N Construct the Actor network and Critic network of the Actor-Critic algorithm. The Actor is the policy function π. θ (s), where the input is the current state and the output is an action; Critic is the value function V. π (s), the network estimates the value function of the current policy and evaluates the quality of the Actor network; in order to control the drone, it is necessary to minimize the objective function J, so the problem of solving the optimal control quantity is transformed into the following quadratic optimization problem:

[0292]

[0293] Step 5: The network parameters of this invention are set as follows: training epochs: episode = 1000, step size = 0.2, batch size ≥ 24, learning rate ≤ 3 × 10⁻⁶. -4 ;

[0294] Step Six: Based on the network parameters set in Step Five, input the reward value based on the artificial potential field constructed in Step Three into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in Step Four. Use the dynamic model of the quadcopter UAV constructed in Step Two as the controlled object, and train it in various static and dynamic multi-obstacle environments constructed in Step One. Output the training weight file after each round of training. By comparing with the previous training results, select the training weight file corresponding to the training result with the highest reward value and the shortest path as the optimal weight file.

[0295] Step 7: Input the optimal weight file obtained in Step 6 and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in Step 4 into various static and dynamic multi-obstacle environments constructed in Step 1 for experimental testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs and threat index results.

[0296] like Figure 2 As shown, the present invention provides a method for defining the coordinate system of a quadcopter unmanned aerial vehicle.

[0297] The quadcopter unmanned aerial vehicle (UAV) is a nonlinear, multivariable, and strongly coupled underactuated control system, making direct dynamic modeling of it quite complex. To simplify the model, the following assumptions are made regarding the research object:

[0298] (1) A quadcopter is a uniformly symmetrical rigid body;

[0299] (2) The mass and moment of inertia of the quadcopter remain unchanged;

[0300] (3) The geometric center of a quadcopter coincides with its center of gravity;

[0301] (4) Quadrotor aircraft are only subject to gravity and propeller thrust.

[0302] Based on the Newton-Euler method, the dynamic characteristics of the UAV are analyzed, and an accurate UAV model is established to understand and analyze the system characteristics of the UAV, which also serves as a simulation model for simulation experiments. Secondly, the dynamic model is reasonably simplified to serve as a predictive model for model predictive control methods.

[0303] like Figure 3 As shown, the present invention provides an improved method for artificial potential fields.

[0304] (1) Gravitational potential field

[0305] The gravitational potential field is mainly related to the distance between the drone and the target point. The greater the distance, the greater the potential energy experienced by the drone; the smaller the distance, the smaller the potential energy experienced by the drone. Therefore, the function of the gravitational potential field is:

[0306]

[0307] In equation (40), η is the proportional gain coefficient, and ρ(q,q) g Let q be a vector representing the position q of the UAV and the position q of the target point. g Euclidean distance between | qq g |, where the vector direction is from the drone's position to the target point's position. The corresponding gravitational force F attract (q) represents the negative gradient of the gravitational field, and denoted by the gravitational potential field function U. attract The direction of the fastest change in (q).

[0308]

[0309] (2) Repulsive potential field

[0310] An improved obstacle repulsive potential field function is used to address the problems of local optima and target unreachability. Its force analysis is as follows: Figure 3 As shown, an adjustment factor ρ is added to the obstacle repulsion field model of the traditional artificial potential field method. n (q,q g This ensures that the repulsive and attractive forces only decrease to zero when the drone reaches the target point, thus solving the problems of local optima and target unreachability.

[0311] The improved repulsive potential field function is:

[0312]

[0313] In equation (42), k is a direct proportionality coefficient, and ρ(q,q) o ) is a vector, its direction is from the obstacle to the drone, and its magnitude is the Euclidean distance between the drone and the obstacle. o |; D is a constant representing the maximum range of influence of the obstacle on the drone; n is an optional positive integer.

[0314] The corresponding repulsive force is the negative gradient force of the repulsive potential field:

[0315]

[0316] In equation (43), F repel1 The direction is that the obstacle is pointing at the drone; F repel2The direction is that the drone is pointing towards the target point. The improved repulsive field function incorporates the distance between the drone and the target point. This reduces the gravitational and repulsive forces acting on the drone to a certain extent as it flies towards the target. Only when the drone reaches the target point will the gravitational and repulsive forces simultaneously decrease to zero, meaning the target point becomes the point of minimum potential energy. This solves the problems of local optima and target unreachability.

[0317] (3) Resultant potential field

[0318] Based on the gravitational and repulsive field functions defined above, the composite field of the entire operating space can be obtained. The magnitude of the resultant potential field of the UAV is the sum of the repulsive and gravitational potential fields acting on the UAV. Therefore, the total function of the resultant potential field is:

[0319] U(q)=U attract (q)+U repel (q) (44)

[0320] The net force is

[0321]

[0322] The direction of the resultant force determines the flight direction of the drone, and the magnitude of the resultant force determines the flight acceleration of the drone.

[0323] The effects of the present invention will be further illustrated below with simulation experiments:

[0324] 1. Simulation Experiment Conditions

[0325] The hardware platform for the simulation experiment of this invention is as follows: CPU: Intel I5-13490F, with a main frequency of 2.5GHz and 16G of RAM; GPU: RTX4060Ti.

[0326] The software platform of the simulation experiment platform of this invention is: Windows 11 operating system and PyCharm, PyTorch 2.01, CUDA 12.4.

[0327] 2. Simulation Steps

[0328] Step 1: Based on the set network parameters, the constructed reward value based on the artificial potential field is introduced into the UAV model prediction and control network based on Actor-Critic and artificial potential field. The constructed quadcopter UAV dynamic model is used as the controlled object. Training is carried out in various static and dynamic multi-obstacle environments. After each training round, the training weight file is output. By comparing with the previous training results, the training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file.

[0329] Step 2: Input the optimal weight file and UAV dynamics model obtained in Step 1 into various static multi-obstacle environments / dynamic multi-obstacle environments for experimental testing, and obtain results such as UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs and threat index.

[0330] 3. Simulation content and result analysis

[0331] The simulation experiments of this invention verify the effectiveness of the UAV model predictive control method based on Actor-Critic and artificial potential field in six static multi-obstacle environments and six dynamic multi-obstacle environments.

[0332] (1) Static multi-obstacle environment experiment

[0333] In a static multi-obstacle environment experiment, the proposed Actor-Critic APF MPC algorithm and the traditional Actor-Critic MPC algorithm were compared in the Actor-Critic-based UAV model predictive control method, which incorporates artificial potential field processing as a reward (APF-Reward) or a traditional reward. Due to the randomness of reinforcement learning training, different random number seeds were set for multiple training sessions during the experiment. Figure 5 This study evaluates the path planning performance of UAVs controlled by traditional techniques (Actor-Critic MPC algorithm) in static multi-obstacle environments.

[0334] See Figure 6 By introducing an artificial potential field to process the reward value in this invention (Actor-CriticAPF MPC algorithm), the convergence speed of the algorithm can be accelerated, local optima can be avoided, the learning direction can be guided, and the sparse reward problem can be solved.

[0335] To compare the improvements of this invention (Actor-Critic_APF_MPC algorithm) over the traditional technique (Actor-CriticMPC algorithm), simulation experiments were conducted using metrics such as reward value r, flight path length (L), maximum path distortion LS, global path distortion GS, and whether a collision occurred to evaluate the flight trajectory. The two algorithms were then applied to... Figure 5 Six different static multi-obstacle environments were tested. The metrics for each environment were recorded, and the results are shown in Table 1. Table 1: Comparison of metrics between the present invention (AC APF MPC algorithm) and the traditional technique (AC MPC algorithm) in static multi-obstacle environments.

[0336]

[0337] As shown in Table 1, the UAV using the present invention (Actor-Critic APF MPC algorithm) significantly outperforms the UAV using the traditional technology (Actor-Critic MPC algorithm) in four metrics: reward value r, maximum path distortion LS, global path distortion GS, and collision detection. The reward value r is a comprehensive indicator of path performance. Experimental data shows that the UAV using the present invention (Actor-Critic APF MPC algorithm) outperforms the UAV using the traditional technology (Actor-Critic MPC algorithm) in all six environments. Maximum path distortion LS and global path distortion GS represent the global stability and smoothness of the UAV's flight. Experimental data shows that the UAV using the present invention (Actor-Critic APF MPC algorithm) outperforms the UAV using the traditional technology (Actor-Critic MPC algorithm) in both LS and GS in all six environments. The drone using the present invention (Actor-CriticAPF MPC algorithm) and the drone using the traditional technology (Actor-Critic MPC algorithm) have similar L values ​​in the above 6 environments. However, in Env_2 and Env_4, the drone using the traditional technology (Actor-Critic MPC algorithm) collides with obstacles in order to obtain the shortest path, while the drone using the present invention (Actor-CriticAPF MPC algorithm) does not collide with obstacles.

[0338] (2) Dynamic multi-obstacle environment

[0339] Experiments were conducted in a dynamic multi-obstacle environment to compare the reward (APF-Reward) introduced by the Actor-Critic model predictive control method for unmanned aerial vehicles with a general reward, resulting in the present invention (Actor-Critic APF MPC algorithm) and the traditional technique (Actor-Critic MPC algorithm). Due to the randomness of reinforcement learning training, different random number seeds were set for multiple training sessions during the experiments. Figure 7 The cumulative reward curves are shown for six dynamic multi-obstacle environments.

[0340] like Figure 7 As shown, both curves exhibit convergence characteristics, demonstrating the effectiveness of the present invention (Actor-CriticAPF MPC algorithm) and the traditional technique (Actor-Critic MPC algorithm). The cumulative reward of the present invention (Actor-CriticAPF MPC algorithm) in six different dynamic multi-obstacle environments is greater than that of the traditional technique (Actor-Critic MPC algorithm), and it also has higher stability. Figure 8The path planning performance of a UAV using traditional technology (Actor-Critic MPC algorithm) in a dynamic multi-obstacle environment is shown. Figure 9 The path planning performance of the UAV using the present invention (Actor-Critic APF MPC algorithm) in a dynamic multi-obstacle environment is shown.

[0341] To compare the improvements of the present invention (Actor-Critic APF MPC algorithm) over the traditional technique (Actor-Critic MPC algorithm) in dynamic multi-obstacle environments, simulation experiments were conducted. Indicators such as reward value r, flight path length L, maximum path distortion LS, global path distortion GS, collision occurrence, and threat index were used to evaluate the flight trajectory. The two algorithms were applied to six different static multi-obstacle environments. The indicators for each environment were recorded, and the results are shown in Table 2.

[0342] Table 2: Comparison of performance indicators between the present invention (AC APF MPC algorithm) and the traditional technology (AC MPC algorithm) in dynamic multi-obstacle environments.

[0343]

[0344]

[0345] As shown in Table 2, in dynamic multi-obstacle environments, the UAV using the present invention (Actor-Critic APF MPC algorithm) that incorporates an artificial potential field to process the reward value significantly outperforms the UAV using the traditional technique (Actor-Critic MPC algorithm) in five indicators: reward value r, flight path length L, global path distortion GS, collision and threat index. Experimental data shows that the UAV using the present invention (Actor-Critic APF MPC algorithm) outperforms the UAV using the traditional technique (Actor-Critic MPC algorithm) in all six environments, indicating that the former has better training effect and convergence speed. L represents the efficiency of UAV flight. The experimental data shows that the UAV using the present invention (Actor-Critic APF MPC algorithm) outperforms the UAV using the traditional technique (Actor-Critic MPC algorithm) in all six environments. In Env-2, Env-5, and Env-6, drones using the traditional technology (Actor-Critic MPC algorithm) collided with obstacles, resulting in a threat index of ∞. LS and GS were superior to drones using the present invention (Actor-Critic APF MPC algorithm). The threat index measures the degree of threat a drone faces from obstacles. Experimental data shows that the threat index of drones using the present invention (Actor-Critic APF MPC algorithm) is lower than that of drones using the traditional technology (Actor-Critic MPC algorithm).

[0346] This invention proposes a model predictive control method for unmanned aerial vehicles (UAVs) based on Actor-Critic and artificial potential fields. By introducing the reward from the artificial potential field processing into Actor-Critic, the convergence speed and accuracy of the algorithm are improved. Model predictive control combines the path generated by the artificial potential field as a reference trajectory with the UAV's real-time trajectory as system input. This invention combines the short-term optimization of model predictive control with the exploratory and end-to-end training characteristics of Actor-Critic, thus unifying the advantages of artificial potential fields, model-based predictive control, and model-free Actor-Critic algorithms. Through training in numerous dynamic multi-obstacle environments, the UAV gradually learns to plan its path in advance within the constructed artificial potential field, avoiding obstacles. In the experimental phase, UAVs using this invention (Actor-Critic APF MPC algorithm) and those using traditional techniques (Actor-Critic MPC algorithm) were compared, and the trained model was tested in multiple static and dynamic multi-obstacle environments. Experiments show that the UAV using the present invention (Actor-CriticAPF MPC algorithm) performs well in terms of cumulative reward r, flight path length L, global path distortion GS, collision detection, and threat index. This strongly demonstrates that the present invention, when applied to UAVs, has good online planning capabilities, can detect obstacles in advance, and avoid them in time.

Claims

1. A predictive control method for unmanned aerial vehicles (UAVs) based on Actor-Critic and artificial potential fields, characterized in that, Includes the following steps: Step 1: Construct various static and dynamic multi-obstacle environments; Step 2: Construct a dynamic model of the quadcopter drone; Define geographic coordinate system O I (X I ,Y I Z I ) and the UAV body coordinate system O B (X B ,Y B Z B The drone is in the geographic coordinate system O I The position vector in the vector is defined as ζ: The velocity vector ν of the UAV is defined as: The attitude vector of the UAV in space is defined as η: In equation (3), the roll angle φ represents the distance the UAV travels along the geographic coordinate system X. I The rotation angle; the pitch angle θ represents the UAV's rotation along the geographic coordinate system Y. I The rotation angle; the heading angle ψ represents the UAV's rotation along the geographic coordinate system Z. I The rotation angle, Euler angle, and angular velocity vector ω are: In equation (4), p, q, and r represent the distances along the X-axis of the UAV in the body coordinate system, respectively. B Y B Z B Angular velocity of the axis of rotation; Step 3: Construct a reward value based on an artificial potential field; Step 4: Construct a network for a UAV model predictive control algorithm based on Actor-Critic and artificial potential fields; The specific implementation method of step four is as follows: The purpose of UAV control is trajectory tracking control; therefore, the output of the UAV system is defined as the UAV's position ζ in the geographic coordinate system and its heading angle ψ during flight. Thus, the UAV system output Y is: In Equation (15), the output of the UAV system includes the position x, y, z of the UAV in the geographic coordinate system and the heading angle ψ during flight. In a static multi-obstacle environment / dynamic multi-obstacle environment, the target state R(k) of the UAV is set. During flight, the UAV measures its state X(k) in real time and feeds it back to the input. The measured UAV state X(k) is subtracted from the target state R(k) to obtain the deviation E(k) = R(k) - X(k). The deviation E(k) is used by the model predictive control module to generate the output control quantity U(k) to drive the UAV to the target position to eliminate the deviation E(k). According to equations (1), (2), (3) and (4), the state of the UAV system is defined as follows: Define the control output of the unmanned aerial vehicle (UAV) system as U = [U1 U2 U3 U4] T The output of the unmanned aerial vehicle (UAV) system is defined as Y = [xy zψ]. T Discretize the continuous state-space model; the discretized linear state-space model of the UAV is as follows: X(k+1|k)=A·X(k)+B·U(k) (16) In equation (16), k represents the current time step, A represents the weight coefficient matrix of the system state, B represents the weight coefficient matrix of the controller output, X(k) represents the UAV system state at time k, U(k) represents the controller output at time k, and X(k+1|k) represents the UAV system state predicted at time k+1. Thus, the system state quantity X within the prediction interval N is obtained. k,N With control quantity U k,N They are respectively: X k,N =[X(k|k)X(k+1|k)…X(k+N|k)] T (17) YOU k,N =[U(k|k)U(k+1|k)…U(k+N-1|k)] T (18) In Equation (17), X(k|k) is the UAV state quantity at time k, X(k+1|k) is the UAV state quantity predicted at time k+1, and X(k+N|k) is the UAV state quantity predicted at time k+N; in Equation (18), U(k|k) is the UAV control output quantity at time k, U(k+1|k) is the UAV control output quantity predicted at time k+1, and U(k+N-1|k) is the UAV control output quantity predicted at time k+N-1. Substituting equations (17) and (18) into equation (16), we obtain the predicted state X. k,N for: X k,N =M·X(k)+C·U k,N (19) In equation (19), X(k) is the UAV state variable at time k, and M = [IAA 2 …A N ] T , Since the primary control objective of the UAV is to track a desired trajectory, the desired trajectory R within the prediction interval N is defined. k,N for: R k,N =[R(k|k)R(k+1|k)…R(k+N|k)] (20) In equation (20), R(k+1) is the expected state quantity at time k+1, R(k+2) is the expected state quantity at time k+2, and R(k+N) is the expected state quantity at time k+N. Therefore, the trajectory error X within the prediction interval N k,N for: E k,N =[[R(k|k)-X(k|k)][R(k+1|k)-X(k+1|k)]…[R(k+N|k)-X(k+N|k)]] (21) In equation (21), [R(k|k)-X(k|k)] is the state error of the UAV system at time k, [R(k+1|k)-X(k+1|k)] is the state error of the UAV system at time k+1, and [R(k+N|k)-X(k+N|k)] is the state error of the UAV system at time k+N. Define the objective function J: In equation (22), Q and R are the weighting coefficient matrices of the error and the control quantity, respectively; Substituting equations (18) and (21) into equation (22), and simplifying equation (22), we get: J=E(k) T ·G·E(k)+2E(k) T ·F·U k,N +U k,N T ·H·U k,N (24) In equation (23), In equation (24), The Actor-Critic algorithm, by introducing an Actor policy network and a Critic evaluation network, improves the UAV control strategy, specifically including: Return at time k k The expression is: In equation (25), R k+i Let γ be the reward at time k+i, and γ be the loss factor. If the evaluation network Critic is a state value evaluation network, the input is the state; the loss function of the evaluation network Critic is calculated using the mean squared error loss function, which is the mean of the cumulative squared values ​​of the TD error values, and the expression is as follows: In equation (26), L critic Let V(s) be the loss function when the evaluation network Critic is an evaluation network with state values. k ) represents the state s at time k. k The corresponding value, V(s) k+1 ) represents the state s at time k+1. k+1 The corresponding value; If the evaluation network Critic is an evaluation network that evaluates actions based on their values ​​(action values), i.e., the input is state and action, then the loss function of the evaluation network Critic is as follows: In equation (27), L critic Let Q(s) be the loss function when the evaluation network Critic is an evaluation network for action values. k ,a k At time k, the controlled object is in state s. k At that time, take action a k The reward received at that time; The optimization objectives of the Actor network are as follows: In equation (28), π * Representing the optimal strategy, since this formula means that when the TD error value is greater than 0, the probability of choosing this action is increased, and when the TD error value is less than 0, the objective is to minimize the loss function -L. actor ; Therefore, the value function V for evaluating the network Critic is... π (s) parameter θ q The updated formula is: In equation (29), α1 is the descent step size. For descent gradient; Therefore, the policy function π of the Actor network θ (s) parameter θ π Updated formula: In equation (30), α2 is the descent step size. For descent gradient; Minimize the objective function J; the problem of finding the optimal control quantity is transformed into the following quadratic optimization problem: In equation (31), U(k) min and U(k) max These represent the minimum and maximum control values, respectively. The Actor network outputs the real-time optimal F and H weight coefficient matrices, and the Critic network evaluates the policy performance using TDerror. Then, this quadratic optimization problem is solved to obtain the optimal control sequence U within the prediction interval N. k,N Then, the first item U(k+1|k) in the sequence is selected as the output of the MPC controller at the current time k, and rolling optimization and feedback correction are performed to control the quadcopter UAV to perform trajectory tracking. Step 5: Set network parameters; Step Six: Based on the network parameters set in Step Five, input the reward value based on the artificial potential field constructed in Step Three into the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in Step Four. Use the dynamic model of the quadcopter UAV constructed in Step Two as the controlled object, and train it in various static and dynamic multi-obstacle environments constructed in Step One. Output the training weight file after each round of training. By comparing with the previous training results, select the training weight file corresponding to the training result with the highest reward value and the shortest path as the optimal weight file. Step 7: Input the optimal weight file obtained in Step 6 and the network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field constructed in Step 4 into various static and dynamic multi-obstacle environments constructed in Step 1 for testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurs and threat index results.

2. The UAV model predictive control method based on Actor-Critic and artificial potential field according to claim 1, characterized in that, The implementation method of step one includes: Step 101: Set up multiple static multi-obstacle environments: Set the center position (x,y,z) and radius R of the sphere, the center position (x,y,z), radius R and height H of the bottom circle of the cylinder, and the center position (x,y,z), radius R and height H of the bottom circle of the cone; Step 102: Set up multiple dynamic multi-obstacle environments: Set the center position (x, y, z) of the sphere, radius R, and the trajectory and velocity v of the obstacles.

3. The UAV model predictive control method based on Actor-Critic and artificial potential field according to claim 1, characterized in that, The specific implementation method of step two also includes: The state space of the control variable is defined as follows: U=[U1 U2 U3 U4] T (5) In equation (5), The sum of the thrust generated by the four rotors represents the thrust of the quadcopter UAV along the Z-axis of the machine coordinate system. B Displacement motion; The rolling torque represents the torque of the quadcopter drone along the X-axis. B Rotating and tumbling motion; The pitch moment represents the pitch moment of the quadcopter UAV along the Y-axis. B Rotational pitch motion; The yaw moment represents the torque along the Z-axis. B Rotational yaw motion; K is the lift coefficient; K d ω1, ω2, ω3, and ω4 are the anti-torque coefficients; l is the distance from the center of each rotor to its center of mass; ω1, ω2, ω3, and ω4 are the rotational speeds of the four rotors, respectively. The state space and state vector of the dynamic model of the quadcopter UAV are as follows: In equation (6), the state vector of the quadcopter UAV is Therefore, the state of the quadcopter UAV at time k is X(k)∈X; x k y k z It is obtained by decomposing the air drag coefficient of the UAV in an inertial reference frame; k φ k θ k ψ The total air resistance torque coefficients are respectively in X I Y I Z I Components of the axis; I x I y I z For quadcopter UAVs along the aircraft system X B Y B Z B Moment of inertia of the shaft; J RP ω is the moment of inertia of the propeller along its axis of rotation; Ω = -ω1 + ω2 - ω3 + ω4 is the algebraic sum of the motor speeds.

4. The UAV model predictive control method based on Actor-Critic and artificial potential field according to claim 1, characterized in that, The specific implementation method of step three is as follows: The reward function is set with the following four rewards: (1) reward r1 for attracting the drone to the target; (2) reward r2 for repelling the drone from the obstacle; (3) collision penalty r3; (4) reward r4 for reaching the target. (1) Rewards for attracting drones to the target The attraction of the target to the UAV is proportional to the distance between the controlled object and the target point; as shown in equation (9), the reward r1 caused by the attraction of the target to the UAV in the artificial potential field is simplified: D attract =|q-q g | (7) D 0_g =|q0-q g | (8) In equation (7), D attract Let q be the distance between the drone and the target point, and q be the drone's position. g The target location; D in equation (8) 0_g Let q0 be the distance between the initial position and the target position, and q0 be the starting position; in equation (9), This represents the distance between the UAV and the target at time k; This represents the distance between the drone and the target at time k+1; This represents the change in the drone's position between time k and k+1. At that time, it indicates that the drone is approaching the target. When, it indicates that the drone's position has not changed. At that time, it indicates that the drone is moving away from the target; (2) Rewards for drones that are repelled by obstacles When the drone enters the repulsive force range D of the obstacle, the reward r2 caused by the obstacle's repulsion of the drone is set by simplifying the artificial potential field; D repel =|q-q o | (10) When the drone is within the repulsive field range D of the obstacle, the obstacle exerts a certain repulsive force on the drone, causing the drone to move away from the obstacle and preventing the drone from colliding with the obstacle; in equation (10), D repel Let q be the distance between the drone and the obstacle, and q be the drone's position. o The location of the obstacle; in equation (11), This represents the distance between the drone and the obstacle at time k; This represents the distance between the drone and the obstacle at time k+1; This represents the change in distance caused by the drone's repulsion from obstacles between time k and k+1. When, it indicates that the drone is approaching the obstacle, when When, it indicates that the drone's position has not changed. This indicates that the drone is moving away from the obstacle; (3) Collision penalty When the drone collides with an obstacle, it will receive a collision penalty of r3. In equation (12), the collision reward is equal to the distance between the UAV and the center of the obstacle at the next moment. Subtract the obstacle radius R and divide by the obstacle radius R minus 1; (4) Rewards for reaching the goal When the drone reaches the target point, the environment will give the drone a positive reward of r4; In equation (13), when the distance between the drone and the target point is 0, the environment rewards the drone with 3; when the distance between the drone and the target point is not 0, the environment rewards the drone with 0. The reward for artificial potential field processing is defined as follows: APF-Reward=λ1·r1+λ2·r2+λ3·r3+λ4·r4 (14) In equation (14), APF-Reward is the reward value of artificial potential field processing; λ1, λ2, λ3, and λ4 represent the weights of the four rewards, respectively.

5. The UAV model predictive control method based on Actor-Critic and artificial potential field according to claim 1, characterized in that, The specific implementation method of step seven is as follows: The experimental data, including path length L, maximum path twist LS, global path twist GS, collision occurrence, and threat index, are calculated as follows: (1) Path length L In equation (32), S k This represents the distance the drone travels from time k to k+1, which is equal to the distance between the drone and the target at time k. Distance between the UAV and the target at time k+1 The absolute value of the difference; In equation (33), the path length L is equal to S. k The iterations are summed, where N is the number of algorithm iterations; (2) Path maximum twist LS The distortion τ of the drone's flight curve at time k k for: In equation (34), T k Let N be the tangent vector of the UAV's flight curve at time k. k Let B be the normal vector of the UAV's flight curve at time k. k Let B′ be the cross product of the normal vectors of the UAV's flight curve at time k (also known as the double normal vectors of the hyperbola). k Let be the derivative of the two normal vectors of the UAV's flight curve at time k; LS=max{τ 0 ,t 1 ,...,t N } (35) (3) Path global distortion GS Integrating the twist degree on the path, we obtain the global twist metric GS of the path. In Equation (36), τ(s) is the twist degree of point s on the path, and a and b are the start and end points of the UAV flight path. (4) Whether a collision occurred In equation (37), the distance between the UAV and the obstacle at time k is... When the distance between the drone and the obstacle is less than or equal to the radius R of the obstacle, a collision occurs; when the distance between the drone and the obstacle at time k... When the radius R of the obstacle is greater than the obstacle's radius, the drone does not collide with the obstacle. (5) Threat Index In equation (38), when When a drone collides with an obstacle, the threat level equals ∞; when When a drone is within the obstacle's repulsion range but does not collide with it, the threat index is equal to... ν k Let k be the speed of the drone at time k. Let k be the distance between the drone and the center of the obstacle; when The threat index is 0 when the drone is not within the range of the obstacle's repulsion effect.

6. A UAV model prediction and control system based on Actor-Critic and artificial potential field according to any one of claims 1 to 5, characterized in that, include: Environment building module: used to build various static and dynamic multi-obstacle environments; The quadcopter drone dynamics model building module is used to build the dynamics model of a quadcopter drone. Artificial potential field-based reward value construction module: used to construct reward values ​​based on artificial potential fields; Network building module: Used to build networks for predictive control methods of UAV models based on Actor-Critic and artificial potential fields; Network training module: According to the set network parameters, the constructed reward value based on artificial potential field is input into the constructed network of UAV model predictive control algorithm based on Actor-Critic and artificial potential field. The constructed dynamic model of quadcopter UAV is used as the control object. Training is carried out in various constructed static multi-obstacle environments and dynamic multi-obstacle environments. After each round of training, the training weight file is output. The training weight file corresponding to the training result with the highest reward value and the shortest path is selected as the optimal weight file; Test result output module: This module is used to input the optimal weight file and the constructed network of the UAV model predictive control algorithm based on Actor-Critic and artificial potential field into various static and dynamic multi-obstacle environments for testing, and obtain the UAV flight path map, reward value r, path length L, maximum path distortion LS, global path distortion GS, whether a collision occurred, and threat index results.

7. A UAV model prediction and control device based on Actor-Critic and artificial potential field, characterized in that, include: Memory: A computer program for predictive control of a UAV model based on Actor-Critic and artificial potential field as described in any one of claims 1-5, which is a computer-readable device; Processor: Used to implement the UAV model predictive control method based on Actor-Critic and artificial potential field as described in any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the UAV model predictive control method based on Actor-Critic and artificial potential field as described in any one of claims 1-5.

Images