A reinforcement learning optimized UAV trajectory tracking model predictive control method

Abstract

The present invention is applicable to the field of drone control technology and provides a reinforcement learning-optimized model predictive control method for drone trajectory tracking, comprising the following steps: Step S1: Establishing a quadrotor drone dynamics model, including a position dynamics model and an attitude dynamics model; Step S2: Constructing a model predictive controller based on the drone dynamics model and formulating a model predictive control optimization problem to obtain a drone trajectory tracking cost function; Step S3: Constructing a proximal policy optimization algorithm network, including an actor network and a critic network, using the proximal policy optimization algorithm to adjust the weight matrix of the model predictive controller cost function, and solving the model predictive control optimization problem to obtain the optimal control input. This invention not only solves the difficulty in selecting controller parameters in existing drone control methods but also improves the drone's processing capabilities and stability in diverse mission environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle (UAV) control, and in particular relates to a prediction control method for an unmanned aerial vehicle (UAV) trajectory tracking model optimized through reinforcement learning. Background Art

[0002] With the rapid development of drone technology, its applications in military, civilian, and scientific research fields are becoming increasingly widespread. When performing missions such as reconnaissance, monitoring, aerial photography, and logistics distribution, drones must accurately track pre-set flight paths to ensure mission success. This places higher demands on the performance of trajectory tracking controllers. The stability and accuracy of trajectory tracking controllers directly impact the effectiveness of drone missions, especially in complex and changing environments, such as those facing challenges like shifting winds and obstacle avoidance.

[0003] Currently, the control algorithms for drone trajectory tracking controllers are showing a trend toward diversification. The traditional PID control algorithm has been widely used due to its simplicity and ease of implementation, but its control performance in complex environments is less than ideal. To overcome this limitation, researchers have begun to apply intelligent control algorithms such as fuzzy control, neural network control, and sliding mode control to trajectory tracking control to improve the adaptability and robustness of drones. At the same time, trajectory tracking control methods based on optimization algorithms, such as linear quadratic tracking controllers, have become a research hotspot due to their ability to solve optimal control problems and accurately track the desired trajectory. Furthermore, some researchers have applied machine learning algorithms to trajectory tracking control, improving controller performance through model training.

[0004] Model predictive control (MPC) for unmanned aerial vehicles (UAVs) has attracted considerable attention due to its advantages, including online learning of system model parameters, adapting to changes in system parameters and external disturbances, improving anti-interference capabilities, and optimizing control performance. However, MPC also faces several challenges: It requires solving an optimization problem in real time to predict future states and generate control inputs accordingly. This often involves extensive computation, especially when dealing with high-dimensional systems and complex constraints. This can lead to delays in the generation of control commands, compromising the system's real-time responsiveness. Furthermore, MPC performance is highly dependent on the accuracy of the system model. In practical applications, obtaining a completely accurate system model is difficult due to factors such as system parameter uncertainty, external disturbances, and unmodeled dynamics. Furthermore, parameter adjustments in MPC (such as the prediction horizon, control horizon, and weight coefficients) significantly impact control performance, but often rely on experience and trial-and-error, lacking systematic guidance. Improper adjustments can lead to degraded control performance and even system instability. Snow patrols often require patrols on steep slopes, in deep snow, and amidst obstacles such as trees and rocks, placing high demands on patrol personnel's safety and efficiency. Aiming at the application of MPC in the complex and changeable environment of snow patrol, this paper proposes a reinforcement learning optimized UAV trajectory tracking model predictive control method. Summary of the Invention

[0005] The purpose of the present invention is to provide a reinforcement learning optimized UAV trajectory tracking model predictive control method, aiming to solve the problems raised in the above background technology.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A reinforcement learning optimized UAV trajectory tracking model predictive control method includes the following steps:

[0008] Step S1, establishing a quadrotor UAV dynamics model, including a position dynamics model and an attitude dynamics model;

[0009] Step S2: Based on the UAV dynamics model, a model predictive controller is constructed, and a model predictive control optimization problem is established to obtain a UAV trajectory tracking cost function;

[0010] Step S3: Construct a proximal policy optimization algorithm network, including an actor network and a critic network, adjust the weight matrix of the model predictive controller cost function through the proximal policy optimization algorithm, and solve the model predictive control optimization problem to obtain the optimal control input.

[0011] Furthermore, in step S1, the position dynamics model constructed is as follows:

[0012]

[0013] in, They represent the displacement acceleration of the UAV in the X, Y, and Z directions in the ground coordinate system; m represents the mass of the quadcopter; g is the acceleration due to gravity; ξ x ,ξ y ,ξ z They represent the coefficients of the total air resistance coefficient of the quadrotor drone after decomposition according to the X, Y, and Z directions respectively; θ, ψ represent the roll angle, pitch angle, and yaw angle of the quadrotor drone, respectively; U1 represents the total lift generated by the propeller of the quadrotor drone; Respectively represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time;

[0014] The constructed posture dynamics model is as follows:

[0015]

[0016] in, They represent the angular accelerations in the X, Y, and Z directions in the quadrotor drone body coordinate system; I x ,I y ,I z are the moments of inertia of the X, Y, and Z axes in the coordinate system of the quadrotor drone; U2, U3, and U4 represent the virtual control quantities in the X, Y, and Z directions in the coordinate system of the quadrotor drone respectively; They represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone respectively; J RP represents the total moment of inertia of the entire motor rotor and propeller around the body axis; kθ, kψ represents the components of the total drag torque coefficient in the X, Y, and Z axes in the body coordinate system; Ω represents the propeller angular velocity vector.

[0017] Furthermore, when constructing the model predictive controller, the quadrotor UAV dynamics model is combined with the quadrotor UAV kinematics model to obtain a complete motion model of the UAV; the quadrotor UAV kinematics model is:

[0018]

[0019] in, They represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time; v x ,v y ,v z Respectively represent the speed of the quadrotor drone in the X, Y, and Z directions in the ground coordinate system; They represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone with respect to time; p, q, and r represent the angular velocities of the pitch angle, roll angle, and yaw angle of the quadrotor drone, respectively.

[0020] Furthermore, when establishing the model predictive control optimization problem, the UAV model predictive discretized kinematic model is used as the state equation. The UAV model predictive discretized kinematic model is:

[0021] X(k+1)=AX(k)+BU(k)

[0022] Where:

[0023]

[0024] U=[F1,F2,F3,F4] T

[0025] A satisfies:

[0026]

[0027] B satisfies:

[0028]

[0029] Where k represents the kth moment of the entire system; k+1 represents the moment after a time interval of k; N represents the length of the prediction domain; They represent the derivatives of the X, Y, and Z axis positions of the UAV in the ground coordinate system with respect to time; x, y, and z represent the positions of the quadrotor UAV in the ground coordinate system; They represent the time derivatives of the quadrotor UAV’s pitch angle, roll angle, and yaw angle respectively; θ, ψ represent the roll angle, pitch angle, and yaw angle of the quadrotor drone, respectively; F1, F2, F3, and F4 represent the control components of the four propellers of the quadrotor drone; I x ,I y ,I z where m represents the mass of the quadrotor drone; and U1 represents the total lift generated by the propellers of the quadrotor drone.

[0030] Furthermore, the model predictive control optimization problem is established based on the UAV model predictive control performance index, and the UAV model predictive control performance index is:

[0031]

[0032] Among them, X(k) represents the position state information of the UAV at time k; U(k) represents the control input information of the UAV at time k; Q k and p k Represent the state error gain coefficient and control input gain coefficient in the cost function, Q k and p k satisfy:

[0033]

[0034] Where Q and p represent the state error gain coefficient and control input gain coefficient in the preset value function respectively; S=diag[S Q ,S p ], S represents the total cost function gain coefficient matrix, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix, respectively, which are given by the proximal policy optimization algorithm.

[0035] Furthermore, the action space of the proximal strategy optimization algorithm is:

[0036] Action=[s1,s2,s3...s n ]

[0037] Where n = R(S), n represents the number of elements in the action space, and R(S) represents the rank of the total cost function gain coefficient matrix; the upper and lower limits of the action space are (-∞, +∞) and conform to the Gaussian distribution;

[0038] According to the requirements of the weight matrix, the action space Action is cropped and numerically scaled to obtain:

[0039] Action'=[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ,s p,1 ,s p,2 ,s p,3 ...s p,k ]

[0040] Where j = R(S Q ), k=R(S p ), j represents the number of elements in the action space that affect the optimization matrix of the state error weights, k represents the number of elements in the action space that affect the optimization matrix of the control input weights, R(S Q ) represents the rank of the state error weight optimization matrix, R(S p ) represents the rank of the control input weight optimization matrix; and then the weight optimization matrix is ​​obtained:

[0041] S Q =diag[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ]

[0042] S p =diag[s p,1 ,s p,2 ,s p,3 ...s p,k ]

[0043] Among them, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix respectively.

[0044] Furthermore, the reward function of the proximal policy optimization algorithm is:

[0045]

[0046] Among them, G represents the total gain coefficient of the reward function, λ represents the error penalty coefficient, l represents the length of the control domain, and x i,t ,y i,t ,z i,t represents the desired position, x i ,y i ,z i represents the current actual position, and i represents the i-th moment in the control domain.

[0047] Furthermore, the critic network adjusts parameters by optimizing the loss function to maximize the reward function; through iterative training and optimization, the critic network enables the reward function to guide the model to optimize performance and dynamically adjust the model predictive controller.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] By combining reinforcement learning with MPC, this invention provides a more flexible, intelligent, and robust solution for drone trajectory tracking. This solution leverages the decision-making mechanisms of reinforcement learning to optimize the parameters of the MPC controller, enabling it to better adapt to complex and changing environments and mission requirements. This allows the drone to adjust its optimal actions in real time based on its current state during patrol missions, ensuring accurate and safe completion of patrol missions. This invention not only addresses the difficulty of selecting controller parameters in existing drone control methods but also improves the drone's processing capabilities and stability in diverse mission environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flowchart of the method of the present invention.

[0051] Figure 2 This is a schematic diagram of the discretized kinematic model for predicting the UAV model in the present invention.

[0052] Figure 3 Schematic diagram of the reinforcement learning and model predictive control network in the present invention.

[0053] Figure 4 The blue line is the target trajectory and the orange line is the drone's trajectory.

[0054] Figure 5 is the trajectory tracking error of the UAV in Example 1. DETAILED DESCRIPTION

[0055] In order to have a clearer understanding of the technical features, objectives and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below, but it should not be understood as limiting the scope of implementation of the present invention.

[0056] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0057] One embodiment of the present invention provides a reinforcement learning optimization UAV trajectory tracking model predictive control method, the flow chart of which is as follows: Figure 1 The method comprises the following steps:

[0058] Step S1, establishing a quadrotor UAV dynamics model, including a position dynamics model and an attitude dynamics model;

[0059] Step S2: Based on the UAV dynamics model, a model predictive controller is constructed, and a model predictive control optimization problem is established to obtain a UAV trajectory tracking cost function;

[0060] Step S3: Construct a proximal policy optimization (PPO) algorithm network, including an actor network and a critic network, adjust the weight matrix of the model predictive controller cost function through the proximal policy optimization algorithm, and solve the model predictive control optimization problem to obtain the optimal control input.

[0061] The details are as follows:

[0062] 1) In step S1, the position dynamics model constructed is as follows:

[0063]

[0064] in, They represent the displacement acceleration of the UAV in the X, Y, and Z directions in the ground coordinate system; m represents the mass of the quadcopter; g is the acceleration due to gravity; ξ x ,ξy ,ξ z They represent the coefficients of the total air resistance coefficient of the quadrotor drone after decomposition according to the X, Y, and Z directions respectively; θ, ψ represent the roll angle, pitch angle, and yaw angle of the quadrotor drone, respectively; U1 represents the total lift generated by the propeller of the quadrotor drone; Respectively represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time;

[0065] The constructed posture dynamics model is as follows:

[0066]

[0067] in, They represent the angular accelerations in the X, Y, and Z directions in the quadrotor drone body coordinate system; I x ,I y ,I z are the moments of inertia of the X, Y, and Z axes in the coordinate system of the quadrotor drone; U2, U3, and U4 represent the virtual control quantities in the X, Y, and Z directions in the coordinate system of the quadrotor drone respectively; They represent the time derivatives of the quadrotor UAV’s pitch angle, roll angle, and yaw angle respectively; J RP represents the total moment of inertia of the entire motor rotor and propeller around the body axis; kθ, kψ represents the components of the total drag torque coefficient in the X, Y, and Z axes in the body coordinate system; Ω represents the propeller angular velocity vector.

[0068] 2. In step S2:

[0069] (1) When constructing the model predictive controller, the quadrotor UAV dynamic model is combined with the quadrotor UAV kinematic model to obtain a complete motion model of the UAV; the quadrotor UAV kinematic model is:

[0070]

[0071] in, They represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time; v x ,v y ,v z Respectively represent the speed of the quadrotor drone in the X, Y, and Z directions in the ground coordinate system; They represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone with respect to time; p, q, and r represent the angular velocities of the pitch angle, roll angle, and yaw angle of the quadrotor drone, respectively.

[0072] (2) When establishing the model predictive control optimization problem, the UAV model is used to predict the discretized kinematic model as the state equation. The principle diagram is as follows Figure 2 As shown in the figure, the discretized kinematic model predicted by the UAV model is:

[0073] X(k+1)=AX(k)+BU(k)

[0074] Where:

[0075]

[0076]

[0077] U=[F1,F2,F3,F4] T

[0078] A satisfies:

[0079]

[0080]

[0081] B satisfies:

[0082]

[0083] Where k represents the kth moment of the entire system; k+1 represents the moment after a time interval of k; N represents the length of the prediction domain; They represent the derivatives of the X, Y, and Z axis positions of the UAV in the ground coordinate system with respect to time; x, y, and z represent the positions of the quadrotor UAV in the ground coordinate system; They represent the time derivatives of the quadrotor UAV’s pitch angle, roll angle, and yaw angle respectively; θ, ψ represent the roll angle, pitch angle, and yaw angle of the quadrotor drone, respectively; F1, F2, F3, and F4 represent the control components of the four propellers of the quadrotor drone; I x ,I y ,I z where m represents the mass of the quadrotor drone; and U1 represents the total lift generated by the propellers of the quadrotor drone.

[0084] (3) The model predictive control optimization problem is established based on the UAV model predictive control performance index. The UAV model predictive control performance index is:

[0085]

[0086] Among them, X(k) represents the position state information of the UAV at time k; U(k) represents the control input information of the UAV at time k; Q k and p k Represent the state error gain coefficient and control input gain coefficient in the cost function, Q k and p k satisfy:

[0087]

[0088] Where Q and p represent the state error gain coefficient and control input gain coefficient in the preset value function respectively; S=diag[S Q ,S p ], S represents the total cost function gain coefficient matrix, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix, respectively, which are given by the proximal policy optimization algorithm.

[0089] 3. In the step S3;

[0090] (1) The action space of the proximal strategy optimization algorithm is:

[0091] Action=[s1,s2,s3...s n ]

[0092] Where n = R(S), n represents the number of elements in the action space, and R(S) represents the rank of the total cost function gain coefficient matrix; the upper and lower limits of the action space are (-∞, +∞) and conform to the Gaussian distribution;

[0093] According to the requirements of the weight matrix, the action space Action is cropped and numerically scaled to obtain:

[0094] Action'=[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ,s p,1 ,s p,2 ,s p,3 ...s p,k ]

[0095] Where j = R(S Q ), k=R(S p ), j represents the number of elements in the action space that affect the optimization matrix of the state error weights, k represents the number of elements in the action space that affect the optimization matrix of the control input weights, R(S Q ) represents the rank of the state error weight optimization matrix, R(S p) represents the rank of the control input weight optimization matrix; and then the weight optimization matrix is ​​obtained:

[0096] S Q =diag[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ]

[0097] S p =diag[s p,1 ,s p,2 ,s p,3 ...s p,k ]

[0098] Among them, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix respectively.

[0099] (2) The reward function of the proximal policy optimization algorithm is:

[0100]

[0101] Where G and λ represent positive gain parameters, l represents the length of the control domain, and x i,t ,y i,t ,z i,t represents the desired position, x i ,y i ,z i represents the current actual position, and i represents the i-th moment in the control domain.

[0102] (3) The critic network adjusts parameters by optimizing the loss function to maximize the reward function. In this way, the advantages of model-free reinforcement learning are combined with the robustness and online replanning capabilities of model predictive control, and the output model-based controller can ensure that the current output is feasible. Through continuous iterative training and optimization, the parameters of the critic network will gradually adjust so that the reward function can better guide the model to achieve better performance in different tasks and environments, thereby dynamically optimizing the performance of the model predictive controller. In addition, in order to improve the overall computational efficiency, the embodiment of the present invention also adopts a method of expanding the prediction domain and control domain to improve the versatility of the controller in facing different tasks and the accuracy of tracking under long control domain conditions.

[0103] like Figure 3As shown in the figure, the system workflow is as follows: The desired trajectory set by the system obtains the position information of each trajectory point at each moment. The environment module obtains the desired position information of the UAV and the actual position information and error under the current weight matrix. It also stores historical data of the interaction between the UAV and the environment (including state, action, and reward information) in a rollout buffer. The actor network in the actor-critic network structure determines the action of the weight optimization matrix based on the current UAV state and environmental information to optimize the cost function of the model predictive control. The model predictive controller predicts the future state of the system based on the current system state and input. It uses the cost function to obtain the control input sequence within the prediction domain and transmits the control domain portion of the optimal control sequence as the control input information to the UAV, thereby obtaining the UAV's actual position information. The environment module assigns rewards to the actions of the weight optimization matrix based on the position error and the reward function. The critic network uses the advantage function based on the reward function to update its own parameters, thereby helping the actor network update its policy parameters. By continuously optimizing and updating the weight matrix, the above steps are repeated at the next moment, achieving rolling optimization of the cost function and improving the UAV's trajectory tracking performance.

[0104] Example 1: Select a quadrotor drone system and set detailed system parameters for it: m = 1 kg, g = 9.8 m / s 2 , I x =4×10 -3 kg·m 2 , I y =4×10 -3 kg·m 2 , I z =8.4×10 -3 kg·m 2 , l a =0.2m, b=2.9×10 -5 N·s 2 , d = 1.1 × 10 -6 N·s 2 , where m is the mass of the quadcopter, g is the acceleration of gravity, and I x , I y and I z are the moments of inertia of the x, y, and z axes respectively, b is the thrust coefficient, and d is the drag coefficient.

[0105] The expected trajectory is set as:

[0106]

[0107] Wherein, t = i × dt, i is the i-th target point, and dt = 0.02s.

[0108] Set the weight matrix of the position ring and attitude ring: the selected position ring weight matrix Q P =diag{40, 40, 40, 1, 1, 1}, R P =diag{1, 1, 1}, attitude ring weight matrix Q A =diag{40, 40, 40, 1, 1, 1}, R A =diag{1,1,1}.

[0109] Finally, the UAV successfully tracked the target trajectory, such as Figure 4 As shown in the figure. The blue line represents the target trajectory, and the orange line represents the motion trajectory of the UAV. It can be found from the figure that the motion trajectory of the UAV almost coincides with the target trajectory, which shows that the trajectory tracking performance of the controller is very good. In addition, the trajectory tracking error of the UAV within 0-14s is analyzed, as shown in the figure. Figure 5 As shown in the figure, the trajectory tracking error can be kept below a low value, which further proves the stability and accuracy of the controller.

[0110] The above are only preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several variations and improvements can be made without departing from the concept of the present invention. These should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A reinforcement learning optimized UAV trajectory tracking model predictive control method, characterized by: The following steps are involved: Step S1, establishing a quadrotor UAV dynamics model, including a position dynamics model and an attitude dynamics model; Step S2: Based on the UAV dynamics model, a model predictive controller is constructed, and a model predictive control optimization problem is established to obtain a UAV trajectory tracking cost function; Step S3: construct a proximal policy optimization algorithm network, including an actor network and a critic network, adjust the weight matrix of the model predictive controller cost function through the proximal policy optimization algorithm, and solve the model predictive control optimization problem to obtain the optimal control input; The model predictive control optimization problem is established based on the UAV model predictive control performance index, and the UAV model predictive control performance index is: ； in, N Indicates the length of the prediction domain; Represents the position status information of the UAV at time k; represents the control input information of the UAV at time k; and Respectively represent the state error gain coefficient and control input gain coefficient in the cost function, and satisfy: ； in, and Respectively represent the state error gain coefficient and the control input gain coefficient in the preset cost function; , S represents the total cost function gain coefficient matrix, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix, respectively, which are given by the proximal strategy optimization algorithm; The action space of the proximal policy optimization algorithm is: ； in, , represents the number of elements in the action space, Represents the rank of the total cost function gain coefficient matrix; the upper and lower limits of the action space are , and conforms to the Gaussian distribution; According to the requirements of the weight matrix, the action space Action After cropping and numerical scaling, we get: ； in, , , Represents the number of elements in the action space that affect the optimization matrix of the state error weight, represents the number of elements in the action space that affect the optimization matrix of the control input weights, represents the rank of the state error weight optimization matrix, Represents the rank of the control input weight optimization matrix; and then the weight optimization matrix is ​​obtained: ； 。 2. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 1 is characterized in that: In step S1, the position dynamics model constructed is as follows: ； in, Respectively represent the displacement acceleration of the UAV in the X, Y, and Z directions in the ground coordinate system; m Indicates the mass of the quadrotor drone; g is the acceleration due to gravity; They represent the coefficients of the total air resistance coefficient of the quadrotor drone after decomposition according to the X, Y, and Z directions respectively; Respectively represent the roll angle, pitch angle, and yaw angle of the quadrotor drone; It represents the total lift generated by the propellers of the quadrotor drone; Respectively represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time; The constructed posture dynamics model is as follows: ； in, Respectively represent the angular accelerations in the X, Y, and Z directions in the quadrotor drone body coordinate system; are the X, Y, and Z axis moments of inertia in the quadrotor drone body coordinate system; They represent the virtual control quantities in the X, Y, and Z directions in the quadrotor drone body coordinate system respectively; They represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone respectively; It represents the total moment of inertia of the entire motor rotor and propeller around the body axis; Respectively represent the components of the total resistance torque coefficient in the X, Y, and Z axes in the body coordinate system; represents the propeller angular velocity vector.

3. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 1 is characterized in that: When constructing the model predictive controller, the quadrotor UAV dynamic model is combined with the quadrotor UAV kinematic model to obtain a complete motion model of the UAV; the quadrotor UAV kinematic model is: ； in, Respectively represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time; Respectively represent the speed of the quadrotor drone in the X, Y, and Z directions in the ground coordinate system; They represent the time derivatives of the quadrotor UAV’s pitch angle, roll angle, and yaw angle respectively; Represent the angular velocity of the quadrotor drone's pitch angle, roll angle, and yaw angle respectively.

4. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 1 is characterized in that: When establishing the model predictive control optimization problem, the UAV model predictive discretized kinematic model is used as the state equation. The UAV model predictive discretized kinematic model is: ； Where: ； ； ； ； A satisfies: ； B satisfies: ； in, k represents the kth moment of the entire system; k +1 indicates the moment after 1 time interval at moment k; Respectively represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time; They represent the positions of the quadrotor drone in the ground coordinate system respectively; They represent the time derivatives of the quadrotor UAV’s pitch angle, roll angle, and yaw angle respectively; Respectively represent the roll angle, pitch angle, and yaw angle of the quadrotor drone; Represents the control components of the four propellers of a quadrotor drone; are the X, Y, and Z axis moments of inertia in the quadrotor drone body coordinate system; Represent the first six elements of X, namely roll angle, roll angular velocity, pitch angle, pitch angular velocity, yaw angle, and yaw angular velocity; m Indicates the mass of the quadrotor drone; Represents the total lift generated by the propellers of the quadrotor drone.

5. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 1 is characterized in that: The reward function of the proximal policy optimization algorithm is: ； in, G represents the total gain coefficient of the reward function, λ represents the error penalty coefficient, l Indicates the length of the control field. Indicates the desired position, Indicates the current actual position. i represents the i-th moment in the control domain.

6. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 5 is characterized in that: The critic network adjusts parameters by optimizing the loss function to maximize the reward function; through iterative training and optimization, the critic network enables the reward function to guide the model to optimize performance and dynamically adjust the model predictive controller.

Images