Prediction control method for reinforcement learning optimization unmanned aerial vehicle trajectory tracking model

Abstract

The invention is suitable for the technical field of unmanned aerial vehicle control, and provides a reinforcement learning optimization unmanned aerial vehicle trajectory tracking model prediction control method, which comprises the following steps: S1, establishing a four-rotor unmanned aerial vehicle dynamics model, which comprises a position dynamics model and an attitude dynamics model; s2, constructing a model prediction controller based on the unmanned aerial vehicle kinetic model, and establishing a model prediction control optimization problem to obtain an unmanned aerial vehicle trajectory tracking cost function; and S3, constructing a near-end strategy optimization algorithm network which comprises an action home network and a reviewer network, adjusting a weight matrix of a cost function of the model prediction controller through a near-end strategy optimization algorithm, and solving a model prediction control optimization problem to obtain optimal control input. According to the method, the problem of difficulty in controller parameter selection in an existing unmanned aerial vehicle control method is solved, and the processing capacity and stability of the unmanned aerial vehicle facing different task environments are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle control, and in particular relates to a prediction control method for unmanned aerial vehicle trajectory tracking model optimized by reinforcement learning. Background Art

[0002] With the rapid development of drone technology, its application in military, civil, scientific research and other fields has become more and more extensive. When drones perform tasks such as reconnaissance, monitoring, aerial photography, logistics distribution, etc., they must accurately track the preset flight trajectory to ensure the success of the mission, which puts higher requirements on the performance of the trajectory tracking controller. The stability and accuracy of the trajectory tracking controller directly affect the performance of the drone's mission, especially when facing complex and changing environments, such as wind direction changes, obstacle avoidance and other challenges.

[0003] At present, the control algorithms of UAV trajectory tracking controllers are showing a trend of diversification. The traditional PID control algorithm has been widely used because of its simplicity and easy implementation, but its control effect in complex environments is not ideal. In order to overcome this limitation, researchers have begun to try to apply intelligent control algorithms such as fuzzy control, neural network control, and sliding mode control to trajectory tracking control to improve the adaptability and robustness of UAVs. At the same time, trajectory tracking control methods based on optimization algorithms, such as linear quadratic tracking controllers, have also become a research hotspot because they can solve optimal control problems to achieve accurate tracking of the desired trajectory. In addition, some researchers have applied machine learning algorithms to trajectory tracking control to improve the performance of the controller by training models.

[0004] Model predictive control (MPC) of unmanned aerial vehicles has attracted much attention due to its advantages such as online learning of system model parameters, adapting to changes in system parameters and external disturbances, improving anti-interference ability and optimizing control performance. However, MPC also faces some challenges: MPC needs to solve the optimization problem in real time to predict the future state and generate control inputs accordingly, which usually involves a lot of calculations. Especially when dealing with high-dimensional systems and complex constraints, it may cause delays in the generation of control instructions and affect the real-time response capability of the system. In addition, the performance of MPC is highly dependent on the accuracy of the system model. In practical applications, it is difficult to obtain a completely accurate system model due to factors such as uncertainty in system parameters, external disturbances, and unmodeled dynamics. At the same time, parameter adjustment in MPC (such as prediction time domain, control time domain, weight coefficient, etc.) has a significant impact on control performance, but it often relies on experience and trial and error, lacks a systematic guidance method, and improper adjustment may lead to a decrease in control performance or even cause system instability. Snow field patrols usually need to be carried out on steep slopes, deep snow areas, and obstacles such as trees and rocks, which places extremely high demands on the safety and efficiency of patrol personnel. Aiming at the application of MPC in the complex and changeable environment of snow field patrol, this paper proposes a reinforcement learning optimized UAV trajectory tracking model predictive control method. Summary of the invention

[0005] The purpose of the present invention is to provide a reinforcement learning optimized UAV trajectory tracking model predictive control method, aiming to solve the problems raised in the above background technology.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A reinforcement learning optimized UAV trajectory tracking model predictive control method comprises the following steps:

[0008] Step S1, establishing a quadrotor UAV dynamics model, including a position dynamics model and an attitude dynamics model;

[0009] Step S2: Based on the UAV dynamics model, a model predictive controller is constructed, and a model predictive control optimization problem is established to obtain a UAV trajectory tracking cost function;

[0010] Step S3: construct a proximal policy optimization algorithm network, including an actor network and a critic network, adjust the weight matrix of the model predictive controller cost function through the proximal policy optimization algorithm, and solve the model predictive control optimization problem to obtain the optimal control input.

[0011] Furthermore, in step S1, the position dynamics model constructed is as follows:

[0012]

[0013] in, They represent the displacement acceleration of the UAV in the X, Y, and Z directions in the ground coordinate system; m represents the mass of the quadrotor UAV; g is the gravitational acceleration; ξ x ,ξ y ,ξ z They represent the coefficients of the total air resistance coefficient of the quadrotor drone decomposed according to the X, Y, and Z directions respectively; θ, ψ represent the roll angle, pitch angle, and yaw angle of the quadrotor drone, respectively; U1 represents the total lift generated by the propeller of the quadrotor drone; They represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time;

[0014] The constructed posture dynamics model is as follows:

[0015]

[0016] in, They represent the angular accelerations in the X, Y, and Z directions in the quadrotor drone body coordinate system; I x ,I y ,I z They are the moments of inertia of the X, Y, and Z axes in the coordinate system of the quadrotor drone body; U2, U3, and U4 respectively represent the virtual control quantities in the X, Y, and Z directions in the coordinate system of the quadrotor drone body; They represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone respectively; J RP represents the total moment of inertia of the entire motor rotor and propeller around the body axis; kθ, kψ represents the components of the total drag torque coefficient in the X, Y, and Z axes in the body coordinate system; Ω represents the propeller angular velocity vector.

[0017] Furthermore, when constructing the model predictive controller, the quadrotor UAV dynamics model is combined with the quadrotor UAV kinematics model to obtain a complete motion model of the UAV; the quadrotor UAV kinematics model is:

[0018]

[0019] in, They represent the derivatives of the position of the drone on the X, Y, and Z axes with respect to time in the ground coordinate system; v x ,v y ,v z They represent the speed of the quadrotor drone in the X, Y, and Z directions in the ground coordinate system respectively; They respectively represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone with respect to time; p, q, and r respectively represent the angular velocities of the pitch angle, roll angle, and yaw angle of the quadrotor drone.

[0020] Furthermore, when establishing the model predictive control optimization problem, the UAV model predicts the discretized kinematic model as the state equation, and the UAV model predicts the discretized kinematic model is:

[0021] X(k+1)=AX(k)+BU(k)

[0022] Where:

[0023]

[0024] U=[F1,F2,F3,F4] T

[0025] A satisfies:

[0026]

[0027] B satisfies:

[0028]

[0029] Where k represents the kth moment of the entire system; k+1 represents the moment after 1 time interval from moment k; N represents the length of the prediction domain; They represent the derivatives of the X, Y, and Z axis positions of the UAV in the ground coordinate system with respect to time; x, y, and z represent the positions of the quadrotor UAV in the ground coordinate system; They represent the time derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone respectively; θ, ψ represent the roll angle, pitch angle, and yaw angle of the quadrotor drone, respectively; F1, F2, F3, and F4 represent the control components of the four propellers of the quadrotor drone; I x ,I y ,I z They are the X, Y, and Z axis moments of inertia in the quadcopter body coordinate system; X1, X2, X3, X4, X5, and X6 represent the first six elements of X, namely roll angle, roll angular velocity, pitch angle, pitch angular velocity, yaw angle, and yaw angular velocity; m represents the mass of the quadcopter; U1 represents the total lift generated by the propellers of the quadcopter.

[0030] Furthermore, the model predictive control optimization problem is established according to the UAV model predictive control performance index, and the UAV model predictive control performance index is:

[0031]

[0032] Among them, X(k) represents the position state information of the UAV at time k; U(k) represents the control input information of the UAV at time k; Q k and p k They represent the state error gain coefficient and control input gain coefficient in the cost function, Q k and p k satisfy:

[0033]

[0034] Where Q and p represent the state error gain coefficient and control input gain coefficient in the preset cost function respectively; S = diag[S Q ,S p ], S represents the total cost function gain coefficient matrix, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix respectively, which are given by the proximal strategy optimization algorithm.

[0035] Furthermore, the action space of the proximal strategy optimization algorithm is:

[0036] Action=[s1,s2,s3...s n ]

[0037] Where n = R(S), n represents the number of elements in the action space, and R(S) represents the rank of the total cost function gain coefficient matrix; the upper and lower limits of the action space are (-∞, +∞) and conform to the Gaussian distribution;

[0038] According to the requirements of the weight matrix, the action space Action is cropped and numerically scaled to obtain:

[0039] Action'=[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ,s p,1 ,s p,2 ,s p,3 ...s p,k ]

[0040] Where j = R(S Q ), k = R(S p ), j represents the number of elements in the action space that affect the optimization matrix of the state error weights, k represents the number of elements in the action space that affect the optimization matrix of the control input weights, R(S Q ) represents the rank of the state error weight optimization matrix, R(S p ) represents the rank of the control input weight optimization matrix; and then the weight optimization matrix is ​​obtained:

[0041] S Q =diag[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ]

[0042] S p =diag[s p,1 ,s p,2 ,s p,3 ...s p,k ]

[0043] Among them, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix respectively.

[0044] Furthermore, the reward function of the proximal strategy optimization algorithm is:

[0045]

[0046] Among them, G represents the total gain coefficient of the reward function, λ represents the error penalty coefficient, l represents the control domain length, and x i,t ,y i,t ,z i,t represents the desired position, x i ,y i ,z i represents the current actual position, and i represents the i-th moment in the control domain.

[0047] Furthermore, the critic network adjusts parameters by optimizing the loss function to maximize the reward function; through iterative training and optimization, the critic network enables the reward function to guide the model to optimize performance and dynamically adjust the model predictive controller.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention combines reinforcement learning with MPC to provide a more flexible, intelligent and robust solution for drone trajectory tracking. The solution uses the decision-making mechanism of reinforcement learning to optimize the parameters of the MPC controller, so that it can better adapt to complex and changing environments and mission requirements. When the drone performs a patrol mission, it can adjust the optimal action in real time according to the current state to ensure that the drone completes the patrol mission accurately and safely. The present invention not only solves the problem of difficult controller parameter selection in existing drone control methods, but also improves the processing capability and stability of drones facing different mission environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flow chart of the method of the present invention.

[0051] Figure 2 This is a schematic diagram of the discretized kinematic model for predicting the UAV model in the present invention.

[0052] Figure 3 Schematic diagram of the reinforcement learning and model predictive control network in the present invention.

[0053] Figure 4 This is the target trajectory tracked by the drone in Example 1, where the blue line is the target trajectory and the orange line is the drone motion trajectory.

[0054] Figure 5 is the trajectory tracking error of the UAV in Example 1. DETAILED DESCRIPTION

[0055] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below, but it should not be construed as limiting the applicable scope of the present invention.

[0056] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0057] An embodiment of the present invention provides a reinforcement learning optimization UAV trajectory tracking model predictive control method, the flow chart of which is as follows: Figure 1 The method comprises the following steps:

[0058] Step S1, establishing a quadrotor UAV dynamics model, including a position dynamics model and an attitude dynamics model;

[0059] Step S2: Based on the UAV dynamics model, a model predictive controller is constructed, and a model predictive control optimization problem is established to obtain a UAV trajectory tracking cost function;

[0060] Step S3: construct a proximal policy optimization (PPO) algorithm network, including an actor network and a critic network, adjust the weight matrix of the model predictive controller cost function through the proximal policy optimization algorithm, and solve the model predictive control optimization problem to obtain the optimal control input.

[0061] The details are as follows:

[0062] 1) In step S1, the position dynamics model constructed is as follows:

[0063]

[0064] in, They represent the displacement acceleration of the UAV in the X, Y, and Z directions in the ground coordinate system; m represents the mass of the quadrotor UAV; g is the gravitational acceleration; ξ x ,ξy ,ξ z They represent the coefficients of the total air resistance coefficient of the quadrotor drone decomposed according to the X, Y, and Z directions respectively; θ, ψ represent the roll angle, pitch angle, and yaw angle of the quadrotor drone, respectively; U1 represents the total lift generated by the propeller of the quadrotor drone; They represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time;

[0065] The constructed posture dynamics model is as follows:

[0066]

[0067] in, They represent the angular accelerations in the X, Y, and Z directions in the quadrotor drone body coordinate system; I x ,I y ,I z They are the moments of inertia of the X, Y, and Z axes in the coordinate system of the quadrotor drone body; U2, U3, and U4 respectively represent the virtual control quantities in the X, Y, and Z directions in the coordinate system of the quadrotor drone body; They represent the time derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone respectively; J RP represents the total moment of inertia of the entire motor rotor and propeller around the body axis; kθ, kψ represents the components of the total drag torque coefficient in the X, Y, and Z axes in the body coordinate system; Ω represents the propeller angular velocity vector.

[0068] 2. In step S2:

[0069] (1) When constructing the model predictive controller, the quadrotor UAV dynamics model is combined with the quadrotor UAV kinematic model to obtain a complete motion model of the UAV; the quadrotor UAV kinematic model is:

[0070]

[0071] in, They represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time; v x ,v y ,v z They represent the speed of the quadrotor drone in the X, Y, and Z directions in the ground coordinate system respectively; They respectively represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone with respect to time; p, q, and r respectively represent the angular velocities of the pitch angle, roll angle, and yaw angle of the quadrotor drone.

[0072] (2) When establishing the model predictive control optimization problem, the UAV model predicts the discretized kinematic model as the state equation. The schematic diagram is as follows: Figure 2 As shown, the discretized kinematic model predicted by the UAV model is:

[0073] X(k+1)=AX(k)+BU(k)

[0074] Where:

[0075]

[0076]

[0077] U=[F1,F2,F3,F4] T

[0078] A satisfies:

[0079]

[0080]

[0081] B satisfies:

[0082]

[0083] Where k represents the kth moment of the entire system; k+1 represents the moment after a time interval of k; N represents the length of the prediction domain; They represent the derivatives of the X, Y, and Z axis positions of the UAV in the ground coordinate system with respect to time; x, y, and z represent the positions of the quadrotor UAV in the ground coordinate system; They represent the time derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone respectively; θ, ψ represent the roll angle, pitch angle, and yaw angle of the quadrotor drone, respectively; F1, F2, F3, and F4 represent the control components of the four propellers of the quadrotor drone; I x ,I y ,I z They are the X, Y, and Z axis moments of inertia in the quadcopter body coordinate system; X1, X2, X3, X4, X5, and X6 represent the first six elements of X, namely roll angle, roll angular velocity, pitch angle, pitch angular velocity, yaw angle, and yaw angular velocity; m represents the mass of the quadcopter; U1 represents the total lift generated by the propellers of the quadcopter.

[0084] (3) The model predictive control optimization problem is established based on the UAV model predictive control performance index. The UAV model predictive control performance index is:

[0085]

[0086] Among them, X(k) represents the position state information of the UAV at time k; U(k) represents the control input information of the UAV at time k; Q k and p k They represent the state error gain coefficient and control input gain coefficient in the cost function, Q k and p k satisfy:

[0087]

[0088] Where Q and p represent the state error gain coefficient and control input gain coefficient in the preset cost function respectively; S = diag[S Q ,S p ], S represents the total cost function gain coefficient matrix, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix respectively, which are given by the proximal strategy optimization algorithm.

[0089] 3. In the step S3;

[0090] (1) The action space of the proximal strategy optimization algorithm is:

[0091] Action=[s1,s2,s3...s n ]

[0092] Where n = R(S), n represents the number of elements in the action space, and R(S) represents the rank of the total cost function gain coefficient matrix; the upper and lower limits of the action space are (-∞, +∞) and conform to the Gaussian distribution;

[0093] According to the requirements of the weight matrix, the action space Action is cropped and numerically scaled to obtain:

[0094] Action'=[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ,s p,1 ,s p,2 ,s p,3 ...s p,k ]

[0095] Where j = R(S Q ), k = R(S p ), j represents the number of elements in the action space that affect the optimization matrix of the state error weights, k represents the number of elements in the action space that affect the optimization matrix of the control input weights, R(S Q ) represents the rank of the state error weight optimization matrix, R(S p) represents the rank of the control input weight optimization matrix; and then the weight optimization matrix is ​​obtained:

[0096] S Q =diag[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ]

[0097] S p =diag[s p,1 ,s p,2 ,s p,3 ...s p,k ]

[0098] Among them, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix respectively.

[0099] (2) The reward function of the proximal strategy optimization algorithm is:

[0100]

[0101] Where G and λ represent positive gain parameters, l represents the control domain length, and x i,t ,y i,t ,z i,t represents the desired position, x i ,y i ,z i represents the current actual position, and i represents the i-th moment in the control domain.

[0102] (3) The critic network adjusts parameters by optimizing the loss function so that the reward function approximates the maximum. In this way, the advantages of model-free reinforcement learning are combined with the robustness and online replanning capabilities of model predictive control, and the output model-based controller can ensure that the current output is feasible. Through continuous iterative training and optimization, the parameters of the critic network will gradually adjust so that the reward function can better guide the model to obtain better performance in different tasks and environments, thereby dynamically optimizing the performance of the model predictive controller. In addition, in order to improve the overall computing efficiency, the embodiment of the present invention also adopts a method of expanding the prediction domain and the control domain to improve the versatility of the controller in facing different tasks and the accuracy of tracking under long control domain conditions.

[0103] like Figure 3As shown in the figure, the workflow of the system is as follows: the expected trajectory set by the system can obtain the position information of the trajectory points at each moment, the environment module obtains the expected position information of the drone and the actual position information and error of the drone under the current weight matrix, and stores the historical data of the interaction between the drone and the environment (including state, action and reward information) in the Rollout buffer. The actor network in the Actor-Critic network structure determines the action of the weight optimization matrix according to the current drone state and environmental information to optimize the cost function of the model predictive control. The model predictive controller predicts the future state of the system based on the current system state and input, obtains the control input sequence in the prediction domain through the cost function, and transmits the control domain part of the optimal control sequence as the control input information to the drone, thereby obtaining the actual position information of the drone. The environment module gives a reward value to the action of the weight optimization matrix according to the position error and the reward function. The critic network uses the advantage function based on the reward function to update its own parameters, thereby helping the actor network to update the policy parameters. Through the continuous optimization and update of the weight matrix, at the next moment, the above steps are repeated to achieve the rolling optimization of the cost function and improve the drone trajectory tracking performance.

[0104] Example 1: Select a quadrotor drone system and set detailed system parameters for it: m = 1 kg, g = 9.8 m / s 2 , I x =4×10 -3 kg·m 2 , I y =4×10 -3 kg·m 2 , I z =8.4×10 -3 kg·m 2 , l a =0.2m, b=2.9×10 -5 N·s 2 , d = 1.1 × 10 -6 N·s 2 , where m is the mass of the quadcopter, g is the acceleration of gravity, and I x ,I y and I z are the moments of inertia of the x, y, and z axes respectively, b is the thrust coefficient, and d is the drag coefficient.

[0105] The expected trajectory is set as:

[0106]

[0107] Wherein, t=i×dt, i is the i-th target point, and dt=0.02s.

[0108] Set the weight matrix of the position loop and attitude loop: the selected position loop weight matrix Q P =diag{40, 40, 40, 1, 1, 1}, R P =diag{1,1,1}, attitude ring weight matrix Q A =diag{40, 40, 40, 1, 1, 1}, R A =diag{1,1,1}.

[0109] Finally, the drone successfully tracked the target trajectory, such as Figure 4 As shown in the figure. The blue line in the figure represents the target trajectory, and the orange line represents the motion trajectory of the drone. It can be found from the figure that the motion trajectory of the drone almost coincides with the target trajectory, which shows that the trajectory tracking performance of the controller is very good. In addition, the trajectory tracking error of the drone within 0-14s is analyzed, as shown in Figure 5 As shown in the figure, the trajectory tracking error can be kept below a low value, which further proves the stability and accuracy of the controller.

[0110] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A reinforcement learning optimized UAV trajectory tracking model predictive control method, characterized in that: The following steps are involved: Step S1, establishing a quadrotor UAV dynamics model, including a position dynamics model and an attitude dynamics model; Step S2: Based on the UAV dynamics model, a model predictive controller is constructed, and a model predictive control optimization problem is established to obtain a UAV trajectory tracking cost function; Step S3: construct a proximal policy optimization algorithm network, including an actor network and a critic network, adjust the weight matrix of the model predictive controller cost function through the proximal policy optimization algorithm, and solve the model predictive control optimization problem to obtain the optimal control input.

2. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 1 is characterized in that: In step S1, the position dynamics model constructed is as follows: in, They represent the displacement acceleration of the UAV in the X, Y, and Z directions in the ground coordinate system; m represents the mass of the quadrotor UAV; g is the gravitational acceleration; ξ x ,ξ y ,ξ z They represent the coefficients of the total air resistance coefficient of the quadrotor drone decomposed according to the X, Y, and Z directions respectively; They represent the roll angle, pitch angle, and yaw angle of the quadrotor drone respectively; U1 represents the total lift generated by the propeller of the quadrotor drone; They represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time; The constructed posture dynamics model is as follows: in, They represent the angular accelerations in the X, Y, and Z directions in the quadrotor drone body coordinate system; I x ,I y ,I z They are the moments of inertia of the X, Y, and Z axes in the coordinate system of the quadrotor drone body; U2, U3, and U4 respectively represent the virtual control quantities in the X, Y, and Z directions in the coordinate system of the quadrotor drone body; They represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone respectively; J RP It represents the total moment of inertia of the entire motor rotor and propeller around the body axis; They respectively represent the components of the total drag torque coefficient on the X, Y, and Z axes in the body coordinate system; Ω represents the propeller angular velocity vector.

3. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 1 is characterized in that: When constructing the model predictive controller, the quadrotor UAV dynamics model is combined with the quadrotor UAV kinematics model to obtain a complete motion model of the UAV; the quadrotor UAV kinematics model is: in, They represent the derivatives of the X, Y, and Z axis positions of the drone in the ground coordinate system with respect to time; v x ,v y ,v z They represent the speed of the quadrotor drone in the X, Y, and Z directions in the ground coordinate system respectively; They respectively represent the derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone with respect to time; p, q, and r respectively represent the angular velocities of the pitch angle, roll angle, and yaw angle of the quadrotor drone.

4. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 1 is characterized in that: When establishing the model predictive control optimization problem, the UAV model predictive discretized kinematics model is used as the state equation. The UAV model predictive discretized kinematics model is: X(k+1)=AX(k)+BU(k) Where: U=[F1,F2,F3,F4] T A satisfies: B satisfies: Where k represents the kth moment of the entire system; k+1 represents the moment after a time interval of k; N represents the length of the prediction domain; They represent the derivatives of the X, Y, and Z axis positions of the UAV in the ground coordinate system with respect to time; x, y, and z represent the positions of the quadrotor UAV in the ground coordinate system; They represent the time derivatives of the pitch angle, roll angle, and yaw angle of the quadrotor drone respectively; They represent the roll angle, pitch angle, and yaw angle of the quadrotor drone respectively; F1, F2, F3, and F4 represent the control components of the four propellers of the quadrotor drone; I x ,I y ,I z They are the X, Y, and Z axis moments of inertia in the quadcopter body coordinate system; X1, X2, X3, X4, X5, and X6 represent the first six elements of X, namely roll angle, roll angular velocity, pitch angle, pitch angular velocity, yaw angle, and yaw angular velocity; m represents the mass of the quadcopter; U1 represents the total lift generated by the propellers of the quadcopter.

5. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 4 is characterized in that: The model predictive control optimization problem is established according to the UAV model predictive control performance index, and the UAV model predictive control performance index is: Among them, X(k) represents the position state information of the UAV at time k; U(k) represents the control input information of the UAV at time k; Q k and p k They represent the state error gain coefficient and control input gain coefficient in the cost function, respectively, Q k and p k satisfy: Where Q and p represent the state error gain coefficient and control input gain coefficient in the preset cost function respectively; S = diag[S Q ,S p ], S represents the total cost function gain coefficient matrix, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix respectively, which are given by the proximal strategy optimization algorithm.

6. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 1 is characterized in that: The action space of the proximal strategy optimization algorithm is: Action=[s1,s2,s3...s n ] Where n = R(S), n represents the number of elements in the action space, and R(S) represents the rank of the total cost function gain coefficient matrix; the upper and lower limits of the action space are (-∞, +∞) and conform to the Gaussian distribution; According to the requirements of the weight matrix, the action space Action is cropped and numerically scaled to obtain: Action'=[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ,s p,1 ,s p,2 ,s p,3 ...s p,k ] Where j = R(S Q ), k = R(S p ), j represents the number of elements in the action space that affect the optimization matrix of the state error weights, k represents the number of elements in the action space that affect the optimization matrix of the control input weights, R(S Q ) represents the rank of the state error weight optimization matrix, R(S p ) represents the rank of the control input weight optimization matrix; and then the weight optimization matrix is ​​obtained: S Q =diag[s Q,1 ,s Q,2 ,s Q,3 ...s Q,j ] S p =diag[s p,1 ,s p,2 ,s p,3 ...s p,k ] Among them, S Q and S p They represent the state error weight optimization matrix and the control input weight optimization matrix respectively.

7. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 6 is characterized in that: The reward function of the proximal policy optimization algorithm is: Among them, G represents the total gain coefficient of the reward function, λ represents the error penalty coefficient, l represents the control domain length, and x i,t ,y i,t ,z i,t represents the desired position, x i ,y i ,z i represents the current actual position, and i represents the i-th moment in the control domain.

8. The reinforcement learning optimized UAV trajectory tracking model predictive control method according to claim 7 is characterized in that: The critic network adjusts parameters by optimizing the loss function to maximize the reward function; through iterative training and optimization, the critic network enables the reward function to guide the model to optimize performance and dynamically adjust the model predictive controller.

Images