A deep-RLMPC-based unmanned aerial vehicle trajectory tracking control method
By constructing an explicit nonlinear dynamic model of the UAV based on the Deep-RLMPC method and expanding the state space, and combining it with a deep neural network to approximate the value function, the problems of high computational complexity and insufficient stability in UAV trajectory tracking control are solved, and efficient and wind-resistant UAV control is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2026-04-29
- Publication Date
- 2026-07-14
Smart Images
Figure CN122387134A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of unmanned aerial vehicle (UAV) control and artificial intelligence technology, specifically to a UAV trajectory tracking control method based on Deep-RLMPC. Background Technology
[0002] With the development of UAV technology, high-precision trajectory tracking and attitude control of UAVs in complex environments has become a research hotspot. UAVs have complex aerodynamic characteristics such as high nonlinearity, strong coupling, and underactuation, and are extremely susceptible to interference from random gusts and turbulence during actual flight.
[0003] To ensure closed-loop stability, traditional MPC typically requires the design of complex terminal costs and constraints, or the use of extremely long prediction steps. For high-dimensional nonlinear UAVs, designing the terminal set is extremely difficult; and extremely long prediction steps lead to an exponential increase in computational complexity, making it difficult to achieve high-performance online solutions on low-computing-power airborne platforms. Pure black-box models lack rigorous mathematical guarantees of stability and convergence. Furthermore, RL algorithms are prone to high-frequency oscillations in continuous action spaces, leading to severe wear on actuators such as ailerons and elevators. Simultaneously, under unmodeled disturbances such as gusts, the system degenerates into a partially observable Markov decision process, resulting in a significant decrease in pure RL control performance. Existing research utilizes RL to learn terminal costs online to relax MPC terminal constraints, but often employs high-order polynomials for value function approximation. Due to the extremely high dimensionality of the UAV state space, polynomial approximation suffers from a severe "curse of dimensionality," making practical deployment impossible.
[0004] To address the aforementioned issues, there is an urgent need for a UAV control framework that not only possesses rigorous theoretical safety guarantees and can adapt to high-dimensional complex dynamics, but also boasts high computational efficiency and strong resistance to wind disturbances. Summary of the Invention
[0005] The purpose of this invention is to provide a UAV trajectory tracking control method based on Deep-RLMPC. By introducing a deep neural network for value function approximation and combining it with an explicit physical model for short-step predictive control, this method solves the problems of high computational cost in traditional MPC and lack of physical security and the curse of dimensionality in pure RL.
[0006] To achieve the above objectives, this invention provides a UAV trajectory tracking control method based on Deep-RLMPC, comprising the following steps:
[0007] Step S1: Construct an explicit nonlinear dynamic prediction model and extended state space for the UAV
[0008] Obtain the continuous state characteristics of the UAV and establish an explicit nonlinear kinematic and dynamic prediction model based on the Newton-Euler equations. .
[0009] To overcome the partially observable Markov decision process problem caused by wind disturbance, the system state is... The dimension is expanded so that the state feature input vector includes not only the basic physical state of the UAV, but also the integral error of trajectory tracking and the current position of the actuator at the previous moment.
[0010] The extended-dimensional state feature vector The mathematical expression is:
[0011] Here, represents the tracking error between the actual flight trajectory of the UAV and the desired waypoint; represents the integral term of the trajectory tracking error; and represents the actual physical deflection command of the actuator at the previous moment. By introducing the integral error and the actuator position at the previous moment into the extended-dimensional state, the system acquires implicit sensing capabilities to cope with some observable Markov decision process characteristics in windy environments.
[0012] Step S2: Build a policy generator based on Deep-RLMPC
[0013] Construct a finite-step prediction optimization problem with a learning terminal cost. In each control cycle... The policy generator predicts the step size. Solve the following cost function minimization problem to obtain the optimal control sequence:
[0014]
[0015]
[0016] in, The control sequence for the forecast period; For the first period of the forecast The basic operational cost of each step; To predict the end of the period, the parameters are: Terminal value functions for parameterization of deep neural networks; For action change penalties, among which This is the weight matrix. This represents the increment of the predictive control command between two adjacent steps, specifically when... hour, This explicitly constrains the deflection amplitude of the UAV's ailerons and elevator, suppressing high-frequency oscillations.
[0017] Step S3: Apply control inputs and collect flight trajectory data
[0018] The first term of the control sequence obtained in step S2 is sent as the actual control command to the UAV actuator. The UAV interacts with the real environment and records the state transition data tuples.
[0019]
[0020] Stored in the experience replay pool. This refers to the cost of a single-step execution.
[0021] The terminal value function Constructed by a multilayer perceptron, the multilayer perceptron uses extended-dimensional state feature vectors. As an input layer, it outputs a scalar value to approximate the expected value of the system's cumulative optimal cost from the end of the prediction period to infinity, i.e.:
[0022] in Discount factor, prediction step size The range of values is Furthermore, the rolling optimization solution is performed on the UAV airborne end through a nonlinear programming solver or a sampling-based model prediction path integral solution.
[0023] When using the model prediction path integral solver for forward candidate trajectory sampling, the worst-case wind disturbance boundary is superimposed on the explicit prediction model, and the model expression is modified as follows:
[0024] in, Let be the applied random gust or turbulent disturbance vector, and satisfy . , This represents the known maximum wind disturbance margin limit.
[0025] Step S4: Construct a policy evaluator based on depth-valued function approximation
[0026] Randomly sample a small batch of data tuples from the experience replay pool, and calculate the loss function using the temporal difference objective:
[0027] in, As a discount factor, The objective value network is used. An adaptive moment estimation optimizer is employed for gradient descent to update the parameters of the deep terminal value function network. Utilizing the updated Replace the terminal cost in step S2 to complete the policy iteration. Attached Figure Description
[0029] The accompanying drawings are provided to further illustrate the present invention and form part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, but do not constitute a limitation on the technical solutions of the present invention.
[0030] Figure 1 This is a flowchart of a UAV trajectory tracking control method based on Deep-RLMPC according to the present invention; Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0033] like Figure 1 As shown, this invention provides a UAV trajectory tracking control method based on Deep-RLMPC, which includes the following steps:
[0034] Step S1: Construct an explicit nonlinear dynamic prediction model and extended state space for the UAV
[0035] Obtain the continuous state characteristics of the UAV and establish an explicit nonlinear kinematic and dynamic prediction model based on the Newton-Euler equations. .
[0036] To overcome the partially observable Markov decision process problem caused by wind disturbance, the system state is... The dimension is expanded so that the state feature input vector includes not only the basic physical state of the UAV, but also the integral error of trajectory tracking and the current position of the actuator at the previous moment.
[0037] The extended-dimensional state feature vector The mathematical expression is:
[0038]
[0039] in, The tracking error between the actual flight trajectory of the UAV and the desired waypoint. This is the integral term of the trajectory tracking error. The actual physical deflection command of the actuator at the previous moment is given; by introducing the integral error and the actuator position at the previous moment into the extended-dimensional state, the system has the implicit perception capability to cope with the characteristics of some observable Markov decision processes in wind-induced disturbance environments.
[0040] Step S2: Build a policy generator based on Deep-RLMPC
[0041] Construct a finite-step prediction optimization problem with a learning terminal cost. In each control cycle... The policy generator predicts the step size. Solve the following cost function minimization problem to obtain the optimal control sequence:
[0042]
[0043]
[0044]
[0045] in, The control sequence for the forecast period; For the first period of the forecast The basic operational cost of each step; To predict the end of the period, the parameters are: Terminal value functions for parameterization of deep neural networks; For action change penalties, among which This is the weight matrix. This represents the increment of the predictive control command between two adjacent steps, specifically when... hour, This explicitly constrains the deflection amplitude of the UAV's ailerons and elevator, suppressing high-frequency oscillations.
[0046] Step S3: Apply control inputs and collect flight trajectory data
[0047] The first term of the control sequence obtained in step S2 is sent as the actual control command to the UAV actuator. The UAV interacts with the real environment and records the state transition data tuples.
[0048]
[0049] Stored in the experience replay pool. This refers to the cost of a single-step execution.
[0050] The terminal value function Constructed by a multilayer perceptron, the multilayer perceptron uses extended-dimensional state feature vectors. As an input layer, it outputs a scalar value to approximate the expected value of the system's cumulative optimal cost from the end of the prediction period to infinity, i.e.:
[0051] in Discount factor, prediction step size The range of values is Furthermore, the rolling optimization solution is performed on the UAV airborne end through a nonlinear programming solver or a sampling-based model prediction path integral solution.
[0052] When using the model prediction path integral solver for forward candidate trajectory sampling, the worst-case wind disturbance boundary is superimposed on the explicit prediction model, and the model expression is modified as follows:
[0053] in, Let be the applied random gust or turbulent disturbance vector, and satisfy . , This represents the known maximum wind disturbance margin limit.
[0054] Step S4: Construct a policy evaluator based on depth-valued function approximation
[0055] Randomly sample a small batch of data tuples from the experience replay pool, and calculate the loss function using the temporal difference objective:
[0056] in, As a discount factor, The objective value network is used. An adaptive moment estimation optimizer is employed for gradient descent to update the parameters of the deep terminal value function network. Utilizing the updated Replace the terminal cost in step S2 to complete the policy iteration.
Claims
1. A UAV trajectory tracking control method based on Deep-RLMPC, characterized in that, Includes the following steps: Step S1: Obtain the continuous state of the UAV, establish an explicit prediction model based on the UAV's kinematics and dynamics equations, and extract the current time. Extended-dimensional state feature vector of UAV ; Step S2: Construct a finite-step model predictive control problem that combines an explicit prediction model with the terminal cost of a deep network, at the current time step. Based on the extended-dimensional state feature vector To predict step size Online solution makes the total cost function Minimize the optimal control sequence The first term of the optimal control sequence is used as the actual control input. Applied to drones; Step S3: The UAV receives control input The interaction with the environment under the influence of the current action transitions to the state of the next time step, and the actual running cost of the current step is calculated. and will include data tuples containing the interaction process Store in the experience replay pool middle; Step S4: From the experience playback pool Randomly sample batches of data, construct a loss function based on the temporal difference algorithm, and perform closed-loop gradient updates on the parameters of the terminal cost of the deep network to complete the policy iteration.
2. The UAV trajectory tracking and control method based on Deep-RLMPC according to claim 1, characterized in that, The explicit prediction model described in step S1 is expressed as follows: in, For drones The physical state vector at time t. To control the input vector; The extended-dimensional state feature vector The mathematical expression is: in, The tracking error between the actual flight trajectory of the UAV and the desired waypoint. This is the integral term of the trajectory tracking error. The actual physical deflection command of the actuator at the previous moment is given; by introducing the integral error and the actuator position at the previous moment into the extended-dimensional state, the system has the implicit perception capability to cope with the characteristics of some observable Markov decision processes in wind-induced disturbance environments.
3. The UAV trajectory tracking and control method based on Deep-RLMPC according to claim 1, characterized in that, The total cost function for optimizing the finite-step model predictive control problem described in step S2 is: The expression is as follows: in, The control sequence for the forecast period; For the first period of the forecast The basic operational cost of each step; To predict the end of the period, the parameters are: Terminal value functions for parameterization of deep neural networks; For action change penalties, among which This is the weight matrix. This represents the increment of the predictive control command between two adjacent steps, specifically when... hour, This explicitly constrains the deflection amplitude of the UAV's ailerons and elevator, suppressing high-frequency oscillations.
4. The UAV trajectory tracking and control method based on Deep-RLMPC according to claim 3, characterized in that, The terminal value function Constructed by a multilayer perceptron, the multilayer perceptron uses extended-dimensional state feature vectors. As an input layer, it outputs a scalar value to approximate the expected value of the system's cumulative optimal cost from the end of the prediction period to infinity, i.e.: in Discount factor, prediction step size The range of values is Furthermore, the rolling optimization solution is performed on the UAV airborne end through a nonlinear programming solver or a sampling-based model prediction path integral solution. When using the model prediction path integral solver for forward candidate trajectory sampling, the worst-case wind disturbance boundary is superimposed on the explicit prediction model, and the model expression is modified as follows: in, Let be the applied random gust or turbulent disturbance vector, and satisfy . , This represents the known maximum wind disturbance margin limit.
5. The UAV trajectory tracking and control method based on Deep-RLMPC according to claim 4, characterized in that, The loss function described in step S4 based on The mathematical expression for constructing the step-time difference objective is as follows: in, To replay experience pool The length of the segment extracted is Historical real flight trajectory data; for The actual running cost per step calculated at each moment; Discount factor; For the target network to be introduced, its parameters By analyzing network parameters Perform exponential moving average smoothing updates.