Quadrotor unmanned aerial vehicle adaptive model predictive control method based on linear actor-critic framework
By adjusting the weights of the MPC controller online using a lightweight linear Actor-Critic framework, the tracking accuracy and energy consumption problems of traditional quadcopter UAVs in disturbed environments are solved, achieving efficient and robust adaptive control.
Patent Information
- Application Number
- CN202610388474.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-27
- Publication Date
- 2026-06-19
AI Technical Summary
Traditional quadcopter drone controllers struggle to maintain optimal control performance with fixed control parameters when faced with unknown wind disturbances and load changes, leading to decreased tracking accuracy and increased energy consumption. Furthermore, existing reinforcement learning schemes are computationally complex and difficult to learn online.
A lightweight linear Actor-Critic framework is adopted, and the controller performance is optimized by adjusting the weight parameters of the MPC controller online and combining a multi-objective reward function and a directional exploration mechanism.
It significantly improves the tracking accuracy and robustness of UAVs under disturbances, ensuring the practicality and reliability of the algorithm, and enabling it to autonomously maintain excellent tracking performance in complex environments.
Smart Images

Figure CN122239771A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the interdisciplinary field of automatic control, reinforcement learning and UAV navigation control, and specifically relates to an adaptive model predictive control method for quadrotor UAVs based on the linear Actor-Critic framework. Background Technology
[0002] Quadrone UAVs face significant challenges in control design due to their strong coupling, nonlinearity, and underactuated dynamics. Model predictive control (MPC) is widely used in this field because of its ability to handle constraints and perform look-ahead optimization. However, the control parameters of traditional MPCs (such as the weight matrix in the cost function) are usually fixed after offline design. When the aircraft encounters uncertainties such as unknown wind disturbances and load changes in real-world applications, fixed controller parameters are difficult to maintain optimal control performance, leading to decreased tracking accuracy, increased energy consumption, and even instability.
[0003] While some research has attempted to introduce reinforcement learning (RL) frameworks to provide adaptive capabilities for UAV controllers, most solutions rely on computationally complex deep neural networks (DNNs) as function approximators. These methods suffer from problems such as unstable training, high computational resource requirements, and difficulty in implementing online learning on airborne platforms, which greatly limits their application in engineering practice. Summary of the Invention
[0004] The purpose of this invention is to provide a computationally efficient, robust, and suitable adaptive control method for quadrotor unmanned aerial vehicles (UAVs). Through a lightweight linear adaptive architecture, the weight parameters of the MPC controller are optimized in real time, enabling the UAV to autonomously maintain excellent tracking performance in complex disturbance environments.
[0005] The "linear Actor-Critic framework" described in this invention refers to its core learnable part (the weight matrix of the Actor module). The weight vector of the Critic module and bias parameters There is a linear relationship between the input and output. Although nonlinear functions such as tanh are introduced at the feature input or output for normalization, the core mechanism of its adaptive adjustment function relies on the updating and action of these linear parameters, thus distinguishing it from deep neural network frameworks that rely on multi-layer nonlinear transformations.
[0006] The technical solution to achieve the purpose of this invention is: an adaptive model predictive control method for quadrotor UAVs based on a linear Actor-Critic framework, comprising the following steps:
[0007] Step (1): Establish a nonlinear dynamic model of the quadcopter UAV considering external wind disturbance, linearize the nonlinear dynamic model at the hovering equilibrium point, and obtain a discrete state space model of the quadcopter UAV for prediction by discretization.
[0008] Step (2): Based on the discrete state space model of the quadcopter UAV, construct the MPC controller and build a linear Actor-Critic framework fused with the MPC controller;
[0009] Step (3): Observe state variables, calculate tracking error, and construct key state feature vectors based on tracking error. ;
[0010] Step (4): Extract the key state feature vectors Input the Actor module from the Actor-Critic framework to generate weighted incremental action vectors. ;
[0011] Step (5): Based on the weighted increment action vector Update the position and attitude weight vectors in the MPC state weight matrix, solve for the optimal control sequence after the update, and set the first control input... Applied to quadcopter drones;
[0012] Step (6): After executing the control command, observe the next state. Calculate instant rewards The state value is estimated by the Critic module in the Actor-Critic framework, and the time difference error is calculated. ;
[0013] Step (7): Based on time difference error The gradient descent method is used to collaboratively update the parameters of the Actor module and the Critic module;
[0014] Step (8): Iteration control: Let Return to step (3) until a termination instruction is received.
[0015] Compared with the prior art, the significant advantages of this invention are:
[0016] This invention significantly improves the tracking accuracy and robustness of UAVs under disturbances through a lightweight linear architecture, multi-objective reward function, and directional exploration mechanism, ensuring the practicality and reliability of the algorithm. Attached Figure Description
[0017] Figure 1 This is a flowchart of adaptive model predictive control for a quadcopter UAV based on the linear Actor-Critic framework.
[0018] Figure 2 This is a schematic diagram of a sudden gust of wind disturbance in the embodiment.
[0019] Figure 3 This is a comparison diagram of 3D trajectory tracking between adaptive model predictive control and standard model predictive control of a quadcopter UAV using the Actor-Critic framework in this embodiment.
[0020] Figure 4 This is a comparison diagram of X-direction position tracking between adaptive model predictive control and standard model predictive control of a quadcopter UAV using the Actor-Critic framework in this embodiment.
[0021] Figure 5 This is a comparison chart of the X-direction errors of the adaptive model predictive control and standard model predictive control of the quadcopter UAV using the Actor-Critic framework in this embodiment.
[0022] Figure 6 This is a comparison diagram of the Y-direction position tracking of the quadcopter UAV using the Actor-Critic framework in this embodiment, comparing adaptive model predictive control with standard model predictive control.
[0023] Figure 7 This is a comparison chart of the Y-direction errors of the adaptive model predictive control and standard model predictive control of the quadcopter UAV using the Actor-Critic framework in this embodiment.
[0024] Figure 8 This is a comparison diagram of Z-direction position tracking between adaptive model predictive control and standard model predictive control of a quadcopter UAV using the Actor-Critic framework in this embodiment.
[0025] Figure 9 This is a comparison chart of the Z-direction errors of the adaptive model predictive control and standard model predictive control of the quadcopter UAV using the Actor-Critic framework in this embodiment.
[0026] Figure 10 This is a comparison of the overall position error between the adaptive model predictive control and the standard model predictive control of the quadcopter UAV using the Actor-Critic framework in this embodiment.
[0027] Figure 11 This is a comparison diagram of roll angle tracking between adaptive model predictive control and standard model predictive control of a quadcopter UAV using the Actor-Critic framework in this embodiment.
[0028] Figure 12 This is a comparison diagram of pitch angle tracking between the adaptive model predictive control and the standard model predictive control of the quadcopter UAV using the Actor-Critic framework in this embodiment.
[0029] Figure 13 This is a comparison diagram of yaw angle tracking between adaptive model predictive control and standard model predictive control of a quadcopter UAV using the Actor-Critic framework in this embodiment.
[0030] Figure 14 This is a monitoring graph showing the ratio of the maximum Q value to the minimum R value in the predictive control cost function of the quadcopter UAV adaptive model in the Actor-Critic framework of the embodiment.
[0031] Figure 15 It is the adaptive change curve of the position weight component in the state weight matrix of the embodiment throughout the simulation time.
[0032] Figure 16 It is the adaptive change curve of the attitude weight component in the state weight matrix of the embodiment throughout the simulation time.
[0033] Figure 17 This is a diagram showing the weight adjustment magnitude of the adaptive model predictive control of the quadcopter UAV using the Actor-Critic framework in the embodiment. Detailed Implementation
[0034] The present invention will now be described in further detail with reference to the accompanying drawings.
[0035] The core idea of this invention is to construct an Actor-Critic (AC) learning framework based on a linear function approximator and deeply integrate it with the rolling optimization process of MPC to form an online parameter self-tuning system.
[0036] The system's workflow follows a complete closed loop: perception → decision-making → optimization → execution → evaluation → learning.
[0037] Perception: Calculate the tracking error at the current moment and extract key tracking error features. .
[0038] Decision: The feature vector constructed by the linear Actor module based on the current tracking error. The decision is made on how to adjust the weight matrix of MPC and generate weight increments. .
[0039] Optimization: MPC uses an adjusted weight matrix Perform rolling optimization, solve the QP problem, and output the optimal control command. .
[0040] Execution: Control Quantities for Quadcopter UAVs The system state evolves to the next moment. .
[0041] Evaluation: Based on the new state after execution Calculate instant rewards Error at the next time step The Critic module evaluates the long-term performance of the current control strategy and calculates the time difference error. .
[0042] Learning: Based on the Critic evaluation results, update the parameters of the Actor module and the Critic module. The Critic module updates to minimize the time difference error and updates the value function parameters. The Actor module optimizes the weight generation strategy along the policy gradient direction.
[0043] Detailed description of the technical solution
[0044] 3.1 System Modeling and MPC Design
[0045] The model of this invention is constructed based on the Newton-Euler formula, establishing a nonlinear dynamic model of a quadcopter UAV considering external wind disturbances. To meet the design requirements of linear MPC, the model is linearized near the hovering equilibrium point, and finally discretized to obtain a future state-space model for prediction.
[0046] (a) Nonlinear dynamic model
[0047] Position dynamics (in Earth's inertial coordinate system):
[0048]
[0049] in, The location of the drone's center of mass. For quality, It is the acceleration due to gravity. These are roll, pitch, and yaw angles, respectively. The total lift generated by the four rotors, The components of wind disturbance in the three translational directions are introduced into the system as additional external forces.
[0050] Attitude dynamics (in body coordinates):
[0051]
[0052] in, Angular velocity, For rotational inertia, This represents the three-axis control torque. Note: Given that the wind disturbance torque is much smaller than the control torque, it is not explicitly described in the attitude dynamics to simplify the model, but its influence will be introduced through translational coupling.
[0053] Posture kinematics (approximate under the small angle assumption):
[0054]
[0055] (b) Linearization and continuous state-space models
[0056] To design a linear MPC, the aforementioned nonlinear system needs to be linearized near the equilibrium point. The hovering state is chosen as the equilibrium point, where the attitude angle... Yaw angle It can be any value (for simplicity, it is usually taken as...). ), and both the angular velocity and linear velocity are zero.
[0057] Define the system state vector Control input vector and perturbation vector as follows:
[0058]
[0059] By linearizing the Jacobian matrix, a continuous linear time-invariant state-space model is obtained:
[0060]
[0061] The typical structure of each continuous matrix is shown below. They precisely describe the dynamic characteristics of the system near the equilibrium point:
[0062] System Matrix This describes the coupling relationship between state variables.
[0063]
[0064] Key coupling item: 4th row, 8th column Indicates pitch angle produce Directional acceleration; 5th row, 7th column Indicates the generation of roll angle Directional acceleration.
[0065] Control Matrix This describes how control inputs affect the system state.
[0066]
[0067] Key item: Row 6, Column 1 Indicates total lift Contribution to acceleration in the z-direction; elements on the diagonal of rows 10, 11, and 12. , , These represent the contributions of roll, pitch, and yaw moments to the corresponding angular accelerations, respectively.
[0068] perturbation matrix This describes how wind disturbances affect the system.
[0069]
[0070] Key items: Lines 4, 5, and 6 indicate wind disturbance. As an external force, it directly and primarily affects linear acceleration in three directions.
[0071] (c) Discretization model
[0072] Digital controllers operate in discrete time, therefore the continuous model needs to be discretized. This invention employs the zero-order hold (ZOH) method, assuming the control input is within one sampling period. The internal temperature remains constant.
[0073] The discretized state-space equation forms the basis of the MPC prediction model:
[0074]
[0075] Discrete matrices are obtained by calculating matrix exponents and their integrals:
[0076]
[0077] For smaller sampling times (For example, 0.02 seconds), a first-order approximation can be used to obtain a concise analytical form:
[0078]
[0079] Discrete matrix under this approximation It retains the same structural properties as continuous matrices and explicitly includes the sampling time. This facilitates implementation and verification.
[0080] (d) MPC controller design
[0081] Based on the discrete model described above, an MPC controller is constructed. At each sampling time... The controller solves the following finite-time optimization problem:
[0082]
[0083] in, To predict the time domain, To control the time domain, For the first The tracking error vector of the step. For the first The control input vector of the step, This is a fixed control weight matrix used to penalize the magnitude of the control input. It is usually set as a diagonal matrix to adjust the weights of each input component. In time step The adaptively adjusted state weight matrix is a time-varying state weight matrix, which is the core object of the online adaptive optimization in this invention. It determines the degree of importance that MPC attaches to different state errors.
[0084] After optimization, the optimal control sequence is obtained. The first element This process is applied to drones and repeated in the next sampling period.
[0085] 3.2 Construction and Operation of the Actor-Critic Framework
[0086] The core innovation of this invention lies in the introduction of a computationally efficient linear Actor-Critic framework, which provides online adaptive capabilities for MPC.
[0087] 3.2.1 State Feature Extraction
[0088] The Actor and Critic modules do not directly process the original 12-dimensional state vector, but instead process the key state feature vectors extracted from it that are most directly related to tracking performance. Its construction method is as follows:
[0089] 1. Calculate the original error: First, calculate the error between the current state and the reference state.
[0090]
[0091] 2. Extract key errors: Select six key error components directly related to position and attitude, and construct a vector:
[0092]
[0093] 3. Constructing Feature Vectors: Take the absolute value of the aforementioned key errors to form the final feature vectors input to the Actor and Critic modules. The absolute value is used because the MPC controller itself has the ability to handle the error direction, and its linear control law is symmetrical with respect to the error direction. Taking the absolute value reduces the dimensionality of the state space, simplifies the learning problem, and enables the network to symmetrically generalize learned experiences, significantly improving learning efficiency.
[0094]
[0095] 3.2.2 Critic Module
[0096] The task of the Critic module is to evaluate the current state. Under these conditions, the long-term expected performance following the Actor strategy is as follows.
[0097] Value function calculation:
[0098] 1. Feature Normalization: To prevent excessively large differences in the range of feature values, the input features are first normalized. Normalize.
[0099]
[0100] in It is a very small positive number to prevent the denominator from being zero.
[0101] 2. Nonlinear feature mapping: Apply the hyperbolic tangent function to the normalized features to introduce a mild nonlinearity and ensure that the output is in the range of [-1, 1].
[0102]
[0103] 3. Linear weighted sum: The mapped features are multiplied by the weight vector and a bias is added to obtain the estimated value of the state.
[0104]
[0105] Here, the value function It's about parameters. A linear function, It is a value function The weight parameter vector, Vectors Mapped to value It is a bias parameter.
[0106] Time difference error The Critic module learns by using time-difference error. This error is defined as follows:
[0107]
[0108] It can measure the difference between the current state value estimate and the value of the immediate reward plus the next state discount, where It's an instant reward. It is a discount factor. It is an estimate of the next state.
[0109] 3.2.3 Actor Module
[0110] The task of the Actor module is to determine the current state. Output an action (i.e., the MPC weight matrix) (Adjustment amount).
[0111] Policy functions and action generation:
[0112] 1. Convert the state feature vector Through the weight matrix Perform a linear transformation.
[0113]
[0114] in It is the learnable weight matrix of the Actor module at step k, used to transform the 6-dimensional state feature vector. Linear mapping to 6-dimensional motion base output The element in the i-th row and j-th column represents the contribution weight of the j-th state feature to the i-th action component.
[0115] 2. The unbounded linear output is compressed to the range [-1, 1] using the hyperbolic tangent function. This bounded output represents the basic, unexplored policy direction.
[0116]
[0117] 3. Mapping from [-1, 1] to a preset weight increment range [ , ].
[0118]
[0119] in" " represents the Hadamard product, and It is a fixed parameter vector that defines the lower limit and range of the increment.
[0120] 4. Directional Exploration Noise Vector: To encourage exploration, a directional exploration noise vector is added to the basic action vector. A directional exploration noise vector is superimposed on top. Constructing the weighted increment action vector .
[0121]
[0122] The vector is not simply random, but is composed of three weighted parts:
[0123]
[0124] in This represents the current exploration intensity, which decays as the number of time steps increases.
[0125]
[0126] When the system detects a severe disturbance (tracking error > 0.18), the exploration intensity will be temporarily increased:
[0127]
[0128] Let be a vector of random perturbation components, each of which is independent and follows a standard Gaussian distribution. This ensures the randomness and traversability of the exploration.
[0129] This is the gradient direction bias component vector. The bias strength is proportional to the relative magnitude of the error change, encouraging continued exploration in the effective direction.
[0130]
[0131] The successful experience reuse component vector is a successful experience memory maintained by the system, recording the offset direction of actions that have historically earned positive rewards relative to the deterministic policy. When generating the current exploration noise vector, an exponentially weighted average is performed on all historical successful offset directions (weights). ), and normalize to obtain successful experience directions. This component enables the system to reuse proven and effective exploration directions, avoiding repeated and invalid random trials, thereby significantly accelerating policy convergence.
[0132]
[0133] in:
[0134] This is a set of indexes of successful actions, which are the time indices corresponding to actions that have historically received positive rewards;
[0135] The action offset is the actual action executed. Its certainty part The difference between them represents the direction of exploration for that action;
[0136] As a success experience decay factor, This is used to reduce the weight of early successes, making the system focus more on recent successes;
[0137] This refers to the current moment, which is the index of the current time step.
[0138] A historical success moment is a time step index of when a successful action occurs.
[0139] 3.2.4 Module Parameter Update
[0140] The parameters of the Actor and Critic modules are updated collaboratively using the gradient descent method and time difference error.
[0141] The goal of the Critic module is to make the value estimate approximate the temporal difference objective. Parameters are updated along the direction of the negative loss gradient, specifically according to the following rules:
[0142]
[0143]
[0144] in For time difference error, The key state feature vector The eigenvectors obtained after normalization and then nonlinear mapping using the hyperbolic tangent function This refers to the learning rate of the Critic module, which is used when the actual results are better than expected. When the current value is estimated, the parameters are adjusted in a way that enhances it; otherwise, they are adjusted in a way that weakens it.
[0145] The goal of the Actor module's update is to adjust the policy towards higher values as evaluated by the Critic module. A policy gradient method is employed, utilizing time-difference error. Driver parameters updated.
[0146]
[0147] Where gradient It is a 6x6 matrix. , This is the learning rate of the Actor module. When the time difference error... When the current action is reinforced, it is more likely to be adopted in similar future states; when At this time, the current action is weakened, making it more likely to be avoided.
[0148] 3.3 Online Adjustment Mechanism of Weight Matrix
[0149] The weighted incremental action vector output by the Actor module It is 6-dimensional and is only used to update the MPC state weight matrix. The diagonal elements corresponding to the position and attitude are fixed, while the diagonal elements corresponding to the velocity and angular velocity remain fixed.
[0150] The specific adjustment rules are as follows:
[0151] 1. Define the weight vector
[0152] Let the first The weight vector of the step is Its components correspond to the MPC state weight matrix. Mid-position error Attitude angle error Weighting coefficients:
[0153]
[0154] 2. Incremental Update
[0155] The weight increment action vector output by the Actor module at the current moment. Applied to This yields a new weight vector:
[0156]
[0157] in" " indicates the Hadamard product (element-by-element multiplication);
[0158] 3. Constraint Handling
[0159] To ensure the stability of the control system and prevent the weights from being too large or too small, the calculated weight vector... Apply saturation constraints to each component:
[0160]
[0161] in This is a saturation function used to constrain the weights within preset upper and lower bounds;
[0162] 4. Construct the complete state-weight matrix
[0163] The constrained 6-dimensional weight vector Combined with fixed velocity and angular velocity weights, it forms a 12x12 diagonal matrix. :
[0164]
[0165] 3.4 Design of Multi-Objective Reward Function
[0166] Instant reward function It serves as a "command stick" guiding the learning direction of Actor-Critics. This invention designs a comprehensive multi-objective reward function, calculated as a weighted sum of the following components:
[0167]
[0168] a. Tracking and improvement rewards
[0169] 1. Design Goals
[0170] Quantitatively evaluating the performance improvement brought about by the control strategy after each step of execution directly encourages the system to reduce tracking errors.
[0171] 2. Core Mechanism
[0172] Immediate performance improvements are evaluated by calculating the relative reduction in tracking error between adjacent time steps.
[0173] Overall improvement reward: based on the L2 norm change of the overall position tracking error of the UAV.
[0174]
[0175] Horizontal Improvement Reward: Special attention is paid to improvements in tracking performance in the horizontal plane (XY direction).
[0176]
[0177] 3. Key Features
[0178] Using the tanh function: saturates the improvement amount to prevent the reward value from becoming too large and ensures learning stability.
[0179] Relative metrics: Using relative rather than absolute values of error change makes rewards instructive across different error ranges.
[0180] Emphasis Level: Through weighting coefficients This reflects a higher requirement for horizontal position tracking.
[0181] 4. Rewards Summary
[0182]
[0183] b. Steady-state performance bonus
[0184] 1. Design Goals
[0185] When the system is able to achieve and maintain a high-precision tracking state, it is given continuous positive stimulation.
[0186] 2. Core Mechanism
[0187] When the tracking error is below a certain threshold, a reward is given in a linear relationship; the smaller the error, the higher the reward.
[0188] Overall performance bonus: When the overall tracking error is below a threshold Activated at that time.
[0189]
[0190] Horizontal performance bonus: When the horizontal tracking error is below a threshold Activated at that time.
[0191]
[0192] 3. Key Features
[0193] Threshold triggering: Rewards are given only when performance reaches a certain standard, encouraging precise positioning.
[0194] Linear relationship: The reward increases linearly as the error decreases, providing a clear performance gradient signal.
[0195] Next step evaluation: Based on the error status in the next step, the good performance already achieved will be rewarded.
[0196] 4. Rewards Summary
[0197]
[0198] c. Controlling the cost of punishment
[0199] 1. Design Goals
[0200] While ensuring tracking performance, energy-saving control is encouraged and drastic changes in control commands are suppressed to improve system smoothness.
[0201] 2. Core Mechanism
[0202] Penalties are imposed from two dimensions: controlling energy consumption and controlling the rate of change.
[0203] Energy consumption penalty: proportional to the L2 norm of the control input vector, suppressing excessive control input.
[0204]
[0205] Smoothness penalty: proportional to the saturation function of the rate of change of the control input, suppressing control jitter.
[0206]
[0207] 3. Key Features
[0208] Two-dimensional penalty: It considers both the absolute size and relative change of the control quantity to comprehensively optimize control quality.
[0209] Saturation treatment: Use the tanh function on the rate of change to avoid imposing degree penalties on necessary rapid adjustments.
[0210] 4. Summary of Penalties
[0211]
[0212] d、 Ratio Penalty
[0213] 1. Design Goals
[0214] To prevent the MPC controller from becoming overly aggressive due to excessive state weights, and to maintain the numerical stability and robustness of the control system.
[0215] 2. Core Mechanism
[0216] When the MPC weight matrix With control weight matrix When the norm ratio exceeds the safety threshold, a penalty of nonlinear growth is imposed.
[0217] Safety threshold: Sets the upper limit of the ratio (e.g., 600).
[0218] Penalty calculation: A quadratic function is used, and the more the limit is exceeded, the faster the penalty increases.
[0219]
[0220] 3. Key Features
[0221] Nonlinear growth: The squared term ensures that the penalty increases sharply when there is a serious over-limit, forming a hard constraint effect.
[0222] Preventative punishment: Intervening before the system becomes unstable to prevent problems from arising in the first place.
[0223] e. Oscillation penalty
[0224] 1. Design Goals
[0225] Real-time detection and suppression of oscillation trends in the control system prevents the system from entering a state of continuous oscillation, thereby improving control smoothness and system stability.
[0226] 2. Core Mechanism
[0227] A dual detection mechanism based on the direction of error change and the degree of control activity.
[0228] Error direction detection: Calculate the current error vector Compared with the previous step error vector The number of components whose signs change between them.
[0229]
[0230] in It is an indicator function.
[0231] Control activity detection: assesses the activity level of control inputs.
[0232]
[0233] in To control the amount of input variation.
[0234] 3. Penalty Triggered
[0235] When both of the following conditions are met simultaneously, the system is determined to have an oscillating trend and a penalty is imposed:
[0236] The error sign changes frequently: (More than half of the 6 error components change direction);
[0237] Active control input: .
[0238] 4. Penalty Calculation
[0239] A combination of fixed-value penalty and error-related dynamic penalty is used:
[0240]
[0241] 5. Key Features
[0242] Double protection: It is only triggered when frequent changes in error direction and active control are detected simultaneously to avoid misjudgment;
[0243] Adaptive penalty: base penalty Ensure strong suppression, additional punishment Based on the current error magnitude, the larger the error during oscillation, the heavier the penalty.
[0244] Real-time intervention: Apply penalties promptly in the early stages of oscillation to prevent it from escalating.
[0245] f. Weight efficiency penalty
[0246] 1. Design Goals
[0247] To prevent the weight vector of MPC from unnecessarily saturating when control performance is already good, the system is ensured to retain sufficient control margin to cope with potentially larger disturbances. Simultaneously, this penalty is automatically relaxed when encountering severe disturbances such as strong winds, prioritizing the system's stability recovery.
[0248] 2. Core Mechanism
[0249] This penalty consists of two logically independent and complementary penalty mechanisms.
[0250] Mechanism 1: Penalty for a single excessively high weight
[0251] Objective: To prevent the weight of any dimension from growing too high unnecessarily and consuming too many control resources.
[0252] Triggering condition: Check the current weight vector Each component. As long as any one weighted component... With the maximum allowed value The ratio exceeded the threshold. (For example, =0.8), which triggers this penalty.
[0253] Penalty calculation: The penalty value is proportional to the average degree of exceeding the limit for all exceeding weights.
[0254]
[0255] in The penalty coefficient is... Ensure that only the portion exceeding the threshold is penalized; the weight of the portion below the threshold does not contribute to the penalty.
[0256] Mechanism 2: Penalty for overall weight saturation
[0257] Objective: To punish inefficient situations where "all control permissions have been exhausted, but the control effect is poor".
[0258] Triggering conditions: Both of the following conditions must be met simultaneously:
[0259] (1) Saturation condition: weight vector The ratio of all components to the maximum weight exceeds a lower threshold. (For example, =0.7). This indicates that all weights are at a high level.
[0260] (2) Performance conditions: Current system tracking error norm Below a performance threshold (For example, =0.2). This indicates that the system has no major disturbances and does not require such a high weight.
[0261] Penalty calculation: Once triggered, apply an additional fixed-value penalty.
[0262]
[0263] 3. Adaptive Strategy
[0264] To enable intelligent switching between the "fine-tuning" and "disturbance recovery" modes, an adaptive scaling factor is introduced. .
[0265] Normal mode: When the system tracking error is small hour, The weighted efficiency penalty takes full effect, encouraging the controller to make fine-grained and efficient adjustments.
[0266] High Wind Mode: When the system tracking error is large hour, The weight efficiency penalty is significantly reduced (by 70%), allowing the controller to increase weights "at any cost" to quickly suppress disturbances and prioritize the restoration of stability.
[0267] 4. Summary of Penalties
[0268] Final weighted efficiency penalty It is the sum of the outputs of the two penalty mechanisms mentioned above, scaled by an adaptive factor:
[0269]
[0270] Example
[0271] 1. Simulation platform and parameter settings
[0272] To verify the effectiveness of the present invention, numerical simulation was performed on a computer configured with an Intel i7-11800H processor and 32GB of memory using the MATLAB R2023b platform.
[0273] Table 1. Parameters of Quadrotor UAV Model
[0274]
[0275] Table 2 Initial MPC Controller Parameter Table
[0276]
[0277] Table 3 Actor-Critic Framework Configuration Parameters
[0278]
[0279] Table 4 Instant Rewards Operating Parameter Table
[0280]
[0281] 2. Wind disturbance settings
[0282] To verify the anti-interference performance of the method of the present invention in complex wind field environments, a sudden gust scenario was set up in the simulation. To reflect the randomness of wind disturbance, this embodiment adopts a random disturbance model of superimposed basic wind noise and sudden gust, where the basic wind noise is a continuous random Gaussian noise and the sudden gust is a randomly triggered pulse-type disturbance.
[0283] (1) Definition of perturbation vector
[0284] Let the first The wind disturbance vector of the step is:
[0285]
[0286] This vector directly acts on the perturbation term in the discrete state-space model of the quadcopter UAV. .
[0287] (2) Basic wind noise
[0288] The basic wind noise is used to simulate persistent turbulence or light wind disturbances. To reflect its randomness, it is modeled as Gaussian white noise independent of each direction.
[0289]
[0290] The covariance matrix is:
[0291]
[0292] Each noise component is sampled independently at each sampling time to ensure the continuous randomness of wind disturbance.
[0293] (3) Sudden gusts of wind
[0294] Sudden gusts are used to simulate sudden strong wind interference. To reflect the randomness of their suddenness, a random triggering mechanism is adopted, with each direction controlled independently. The occurrence time and duration of the gusts are determined by a random process.
[0295] Define the gust intensity vector:
[0296]
[0297] Gust duration (steps):
[0298]
[0299] Probability of triggering a gust of wind per step:
[0300]
[0301] Gust status determined by a timer The description, and its recursive relationship, are as follows:
[0302]
[0303] in , These are uniformly distributed random numbers.
[0304] The sudden gust disturbance is:
[0305]
[0306] (4) Complete perturbation expression
[0307] Based on the above definitions, the complete mathematical expression for wind disturbance is:
[0308]
[0309] in The parameters for wind disturbance conditions are shown in Table 5, which serve as the indicator function.
[0310] Table 5 Parameters for Wind Disturbance Conditions
[0311]
[0312] (5) Simulation repeatability description
[0313] To ensure the repeatability of simulation results, a fixed random seed is used. Initialize the random number generator. This setting ensures that the sequence of random events such as the trigger time and duration of wind disturbances remains consistent each time the same simulation scenario is run, thus reflecting randomness while ensuring the fairness of comparative experiments.
[0314] (6) Description of disturbance conditions
[0315] During the simulation, basic wind noise persists, while sudden gusts occur randomly. The typical time-domain waveform is as follows: Figure 2 As shown. From Figure 2 It can be seen that the basic wind noise is characterized by continuous random fluctuations with small amplitude, while sudden gusts are characterized by pulse-type disturbances with large amplitude, short duration, and random occurrence. Together, they constitute a complex wind field environment with randomness and suddenness.
[0316] 3. Simulation Scenarios and Result Analysis
[0317] The reference trajectory that the drone needs to track is defined by the following state variables:
[0318]
[0319] The specific laws governing this motion are as follows:
[0320]
[0321] process:
[0322] The tracking performance comparison diagram of standard MPC and AC-MPC under disturbed conditions is shown below. Figures 3-10 As shown, from Figures 3-10 It can be seen that the flight tracking position error of the MPC optimized by the Actor-Critic algorithm is significantly smaller than that of the standard MPC.
[0323] Figures 11-13 This indicates that under strong wind disturbances, the system will tilt control resources toward position control to prioritize trajectory tracking. Therefore, the attitude angle tracking error of AC-MPC is slightly larger than that of standard MPC, but it is still within an acceptable engineering range.
[0324] At the same time Figures 15-17 It can be seen that the AC-MPC Critic module immediately senses the increase in error and the decrease in performance, and immediately drives the Actor module to update through the time difference error. The Actor quickly adjusts its output, significantly increasing the position weights in the MPC cost function (e.g., Figure 15 (As shown).
[0325] result:
[0326] AC-MPC generates a control torque with stronger recovery capabilities by increasing the key weights of the control cost function. The AC-MPC controller can quickly pull the UAV back to the reference trajectory, making its steady-state error much smaller than that of the standard MPC.
[0327] in conclusion:
[0328] This embodiment fully verifies the effectiveness and superiority of the method described in this invention in improving the anti-disturbance performance and trajectory tracking accuracy of quadcopter UAVs.
Claims
1. A quadrotor unmanned aerial vehicle adaptive model predictive control method based on a linear Actor-Critic framework, characterized in that, Includes the following steps: Step (1): Establish a nonlinear dynamic model of the quadcopter UAV considering external wind disturbance, linearize the nonlinear dynamic model at the hovering equilibrium point, and obtain a discrete state space model of the quadcopter UAV for prediction by discretization. Step (2): Based on the discrete state space model of the quadcopter UAV, construct the MPC controller and build a linear Actor-Critic framework fused with the MPC controller; Step (3): State variable observation, calculate tracking error, construct key state feature vector according to tracking error ; Step (4): input the key state feature vector to the Actor module in the Actor-Critic framework to generate a weight increment action vector ; Step (5): Update the weight vector corresponding to the position and attitude in the MPC state weight matrix, and solve the optimal control sequence after updating to obtain the first control input acting on the quadrotor unmanned aerial vehicle acting on the quadrotor unmanned aerial vehicle Step (6): After executing the control instruction, the next state is observed , the immediate reward is calculated , and the state value is estimated by the Critic module in the Actor-Critic framework, and the time difference error is calculated ; Step (7): Based on time difference error , the gradient descent method is used to update the parameters of the Actor module and the Critic module. Step (8): Iteration control: Let , return to step (3) until a termination instruction is received.
2. The method of claim 1, wherein, Step (1) The discrete state-space model of the quadcopter UAV is as follows: , where is the state vector, is the control input vector, is the wind disturbance vector; , , are the system, control and disturbance matrices, respectively.
3. The method of claim 2, wherein, Step (2) The objective function for optimizing the MPC controller is: , wherein, is a prediction time horizon, is a control time horizon, is a state vector at time step is a tracking error vector at time step is a control input vector at time step is a preset fixed control weight matrix, is an adaptive adjusted state weight matrix at time step is an adaptive adjusted state weight matrix at time step is an adaptive adjusted state weight matrix at time step 4. The method according to claim 3, characterized in that, Step (3) is as follows: In each control cycle Collect system state vector Calculation and reference trajectory Tracking error vector : , Six key error components directly related to position and attitude were selected to form the key tracking error vector. : , The absolute values of the aforementioned key errors are used to construct the key state feature vector. : , in For position tracking error, This represents the attitude angle tracking error.
5. The method according to claim 4, characterized in that, Step (4) is as follows: Key state feature vectors Input the linear Actor module to generate weighted incremental action vectors. : , in: For the Actor module in the The weight matrix of the step is used to linearly map the state features to the direction of the basic policy. This is the hyperbolic tangent function, used to compress the output to the interval (-1, 1); and This is a preset vector of incremental lower bound and range parameters; " indicates Hadamard product, element-wise multiplication; Explore noise vectors for directionality.
6. The method according to claim 5, characterized in that, Directional exploration noise vector The expression: , in: It is a random perturbation component vector, whose components are independent of each other and follow a standard Gaussian distribution; It is a directional bias vector, and the signs of its components are related to the direction of change of the corresponding error components; The experience reuse vector is obtained by exponentially weighting and normalizing the exploration directions of historical successful actions; To adapt to the exploration intensity, These are the preset weighting coefficients.
7. The method according to claim 6, characterized in that, Step (5) is as follows: Based on weight increment action vector Update the position and pose weight vectors in the MPC state weight matrix: , , in: The weight vector from the previous time step, whose components correspond to the position error and attitude angle error respectively. The weighting coefficients in the text; This is a saturation function used to constrain the weights within preset upper and lower bounds; By combining the constrained 6-dimensional weight vector with the fixed velocity and angular velocity weights, a complete 12x12 state weight matrix is constructed: , Use the updated With fixed Solve step (2) of the MPC controller to obtain the optimal control sequence, and set the first control input It is applied to quadcopter drones.
8. The method according to claim 7, characterized in that, Step (6) specifically involves: After executing the control command, observe the next state. Calculate instant rewards The state value is estimated by the Critic module: , in: These are the eigenvectors after normalized nonlinear mapping; This is the weight vector of the Critic module. These are bias parameters; It is the hyperbolic tangent function; Calculate time difference error : , in To provide comprehensive instant rewards, This is the discount factor.
9. The method according to claim 8, characterized in that, Comprehensive Instant Rewards For a multi-objective weighted combination, its expression is: , in: To track improvement rewards, calculations are based on the relative reduction in the tracking error norm between adjacent time steps; As a steady-state performance reward, a positive reward is given when the current tracking error norm is lower than a preset threshold; To control the cost penalty, calculations are based on the control input norm and its rate of change; The Q / R ratio penalty is activated when the norm ratio of the state weight matrix to the control weight matrix exceeds a safety threshold. As an oscillation penalty, it is activated when frequent changes in the sign of the system error are detected and the control input changes drastically; This is a weight efficiency penalty, activated when the weight vector components exceed a preset threshold or all weights are saturated while tracking performance is good. to These are the preset weighting coefficients for each component.
10. The method according to claim 9, characterized in that, Step (7) is as follows: Critic module update: , , in: This refers to the time difference error; The key state feature vector The eigenvectors are obtained by normalization followed by nonlinear mapping using the hyperbolic tangent function; This is the learning rate of the Critic module. When the actual result is better than expected, i.e., the time difference error... When the value is positive, the parameter is adjusted in a direction that enhances the current value estimate; conversely, it is adjusted in a direction that weakens it. Actor module update: The policy gradient method is adopted, utilizing the time difference error. Driver parameter update: , in: gradient It is a 6x6 matrix. ; It is the learning rate of the Actor module, when the time difference error... When positive, the current action is reinforced, making it more likely to be adopted in similar future states. When the value is negative, the current action is weakened, making it more likely to be avoided.
