MPC hierarchical collaborative optimization control research method based on deep reinforcement learning

Abstract

An MPC hierarchical collaborative optimization control research method based on deep reinforcement learning comprises the steps that A, UA-SAC error learning and compensation control are carried out on an inner layer, and the inner layer senses an adaptive optimization control algorithm through uncertainty and finally obtains an accurate error model through adaptive weight adjustment; step B, outer-layer multi-dimensional coupling optimization: the outer layer realizes progressive reinforcement learning optimization control by applying a TD3 algorithm on the basis of an error model obtained by learning, and specifically comprises three learning stages: an initial learning stage based on expert rules, a progressive learning stage based on performance feedback and a completely autonomous optimization learning stage; and step C, integrating a hierarchical collaborative optimization control system, adopting a hierarchical collaborative control framework, and realizing collaborative optimization of an inner layer and an outer layer through information interaction, and the method has the advantages that the requirements of high-maneuverability autonomous cross-country driving of the vehicle on the aspects of trafficability, maneuverability, stability and safety in complex terrains are met.

Description

Technical Field

[0001] This invention relates to the field of all-terrain platform technology, and specifically to a research method for MPC hierarchical collaborative optimization control based on deep reinforcement learning. Background Technology

[0002] Based on the practical needs of emergency rescue, scientific research, and resource exploration, there is an urgent need for a series of intelligent all-terrain platforms with excellent mobility and maneuverability, multi-mission capability, low environmental footprint, and both information and system capabilities to realize applications in these fields. Existing platforms face the following key issues that urgently need to be addressed during operation:

[0003] (1) It is difficult to balance passability, stability, and safety during all-terrain delivery. Compared to well-maintained structured roads, off-road terrain is extremely complex, and the dynamic coupling relationship between vehicle passability, stability, and safety is much closer. For traditional vehicles, appropriately lowering the center of gravity can reduce the probability of vehicle rollover and improve driving stability and safety. However, in complex terrain, a decrease in center of gravity will greatly weaken the vehicle's passability. In addition, during all-terrain delivery, the vehicle's posture and tire dynamic load fluctuate significantly with changes in terrain, making it more prone to problems such as vehicle instability, bouncing, and rollover. Therefore, analyzing the dynamic coupling mechanism and evolution mechanism of vehicle passability, stability, and safety in all-terrain environments, and improving driving stability and safety while ensuring passability, is an urgent problem that all-terrain delivery platforms need to solve.

[0004] (2) Coordinating long-cycle optimization strategies with real-time control is difficult in controller design. Optimal control (such as model predictive control) relies on rolling optimization in a fixed time domain, making it difficult to balance short-cycle dynamic response with long-cycle global optimization objectives, resulting in limited control performance under complex operating conditions. While simply expanding the prediction window can improve long-term optimization capabilities, it leads to an exponential increase in computational complexity, compromising the real-time requirements of the control system. Considering the long-cycle value function estimation and policy generalization characteristics of deep reinforcement learning (DRL), it can be used to solve complex problems with strong dynamic nonlinearity and high coupling, realizing the spatiotemporal mapping of long-term optimization objectives to short-term control actions. Therefore, integrating the advantages of artificial intelligence and optimal control to improve the global optimization capability of complex control systems while ensuring real-time response has become one of the current research hotspots. Summary of the Invention

[0005] In view of the above problems, the purpose of this invention is to provide a research method for MPC hierarchical collaborative optimization control based on deep reinforcement learning, so as to meet the requirements of vehicles for high-mobility autonomous off-road driving in complex terrain in terms of passability, maneuverability, stability and safety, and overcome the shortcomings of the above-mentioned prior art.

[0006] This invention provides a research method for MPC hierarchical collaborative optimization control based on deep reinforcement learning, which specifically includes the following steps:

[0007] Step A: Inner layer UA-SAC error learning and compensation control.

[0008] The inner layer uses the uncertainty-aware adaptive optimization control (UA-SAC) algorithm, which includes a stochastic target generation mechanism, physical environment modeling, MPC minimization tracking error control, and error estimator integration and uncertainty calculation. Finally, it obtains an accurate error model through adaptive weight adjustment.

[0009] Step B: Optimization of outer layer multidimensional coupling.

[0010] The outer layer is based on the error model obtained through learning, and applies the TD3 algorithm to realize progressive reinforcement learning optimization control, which includes three learning stages: an initial learning stage based on expert rules, a progressive learning stage based on performance feedback, and a fully autonomous optimization learning stage. The entire framework achieves a smooth transition through adaptive weight control, effectively combining the stability of model predictive control with the adaptive capability of reinforcement learning, and exhibits superior performance in the control of uncertain dynamic systems.

[0011] Step C: Integration of hierarchical collaborative optimization control system. A hierarchical collaborative control architecture is adopted to achieve collaborative optimization between inner and outer layers through information interaction.

[0012] As a preferred embodiment of the present invention, step A further includes the following step:

[0013] Step A1: Design of the Uncertainty-Aware SAC Module

[0014] The inner uncertainty-aware SAC module adopts the uncertainty-aware adaptive optimization control (UA-SAC) algorithm to learn model errors and achieve dynamic error compensation, and constructs a reinforcement learning framework that can handle the nonlinear coupled dynamic characteristics in vehicle attitude control.

[0015] Step A11, Definition of State Space in Reinforcement Learning Framework: The state space S_inner of the inner control layer is defined as follows:

[0016]

[0017] Where, θ, z represents the vehicle's pitch angle, roll angle, and vertical displacement, respectively. These represent pitch angular velocity, roll angular velocity, and vertical velocity, respectively; vx, vy, vz, ax, ay, az represent the triaxial velocities and triaxial accelerations, respectively; zfl, zfr, zrl, zrr represent the four-wheel suspension travel; Ffl, Ffr, Frl, Frr represent the four-wheel suspension loads; and eθ, ez represents the tracking error of pitch angle, roll angle, and vertical displacement, respectively, and θd represents the tracking error of vertical displacement. z and d represent the desired attitude targets transmitted from the outer layer, respectively.

[0018] Step A12, Defining the action space in the reinforcement learning framework: The action space A_inner for the inner control is:

[0019] A_inner=[u_fl,u_fr,u_rl,u_rr]

[0020] Where u_fl, u_fr, u_rl, u_rr are the four-wheel damper control signals;

[0021] Step A2: Design of Uncertainty-Aware Adaptive Optimization Control (UA-SAC) Algorithm.

[0022] Step A21, Random Target Generation Mechanism: To improve the algorithm's exploration capability, a target generation strategy based on uncertainty measurement is designed:

[0023] a_target(s)=π_θ(s)+σ_u(s)·ε

[0024] Where σ_u(s) represents the uncertainty measure under state s, and ε~N(0,1) is Gaussian noise;

[0025] Step A22, Physical Environment Modeling: Establish a vehicle dynamics model considering multiphysics coupling:

[0026]

[0027] Where M(q) is the generalized mass matrix. Let G(q) be the Coriolis matrix, G(q) be the gravity term, τ be the control input, and d(t) be the external disturbance. This represents the uncertainty term in the model.

[0028] Step A23: Minimize tracking error control using MPC.

[0029] The objective function of the inner MPC controller in the step is:

[0030] J_MPC=∑(k=0toN-1)[||e_k||2_Q+||u_k||2_R]+||e_N||2_P

[0031] Where e_k is the tracking error at time k, Q is the state error weight matrix, u_k is the control input, R is the control quantity weight matrix, e_N is the end tracking error, and P is the terminal error weight matrix.

[0032] Constraints:

[0033] x_{k+1}=f(x_k,u_k)+w_k

[0034] u_min≤u_k≤u_max

[0035] Δu_min≤Δu_k≤Δu_max

[0036] Where x_k is the system state vector, f(x_k,u_k) is the nonlinear state transition function, w_k is the process noise, u_min is the lower limit of damping force, u_max is the upper limit of damping force, Δu_min is the minimum rate of change, and Δu_max is the maximum rate of change;

[0037] Step A24: Error estimator integration and uncertainty calculation.

[0038] Design an error estimator based on a neural network:

[0039]

[0040] Uncertainty measure is calculated using a variance network:

[0041] σ 2 _u(x,u)=NN_var(x,u;θ_v);

[0042] in, Here, θ_e represents the error estimate of the dynamic system, NN_error represents the error estimation network parameters, and σ represents the error estimation network structure. 2 _u(x,u) represents the control uncertainty variance, θ_v represents the variance network parameters, and NN_var represents the variance prediction network;

[0043] Step A3: A multi-scale adaptive learning mechanism is used to achieve accurate parameter estimation under high-maneuverability conditions.

[0044] Tire mechanical parameter estimation:

[0045] F_x=μ_x(conditions)·N·C_s·s / (1+|s|)

[0046] F_y=μ_y(conditions)·N·C_α·α / (1+|α|)

[0047] Where F_x and F_y are the longitudinal force and lateral force of the tire, respectively, μ_x and μ_y are the longitudinal and lateral friction coefficients under specific working conditions, N is the vertical load of the tire, C_s is the longitudinal slip stiffness coefficient, C_α is the lateral stiffness coefficient, s is the longitudinal slip ratio, and α is the tire slip angle.

[0048] μ is estimated online using a neural network. v is the composite vehicle speed at the tire contact point, T_road is the road surface temperature, surface_type is the road surface type encoding, NN_frictio is the neural network function, and θ_μ is the network parameter set.

[0049] As a preferred embodiment of the present invention, step B further includes the following step:

[0050] Step B1: Outer layer TD3 progressive reinforcement learning optimizes control.

[0051] Step B11, TD3 algorithm architecture design: the outer layer control is based on the error model learned from the inner layer in step A, and the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm is applied to realize progressive reinforcement learning optimization control;

[0052] Step B12, State space of the TD3 algorithm architecture:

[0053]

[0054] in, θ and z represent the current roll angle, pitch angle, and vertical displacement, respectively. These are the attitude angular velocity and vertical velocity, respectively. θref and zref represent the desired attitude angle and vertical position, respectively. eθ and ez represent the attitude tracking error, μ is the ground friction coefficient, slope is the terrain gradient, rough is the ground roughness, bump is the bump intensity, and FL, FR, RL, and RR are the four-wheel suspension loads. σθ and σz represent the attitude estimation uncertainty;

[0055] Step B13, Action Space of the TD3 Algorithm Architecture:

[0056]

[0057] in, θd, zd are the desired target values ​​of roll angle, pitch angle, and vertical displacement, kp is the position control gain weight, kd is the velocity control gain weight, and ka is the acceleration control gain weight.

[0058] Step B2, Three-stage progressive learning strategy,

[0059] Step B21, Phase 1: Initial learning phase based on expert rules. In this phase, the TD3 algorithm acquires basic attitude decision-making capabilities by learning expert rules.

[0060] Expert rule design,

[0061]

[0062]

[0063] Step B22, Phase Two: Progressive Learning Phase Based on Performance Feedback. Design an adaptive learning strategy based on control performance to gradually transition from expert rule guidance to autonomous decision-making.

[0064] Adaptive reward mechanism:

[0065] R_total=w(perf)·R_expert+(1-w(perf))·R_task

[0066]

[0067] R_task=R_attitude+R_efficiency+R_safety

[0068] Where R_total is the total reward, R_expert is the expert rule reward, R_task is the task reward, and w(perf) is the performance weight function. The target value for TD3 decision-making. The target value for the expert rule.

[0069] R_attitude is the attitude control effect reward, R_efficiency is the energy efficiency optimization reward, and R_safety is the safety reward. The weight function is dynamically adjusted based on performance comparison.

[0070] performance_ratio=Performance_TD3 / Performance_baseline

[0071] w(perf)=sigmoid(β·(performance_threshold-performance_ratio))

[0072] Performance_TD3 is the overall performance score of the current TD3 control strategy, Performance_baseline is the performance score of the baseline strategy, performance_ratio is the performance improvement ratio, performance_threshold is the performance threshold for strategy switching, and β is the function slope parameter.

[0073] Performance evaluation metrics:

[0074] Performance=α1·(1 / (1+RMS_attitude_error))+a2·(1 / (1+energy_consumption))+α3·safety_margin

[0075] Where RMS_attitude_error is the root mean square value of the vehicle attitude error, energy_consumption is the total energy consumption of the actuator, safety_margin is the safety margin, and α1, α2, and α3 are the corresponding weight coefficients.

[0076] As TD3 performance improves, the weight w(perf) gradually decreases, and the system naturally transitions to a fully autonomous optimization learning stage.

[0077] Step B23, Phase Three: The Fully Autonomous Optimization Learning Phase

[0078] At this stage, the TD3 algorithm completely takes over the attitude target decision-making task and optimizes long-term performance through reinforcement learning:

[0079] Strategy output:

[0080] in, For policy network functions;

[0081] The Q function is updated through a dual-network structure:

[0082] Q_θ1(s,a)←r+γ·min(Q_θ1′(s′,π_φ′(s′)),Q_θ2′(s′,π_φ′(s′)))

[0083] Q_θ2(s,a)←r+γ·min(Q_θ1′(s′,π_φ′(s′)),Q_θ2′(s′,π_φ′(s′)))

[0084] Where Q_θ1 is the principal value function network 1, Q_θ2 is the principal value function network 2, Q_θ1′ and Q_θ2′ are the target value networks 1 and 2, r is the immediate reward, γ is the discount factor, s′ is the next state, and π_φ′ is the target policy network.

[0085] The policy network updates via policy gradients:

[0086] The gradient of the policy objective;

[0087] Step B3: Expand and optimize objectives from multiple dimensions.

[0088] Step B31: The design of the reward function for the outer control layer considers multiple optimization objectives:

[0089] R_total=w1·R_attitude+w2·R_efficiency+w3·R_safety

[0090] The specific calculation formula is as follows:

[0091] Step B32, Attitude Control Reward:

[0092] Where: w_θ, w_z is the weighting coefficient for pitch angle, roll angle, and vertical displacement, θ, For the current pitch angle and roll angle, θ_max, z is the maximum allowable attitude angle, used for normalization; z is the current vertical displacement; z_target is the target vertical displacement; z_max is the maximum vertical displacement range, used for normalization.

[0093] Step B33, Energy Efficiency Optimization Reward:

[0094] R_efficiency=-tanh(P_total / P_reference-1)-w_act·actuator_usage

[0095] Where: P_total is the total power consumption of the system, tanh() is the hyperbolic tangent function, which limits the power consumption ratio to a bounded range, P_reference is the reference power consumption, w_act is the actuator utilization weight, actuator_usage is the usage intensity of the damper actuator;

[0096] Step B34, Security Rewards:

[0097] R_safety=-w_tilt·rollover_risk-w_bottom·bottoming_risk+w_margin·stability_margin

[0098] Where: w_tilt is the rollover risk weight, rollover_risk is the vehicle rollover risk index, w_bottom is the bottoming risk weight, bottoming_risk is the suspension bottoming risk, w_margin is the stability margin weight, and stability_margin is the system stability margin.

[0099] As a preferred embodiment of the present invention, step C further includes the following step:

[0100] Step C1, Information Interaction Interface:

[0101]

[0102] Among them, control_performance is the control performance feedback, uncertainty_estimates is the uncertainty estimate, and terrain_preview is the terrain preview information;

[0103] Among them, the inner layer to the outer layer contains: current attitude state, tracking error, suspension load, control performance feedback, and uncertainty estimation; the outer layer to the inner layer contains: desired attitude target, control weight parameters, and terrain preview information.

[0104] Step C2: Collaboratively optimize the objective function:

[0105] J cooperative =α·normalize(J_MPC)+β·normalize(J outer )+γ·normalize(J interaction )

[0106] Among them, J cooperative J is the collaborative objective function, J_MPC is the inner cost function, and J... outer The outer cost function, J, has a value equal to the total reward R_total. interaction For the interaction cost term, α, β, and γ are the corresponding weight coefficients, and normalize() is the normalization function;

[0107] J_interaction is used to evaluate the coordination and consistency between inner and outer layer controls, enabling the two layers of controls to work together without conflicting with each other. It is an important component of the hierarchical collaborative optimization objective function.

[0108] J_interaction=w1·||target_deciation|| 2 +w2·||parameter_mismatch|| 2

[0109] Where w1 is the target deviation weight coefficient, w2 is the parameter mismatch weight coefficient, target_deviation is the attitude target deviation term, and parameter_mismatch is the control parameter deviation term;

[0110]

[0111] in: θd, zd are the desired attitude targets output by the outer TD3 layer. θ_actual and z_actual represent the actual posture achieved by the inner layer control.

[0112] parameter_mismatch=[kp_used-kp_desired,

[0113] kd_used-kd_desired,ka_used-ka_desired]

[0114] Where: kp_desired, kd_desired, and ka_desired are the desired control weight parameters output by the outer TD3, and kp_used, kd_used, and ka_used are the control weight parameters actually used by the inner MPC;

[0115] Step C3: The main control loop of the algorithm.

[0116] Step C31, Initialization: Initialize(UA_SAC,TD3,MPC_params)

[0117] Step C32, execute repeatedly:

[0118] For each control cycle:

[0119] Step C321, State Awareness:

[0120] state_inner = SenseVehicleState() / / Vehicle attitude, suspension state

[0121] state_outer = SenseEnvironment() / / Terrain and mission information

[0122] Step C322, Outer Layer Decision:

[0123] [φd,θd,zd,kp,kd,ka]=TD3.SelectAction(state_outer)

[0124] projected_targets=ProjectConstraints([φd,θd,zd])

[0125] Step C323, Inner Layer Control:

[0126] error_est=UA_SAC.EstimateError(state_inner)

[0127] reference=[projected_targets,kp,kd,ka]

[0128] [u_fl,u_fr,u_rl,u_rr] = MPC.Solve(state_inner,reference,error_est) Step C324, Execution Control:

[0129] ExecuteControl([u_fl,u_fr,u_rl,u_rr])

[0130] Step C325, Performance Evaluation and Feedback:

[0131] performance=EvaluatePerformance()

[0132] feedback_to_outer=[current_attitude,tracking_errors,loads]

[0133] Step C326, Learning and updating:

[0134] experience_inner=(state_inner,[u_fl,u_fr,u_rl,u_rr],reward_inner,next_state)

[0135] experience_outer=(state_outer,[φd,θd,zd,kp,kd,ka],reward_outer,next_state)

[0136] UpdateUA_SAC(experience_inner)

[0137] UpdateTD3(experience_outer).

[0138] The beneficial effects of this invention are as follows:

[0139] 1. This invention improves the ability of all-terrain platforms to maneuver flexibly, operate autonomously in various complex terrains and geological environments, and maintain a low environmental footprint. It can solve practical problems such as rapid delivery, life search, environmental detection, and fire extinguishing and smoke removal in wildfire rescue, post-disaster road closures, and complex terrain environments with low gravity conditions. It also accelerates the practical application of intelligent equipment and reduces the dependence on foreign countries for key core technologies.

[0140] 2. The DRL of this invention introduces a deep neural network, which greatly saves storage memory, increases sample capacity and generalization performance, and the strategy learned by reinforcement learning can better cope with random perturbations and uncertainties, thereby improving the robustness of the system.

[0141] 3. This invention relates to research on hierarchical collaborative optimization control of MPC based on deep reinforcement learning. The core research content focuses on dynamic error compensation of the inner-layer MPC and multidimensional coupled optimization of the outer layer, investigating the error learning and global collaborative control architecture driven by DRL. Key breakthroughs include: designing long-period optimization strategies under finite prediction windows; constructing a representation learning framework for nonlinear horizontal-vertical-vertical interactive coupled dynamic characteristics; researching hierarchical control collaborative mechanisms; and designing an auxiliary optimization module based on action constraint projection to enhance the multidimensional expansion optimization of the global optimization objective while maintaining the real-time response capability of the inner-layer MPC. Attached Figure Description

[0142] Figure 1 This is a schematic diagram of the hierarchical control framework of the present invention.

[0143] Figure 2 This is a technical roadmap for the present invention. Detailed Implementation

[0144] See Figure 1-2 This invention provides a research method for MPC hierarchical collaborative optimization control based on deep reinforcement learning, comprising the following steps:

[0145] Step A: Inner layer UA-SAC error learning and compensation control.

[0146] The inner layer uses the uncertainty-aware adaptive optimization control (UA-SAC) algorithm, which includes a stochastic target generation mechanism, physical environment modeling, MPC minimization tracking error control, and error estimator integration and uncertainty calculation. Finally, it obtains an accurate error model through adaptive weight adjustment.

[0147] Step A1: Design of the Uncertainty-Aware SAC Module

[0148] The inner uncertainty-aware SAC module adopts the uncertainty-aware adaptive optimization control (UA-SAC) algorithm to learn model errors and achieve dynamic error compensation, and constructs a reinforcement learning framework that can handle the nonlinear coupled dynamic characteristics in vehicle attitude control.

[0149] Step A11, Definition of State Space in Reinforcement Learning Framework: The state space Sinner of the inner layer control is defined as follows:

[0150]

[0151] Where, θ, z represents the vehicle's pitch angle, roll angle, and vertical displacement, respectively. These represent pitch angular velocity, roll angular velocity, and vertical velocity, respectively; vx, vy, vz, ax, ay, az represent the triaxial velocities and triaxial accelerations, respectively; zfl, zfr, zrl, zrr represent the four-wheel suspension travel; Ffl, Ffr, Frl, Frr represent the four-wheel suspension loads; and eθ, ez represents the tracking error of pitch angle, roll angle, and vertical displacement, respectively, and θd represents the tracking error of vertical displacement. z and d represent the desired attitude targets transmitted from the outer layer, respectively.

[0152] Step A12, Defining the action space in the reinforcement learning framework: The action space A_inner for the inner control is:

[0153] A_inner=[u_fl,u_fr,u_rl,u_rr]

[0154] Where u_fl, u_fr, u_rl, u_rr are the four-wheel damper control signals;

[0155] Step A2: Design of Uncertainty-Aware Adaptive Optimization Control (UA-SAC) Algorithm.

[0156] Step A21, Random Target Generation Mechanism: To improve the algorithm's exploration capability, a target generation strategy based on uncertainty measurement is designed:

[0157] a_target(s)=π_θ(s)+σ_u(s)·ε

[0158] Where σ_u(s) represents the uncertainty measure under state s, and ε~N(0,1) is Gaussian noise;

[0159] Step A22, Physical Environment Modeling: Establish a vehicle dynamics model considering multiphysics coupling:

[0160]

[0161] Where M(q) is the generalized mass matrix. Let G(q) be the Coriolis matrix, G(q) be the gravity term, τ be the control input, and d(t) be the external disturbance. This represents the uncertainty term in the model.

[0162] Step A23: Minimize tracking error control using MPC.

[0163] The objective function of the inner MPC controller in the step is:

[0164] J_MPC=Σ(k=0to N-1)[||e_k||2_Q+||u_k||2_R]+||e_N||2_P

[0165] Where e_k is the tracking error at time k, Q is the state error weight matrix, u_k is the control input, R is the control quantity weight matrix, e_N is the end tracking error, and P is the terminal error weight matrix.

[0166] Constraints:

[0167] x_{k+1}=f(x_k,u_k)+w_k

[0168] u_min≤u_k≤u_max

[0169] Δu_min≤Δu_k≤Δu_max

[0170] Where x_k is the system state vector, f(x_k,u_k) is the nonlinear state transition function, w_k is the process noise, u_min is the lower limit of damping force, u_max is the upper limit of damping force, Δu_min is the minimum rate of change, and Δu_max is the maximum rate of change;

[0171] Step A24: Error estimator integration and uncertainty calculation.

[0172] Design an error estimator based on a neural network:

[0173] ê(t)=NN_error(x(t),u(t);θ_e)

[0174] Uncertainty measure is calculated using a variance network:

[0175] σ 2 _u(x,u)=NN_var(x,u;θ_v);

[0176] Where ê(t) is the dynamic system error estimate, θ_e is the error estimation network parameter, NN_error is the error estimation network structure, and σ 2 _u(x,u) represents the control uncertainty variance, θ_v represents the variance network parameters, and NN_var represents the variance prediction network;

[0177] Step A3: A multi-scale adaptive learning mechanism is used to achieve accurate parameter estimation under high-maneuverability conditions.

[0178] Tire mechanical parameter estimation:

[0179] F_x=μ_x(conditions)·N·C_s·s / (1+|s|)

[0180] F_y=μ_y(conditions)·N·C_α·α / (1+|α|)

[0181] Where F_x and F_y are the longitudinal force and lateral force of the tire, respectively, μ_x and μ_y are the longitudinal and lateral friction coefficients under specific working conditions, N is the vertical load of the tire, C_s is the longitudinal slip stiffness coefficient, C_α is the lateral stiffness coefficient, s is the longitudinal slip ratio, and α is the tire slip angle.

[0182] μ is estimated online using a neural network. v is the composite vehicle speed at the tire contact point, T_road is the road surface temperature, surface_type is the road surface type encoding, NN_friction is the neural network function, and θ_μ is the network parameter set.

[0183] Step B: Optimization of outer layer multidimensional coupling.

[0184] The outer layer is based on the error model obtained through learning, and applies the TD3 algorithm to realize progressive reinforcement learning optimization control, which includes three learning stages: an initial learning stage based on expert rules, a progressive learning stage based on performance feedback, and a fully autonomous optimization learning stage. The entire framework achieves a smooth transition through adaptive weight control, effectively combining the stability of model predictive control with the adaptive capability of reinforcement learning, and exhibits superior performance in the control of uncertain dynamic systems.

[0185] Step B1: Outer layer TD3 progressive reinforcement learning optimizes control.

[0186] Step B11, TD3 algorithm architecture design: the outer layer control is based on the error model learned from the inner layer in step A, and the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm is applied to realize progressive reinforcement learning optimization control;

[0187] Step B12, State space of the TD3 algorithm architecture:

[0188]

[0189] in, θ and z represent the current roll angle, pitch angle, and vertical displacement, respectively. These are the attitude angular velocity and vertical velocity, respectively. θref and zref represent the desired attitude angle and vertical position, respectively. eθ and ez represent the attitude tracking error, μ is the ground friction coefficient, slope is the terrain gradient, rough is the ground roughness, bump is the bump intensity, and FL, FR, RL, and RR are the four-wheel suspension loads. σθ and σz represent the attitude estimation uncertainty;

[0190] Step B13, Action Space of the TD3 Algorithm Architecture:

[0191]

[0192] in, θd, zd are the desired target values ​​of roll angle, pitch angle, and vertical displacement, kp is the position control gain weight, kd is the velocity control gain weight, and ka is the acceleration control gain weight.

[0193] Step B2, Three-stage progressive learning strategy,

[0194] Step B21, Phase 1: Initial learning phase based on expert rules. In this phase, the TD3 algorithm acquires basic attitude decision-making capabilities by learning expert rules.

[0195] Expert rule design,

[0196]

[0197] Step B22, Phase Two: Progressive Learning Phase Based on Performance Feedback. Design an adaptive learning strategy based on control performance to gradually transition from expert rule guidance to autonomous decision-making.

[0198] Adaptive reward mechanism:

[0199] R_total=w(perf)·R_expert+(1-w(perf))·R_task

[0200]

[0201] R_task=R_attitude+R_efficiency+R_safety

[0202] Where R_total is the total reward, R_expert is the expert rule reward, R_task is the task reward, and w(perf) is the performance weight function. The target value for TD3 decision-making. The expert rule target value is defined by R_attitude, which is the attitude control effect reward, R_efficiency, which is the energy efficiency optimization reward, and R_safety, which is the safety reward. The weight function is dynamically adjusted based on performance comparison.

[0203] performance_ratio=Performance_TD3 / Performance_baseline

[0204] w(perf)=sigmoid(β·(performance_threshold-performance_ratio))

[0205] Performance_TD3 is the overall performance score of the current TD3 control strategy, Performance_baseline is the performance score of the baseline strategy, performance_ratio is the performance improvement ratio, performance_threshold is the performance threshold for strategy switching, and β is the function slope parameter.

[0206] Performance evaluation metrics:

[0207] Performance=α1·(1 / (1+RMS_attitude_error))+a2·(1 / (1+energy_consumption))+α3·safety_margin

[0208] Where RMS_attitude_error is the root mean square value of the vehicle attitude error, energy_consumption is the total energy consumption of the actuator, safety_margin is the safety margin, and α1, α2, and α3 are the corresponding weight coefficients.

[0209] As TD3 performance improves, the weight w(perf) gradually decreases, and the system naturally transitions to a fully autonomous optimization learning stage.

[0210] Step B23, Phase Three: The Fully Autonomous Optimization Learning Phase

[0211] At this stage, the TD3 algorithm completely takes over the attitude target decision-making task and optimizes long-term performance through reinforcement learning:

[0212] Strategy output:

[0213] in, For policy network functions;

[0214] The Q function is updated through a dual-network structure:

[0215] Q_θ1(s,a)←r+γ·min(Q_θ1′(s′,π_φ′(s′)),Q_θ2′(s′,π_φ′(s′)))

[0216] Q_θ2(s,a)←r+γ·min(Q_θ1′(s′,π_φ′(s′)),Q_θ2′(s′,π_φ′(s′)))

[0217] Where Q_θ1 is the principal value function network 1, Q_θ2 is the principal value function network 2, Q_θ1′ and Q_θ2′ are the target value networks 1 and 2, r is the immediate reward, γ is the discount factor, s′ is the next state, and π_φ′ is the target policy network.

[0218] The policy network updates via policy gradients:

[0219] The gradient of the policy objective;

[0220] Step B3: Expand and optimize objectives from multiple dimensions.

[0221] Step B31: The design of the reward function for the outer control layer considers multiple optimization objectives:

[0222] R_total=w1·R_attitude+w2·R_efficiency+w3·R_safety

[0223] The specific calculation formula is as follows:

[0224] Step B32, Attitude Control Reward:

[0225]

[0226] Where: w_θ, w_z is the weighting coefficient for pitch angle, roll angle, and vertical displacement, θ, For the current pitch angle and roll angle, θ_max, z is the maximum allowable attitude angle, used for normalization; z is the current vertical displacement; z_target is the target vertical displacement; z_max is the maximum vertical displacement range, used for normalization.

[0227] Step B33, Energy Efficiency Optimization Reward:

[0228] R_efficiency=-tanh(P_total / P_reference-1)-w_act·actuator_usage

[0229] Where: P_total is the total power consumption of the system, tanh() is the hyperbolic tangent function, which limits the power consumption ratio to a bounded range, P_reference is the reference power consumption, w_act is the actuator utilization weight, actuator_usage is the usage intensity of the damper actuator;

[0230] Step B34, Security Rewards:

[0231] R_safety=-w_tilt·rollover_risk-w_bottom·bottoming_risk+w_margin·stability_margin

[0232] Where: w_tilt is the rollover risk weight, rollover_risk is the vehicle rollover risk index, w_bottom is the bottoming risk weight, bottoming_risk is the suspension bottoming risk, w_margin is the stability margin weight, and stability_margin is the system stability margin.

[0233] Step C: Integration of hierarchical collaborative optimization control system. A hierarchical collaborative control architecture is adopted to achieve collaborative optimization between inner and outer layers through information interaction.

[0234] Step C1, Information Interaction Interface:

[0235]

[0236] Among them, control_performance is the control performance feedback, uncertainty_estimates is the uncertainty estimate, and terrain_preview is the terrain preview information;

[0237] Among them, the inner layer to the outer layer contains: current attitude state, tracking error, suspension load, control performance feedback, and uncertainty estimation; the outer layer to the inner layer contains: desired attitude target, control weight parameters, and terrain preview information.

[0238] Step C2: Collaboratively optimize the objective function:

[0239] J cooperative =α·normalize(J_MPC)+β·normalize(J outer )+γ·normalize(J interaction )

[0240] Among them, J cooperative J is the collaborative objective function, J_MPC is the inner cost function, and J... outer The outer cost function, J, has a value equal to the total reward R_total. interacyion For the interaction cost term, α, β, and γ are the corresponding weight coefficients, and normalize() is the normalization function;

[0241] J_interaction is used to evaluate the coordination and consistency between inner and outer layer controls, enabling the two layers of controls to work together without conflicting with each other. It is an important component of the hierarchical collaborative optimization objective function.

[0242] J_interaction=w1·||target_deviation|| 2 +w2·||parameter_mismatch|| 2

[0243] Where w1 is the target deviation weight coefficient, w2 is the parameter mismatch weight coefficient, target_deviation is the attitude target deviation term, and parameter_mismatch is the control parameter deviation term;

[0244]

[0245] in: θd, zd are the desired attitude targets output by the outer TD3 layer. θ_actual and z_actual represent the actual posture achieved by the inner layer control.

[0246] parameter_mismatch=[kp_used-kp_desired,kd_used-kd_desired,ka_used-ka_desired]

[0247] Where: kp_desired, kd_desired, and ka_desired are the desired control weight parameters output by the outer TD3, and kp_used, kd_used, and ka_used are the control weight parameters actually used by the inner MPC;

[0248] Step C3: The main control loop of the algorithm.

[0249] Step C31, Initialization: Initialize(UA_SAC,TD3,MPC_params)

[0250] Step C32, execute repeatedly:

[0251] For each control cycle:

[0252] Step C321, State Awareness:

[0253] state_inner = SenseVehicleState() / / Vehicle attitude, suspension state

[0254] state_outer = SenseEnvironment() / / Terrain and mission information

[0255] Step C322, Outer Layer Decision:

[0256] [φd,θd,zd,kp,kd,ka]=TD3.SelectAction(state_outer)

[0257] projected_targets=ProjectConstraints([φd,θd,zd])

[0258] Step C323, Inner Layer Control:

[0259] error_est=UA_SAC.EstimateError(state_inner)

[0260] reference=[projected_targets,kp,kd,ka]

[0261] [u_fl,u_fr,u_rl,u_rr]=MPC.Solve(state_inner,reference,error_est)

[0262] Step C324, Execution Control:

[0263] ExecuteControl([u_fl,u_fr,u_rl,u_rr])

[0264] Step C325, Performance Evaluation and Feedback:

[0265] performance=EvaluatePerformance()

[0266] feedback_to_outer=[current_attitude,tracking_errors,loads]

[0267] Step C326, Learning and updating:

[0268] experience_inner=(state_inner,[u_fl,u_fr,u_rl,u_rr],reward_inner,next_state)

[0269] experience_outer=(state_outer,[φd,θd,zd,kp,kd,ka],reward_outer,next_state)

[0270] UpdateUA_SAC(experience_inner)

[0271] UpdateTD3(experience_outer).

[0272] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A research method for MPC hierarchical collaborative optimization control based on deep reinforcement learning, characterized in that, Includes the following steps: Step A: Inner layer UA-SAC error learning and compensation control. The inner layer uses an uncertainty-aware adaptive optimization control algorithm, which includes a random target generation mechanism, physical environment modeling, MPC minimization tracking error control, and error estimator integration and uncertainty calculation. Finally, it obtains an accurate error model through adaptive weight adjustment. Step B: Optimization of outer layer multidimensional coupling. The outer layer uses the TD3 algorithm to implement progressive reinforcement learning optimization control based on the error model obtained from learning. Specifically, it includes three learning stages: an initial learning stage based on expert rules, a progressive learning stage based on performance feedback, and a fully autonomous optimization learning stage. Step C: Integration of hierarchical collaborative optimization control system. A hierarchical collaborative control architecture is adopted to achieve collaborative optimization between inner and outer layers through information interaction.

2. The method for MPC hierarchical collaborative optimization control based on deep reinforcement learning according to claim 1, characterized in that, Step A also includes the following steps: Step A1: Design of the Uncertainty-Aware SAC Module The inner uncertainty-aware SAC module employs an uncertainty-aware adaptive optimization control algorithm to learn model errors and achieve dynamic error compensation, and constructs a reinforcement learning framework capable of handling nonlinear coupled dynamic characteristics in vehicle attitude control. Step A11, Definition of State Space in Reinforcement Learning Framework: The state space of the inner control layer, Sinner, is defined as follows: in, These are the vehicle body pitch angle, roll angle, and vertical displacement. Let vx, vy, vz, ax, ay, az represent pitch angular velocity, roll angular velocity, and vertical velocity, respectively; vx, vy, vz, ax, ay, az represent triaxial velocity and triaxial acceleration, respectively; zfl, zfr, zrl, zrr represent four-wheel suspension travel; and Ffl, Ffr, Frl, Frr represent four-wheel suspension load. These are the tracking errors for pitch angle, roll angle, and vertical displacement, respectively. These are the desired attitude targets transmitted from the outer layer; Step A12, Defining the action space in the reinforcement learning framework: The action space A_inner for the inner control is: A_inner=[u_fl,u_fr,u_rl,u_rr] Where u_fl, u_fr, u_rl, u_rr are the four-wheel damper control signals; Step A2: Design of uncertainty-aware adaptive optimization control algorithm. Step A21, Stochastic Target Generation Mechanism: Design a target generation strategy based on uncertainty measurement: a_target(s)=π_θ(s)+σ_u(s)·ε Where σ_u(s) represents the uncertainty measure under state s, and ε~N(0,1) is Gaussian noise; Step A22, Physical Environment Modeling: Establish a vehicle dynamics model considering multiphysics coupling: Where M(q) is the generalized mass matrix. Let G(q) be the Coriolis matrix, G(q) be the gravity term, τ be the control input, and d(t) be the external disturbance. This represents the uncertainty term in the model. Step A23: Minimize tracking error control using MPC. The objective function of the inner MPC controller in the step is: J_MPC=∑(k=0toN-1)[||e_k|| 2 _Q+||u_k|| 2 _R]+||e_N|| 2 _P Where e_k is the tracking error at time k, Q is the state error weight matrix, u_k is the control input, R is the control quantity weight matrix, e_N is the end tracking error, and P is the terminal error weight matrix. Constraints: x_{k+1}=f(x_k,u_k)+w_k u_min≤u_k≤u_max Δu_min≤Δu_k≤Δu_max Where x_k is the system state vector, f(x_k,u_k) is the nonlinear state transition function, w_k is the process noise, u_min is the lower limit of damping force, u_max is the upper limit of damping force, Δu_min is the minimum rate of change, and Δu_max is the maximum rate of change; Step A24: Error estimator integration and uncertainty calculation. Design an error estimator based on a neural network: Uncertainty measure is calculated using a variance network: σ 2 _u(x,u)=NN_var(x,u;θ_v); in, Here, θ_e represents the error estimate of the dynamic system, NN_error represents the error estimation network parameters, and σ represents the error estimation network structure. 2 _u(x,u) represents the control uncertainty variance, θ_v represents the variance network parameters, and NN_var represents the variance prediction network; Step A3: Multi-scale adaptive learning mechanism. Tire mechanical parameter estimation: F_x=μ_x(conditions)·N·C_s·s / (1+|s|) F_y=μ_y(conditions)·N·C_α·α / (1+|α|) Where F_x and F_y are the longitudinal force and lateral force of the tire, respectively, μ_x and μ_y are the longitudinal and lateral friction coefficients under specific working conditions, N is the vertical load of the tire, C_s is the longitudinal slip stiffness coefficient, C_α is the lateral stiffness coefficient, s is the longitudinal slip ratio, and α is the tire slip angle. μ is estimated online using a neural network. v is the composite vehicle speed at the tire contact point, T_road is the road surface temperature, surface_type is the road surface type encoding, NN_friction is the neural network function, and θ_μ is the network parameter set.

3. The method for MPC hierarchical collaborative optimization control based on deep reinforcement learning according to claim 1, characterized in that, Step B also includes the following steps: Step B1: Outer layer TD3 progressive reinforcement learning optimizes control. Step B11, TD3 algorithm architecture design: the outer layer control is based on the error model learned from the inner layer in step A, and the TD3 algorithm is applied to realize progressive reinforcement learning optimization control; Step B12, State space of the TD3 algorithm architecture: in, These are the current roll angle, pitch angle, and vertical displacement, respectively. These are the attitude angular velocity and vertical velocity, respectively. These represent the desired attitude angle and vertical position, respectively. Here, μ represents the attitude tracking error, μ represents the ground friction coefficient, slope represents the terrain gradient, rough represents the ground roughness, bump represents the bump intensity, and FL, FR, RL, and RR represent the four-wheel suspension loads. For attitude estimation uncertainty; Step B13, Action Space of the TD3 Algorithm Architecture: in, Here, kp represents the desired roll angle, pitch angle, and vertical displacement target values, kd represents the position control gain weight, kd represents the velocity control gain weight, and ka represents the acceleration control gain weight. Step B2, Three-stage progressive learning strategy, Step B21, Phase 1: Initial learning phase based on expert rules. In this phase, the TD3 algorithm acquires basic attitude decision-making capabilities by learning expert rules. Expert rule design, Step B22, Phase Two: Progressive Learning Phase Based on Performance Feedback. Design an adaptive learning strategy based on control performance to gradually transition from expert rule guidance to autonomous decision-making. Adaptive reward mechanism: R_total=w(perf)·R_expert+(1-w(perf))·R_task R_task=R_attitude+R_efficiency+R_safety Where R_total is the total reward, R_expert is the expert rule reward, R_task is the task reward, and w(perf) is the performance weight function. The target value for TD3 decision-making. The expert rule target value is defined by R_attitude, which is the attitude control effect reward, R_efficiency, which is the energy efficiency optimization reward, and R_safety, which is the safety reward. The weight function is dynamically adjusted based on performance comparison. performance_ratio=Performance_TD3 / Performance_baseline w(perf)=sigmoid(β·(performance_threshold-performance_ratio)) Performance_TD3 is the overall performance score of the current TD3 control strategy, Performance_baseline is the performance score of the baseline strategy, performance_ratio is the performance improvement ratio, performance_threshold is the performance threshold for strategy switching, and β is the function slope parameter. Performance evaluation metrics: Performance = α1·(1 / (1+RMS_attitude_error))+a2·(1 / (1+energy_consumption))+α3·safety_margin Where RMS_attitude_error is the root mean square value of the vehicle attitude error, energy_consumption is the total energy consumption of the actuator, safety_margin is the safety margin, and α1, α2, and α3 are the corresponding weight coefficients. As TD3 performance improves, the weight w(perf) gradually decreases, and the system naturally transitions to a fully autonomous optimization learning stage. Step B23, Phase Three: The Fully Autonomous Optimization Learning Phase At this stage, the TD3 algorithm completely takes over the attitude target decision-making task and optimizes long-term performance through reinforcement learning: Strategy output: in, For policy network functions; The Q function is updated through a dual-network structure: Q_θ1(s,a)←r+γ·min(Q_θ1′(s′,π_φ′(s′)),Q_θ2′(s′,π_φ′(s′))) Q_θ2(s,a)←r+γ·min(Q_θ1′(s′,π_φ′(s′)),Q_θ2′(s′,π_φ′(s′))) Where Q_θ1 is the principal value function network 1, Q_θ2 is the principal value function network 2, Q_θ1′ and Q_θ2′ are the target value networks 1 and 2, r is the immediate reward, γ is the discount factor, s′ is the next state, and π_φ′ is the target policy network. The policy network updates via policy gradients: The gradient of the policy objective; Step B3: Expand and optimize objectives from multiple dimensions. Step B31: The design of the reward function for the outer control layer considers multiple optimization objectives: R_total=w1·R_attitude+w2·R_efficiency+w3·R_safety Step B32, Attitude Control Reward: in: These are the weighting coefficients for pitch angle, roll angle, and vertical displacement. The current pitch and roll angles are... z is the maximum allowable attitude angle, used for normalization; z is the current vertical displacement; z_target is the target vertical displacement; z_max is the maximum vertical displacement range, used for normalization. Step B33, Energy Efficiency Optimization Reward: R_efficiency = -tanh(P_total / P_reference-1) - w_act·actuator_usage where: P_total is the total power consumption of the system, tanh() is the hyperbolic tangent function that limits the power consumption ratio to a bounded range, P_reference is the reference power consumption, w_act is the actuator utilization weight, and actuator_usage is the usage intensity of the damper actuator; Step B34, Security Rewards: R_safety=-w_tilt·rollover_risk-w_bottom·bottoming_risk+w_margin ·stability_margin Where: w_tilt is the rollover risk weight, rollover_risk is the vehicle rollover risk index, w_bottom is the bottoming risk weight, bottoming_risk is the suspension bottoming risk, w_margin is the stability margin weight, and stability_margin is the system stability margin.

4. The method for MPC hierarchical collaborative optimization control based on deep reinforcement learning according to claim 1, characterized in that, Step C also includes the following steps: Step C1, Information Interaction Interface: Where control_performance represents control performance feedback, and uncertainty_estimates represents uncertainty estimates. terrain_preview provides terrain preview information; Among them, the inner layer to the outer layer contains: current attitude state, tracking error, suspension load, control performance feedback, and uncertainty estimation; the outer layer to the inner layer contains: desired attitude target, control weight parameters, and terrain preview information. Step C2: Collaboratively optimize the objective function: J cooperative =α·normalize(J_MPC)+β·normalize(J outer )+γ·normalize(J interaction Among them, J cooperative J is the collaborative objective function, J_MPC is the inner cost function, and J... outer The outer cost function, J, has a value equal to the total reward R_total. interaction For the interaction cost term, α, β, and γ are the corresponding weight coefficients, and normalize() is the normalization function; J_interaction is used to evaluate the coordination and consistency between inner and outer control layers; J_interaction=w1·||target_deviation|| 2 +w2·||parameter_mismatch|| 2 Where w1 is the target deviation weight coefficient, w2 is the parameter mismatch weight coefficient, target_deviation is the attitude target deviation term, and parameter_mismatch is the control parameter deviation term; in: The desired attitude target output by the outer TD3 layer. The attitude achieved by the actual control of the inner layer; parameter_mismatch=[kp_used-kp_desired, kd_used-kd_desired,ka_used-ka_desired] Where: kp_desired, kd_desired, and ka_desired are the desired control weight parameters output by the outer TD3, and kp_used, kd_used, and ka_used are the control weight parameters actually used by the inner MPC; Step C3: The main control loop of the algorithm. Step C31, Initialization: Initialize(UA_SAC,TD3,MPC_params) Step C32, execute repeatedly: For each control cycle: Step C321, State Awareness: state_inner = SenseVehicleState() / / Vehicle attitude, suspension state state_outer = SenseEnvironment() / / Terrain and mission information Step C322, Outer Layer Decision: [φd,θd,zd,kp,kd,ka]=TD3.SelectAction(state_outer) projected_targets=ProjectConstraints([φd,θd,zd]) Step C323, Inner Layer Control: error_est=UA_SAC.EstimateError(state_inner) reference=[projected_targets,kp,kd,ka] [u_fl,u_fr,u_rl,u_rr]=MPC.Solve(state_inner,reference,error_est) Step C324, Execution Control: ExecuteControl([u_fl,u_fr,u_rl,u_rr]) Step C325, Performance Evaluation and Feedback: performance=EvaluatePerformance() feedback_to_outer = [current_attitude, tracking_errors, loads] Step C326, learning and updating: experience_inner = (state_inner, [u_fl, u_fr, u_rl, u_rr], reward_inner, next_state) experience_outer = (state_outer, [φd, θd, zd, kp, kd, ka], reward_outer, next_state) UpdateUA_SAC(experience_inner) UpdateTD3(experience_outer).

Images