Unmanned truck trajectory tracking method based on dynamic correction of MPC by reinforcement learning

Abstract

The application belongs to the technical field of automatic driving control, and proposes a trajectory tracking method for unmanned trucks based on dynamic correction of MPC through reinforcement learning, which obtains real-time running state and expected reference trajectory of the unmanned truck through a vehicle-mounted sensor, constructs a current vehicle state vector and a tracking error; the current vehicle state vector is input into an adaptive agent based on deep reinforcement learning, the expected reference trajectory and the current vehicle state vector are input into an MPC controller, a prediction residual is generated and input into the adaptive agent; a comprehensive state vector is constructed based on the tracking error, the current vehicle state vector and the prediction residual, the adaptive agent generates a correction gain based on the comprehensive state vector and inputs the correction gain into the MPC controller; the MPC controller generates a current control instruction; the current control instruction is input into a physical system of the unmanned truck to implement vehicle control, and high-precision trajectory tracking control of the unmanned truck under complex working conditions is realized.

Description

Technical Field

[0001] This invention belongs to the field of autonomous vehicle control technology, and particularly relates to an unmanned truck trajectory tracking method based on reinforcement learning dynamic correction MPC. Background Technology

[0002] With the rapid development of artificial intelligence, 5G communication, and sensor technology, autonomous driving technology is reshaping the global transportation system. Among these technologies, autonomous trucks have become a cutting-edge research area in intelligent transportation and commercial vehicles due to their enormous potential in improving logistics efficiency, reducing labor costs, and ensuring driving safety. However, trucks, as the main carrier of modern road transportation, have dynamic characteristics such as large mass, high inertia, and high center of gravity compared to ordinary passenger cars. In actual operation, trucks exhibit variable load characteristics that change with their cargo status (empty, half-loaded, fully loaded), resulting in significant time-varying and uncertainties in their dynamic models. When trucks operate under complex conditions, the nonlinear characteristics of the system are significantly amplified, easily leading to instability phenomena such as sideslip and fishtailing.

[0003] Among existing trajectory tracking control methods, Model Predictive Control (MPC) has been widely used due to its ability to optimize online and explicitly handle system state and input constraints. However, traditional MPC often relies on fixed models and linear assumptions, failing to adequately consider system nonlinearity and parameter uncertainties. Due to the drastic changes in truck parameters, fixed-parameter MPC predictive models can suffer from severe "model mismatch" with the actual physical system, leading to control commands exhibiting response lag, decreased control accuracy, or even failure under complex operating conditions.

[0004] Meanwhile, Deep Reinforcement Learning (DRL) can learn optimal control strategies without relying on precise models, demonstrating good adaptability to unknown operating conditions. However, purely data-driven RL controllers suffer from low training efficiency, ambiguous safety boundaries, and unpredictable control behavior, making them difficult to directly apply to the low-level control of heavy-duty trucks. Therefore, designing a hybrid control method that can both leverage MPC prior knowledge to ensure basic physical safety and overcome the challenges of time-varying dynamic characteristics is crucial for achieving safe and precise control of unmanned trucks under extreme conditions. Summary of the Invention

[0005] To address the problems of severe dynamic model mismatch, insufficient environmental adaptability, and limited high-dynamic tracking accuracy when traditional control models are applied to unmanned truck trajectory tracking under complex operating conditions, this invention proposes an unmanned truck trajectory tracking method based on reinforcement learning-based dynamic correction MPC. This method introduces a reinforcement learning agent to learn the dynamic prediction residuals of the vehicle under the current operating conditions in real time and dynamically outputs the correction gain of the model parameters. Combined with a strategy that preserves the optimal solution framework of MPC and physical hard constraints, the adaptive prediction equations are reconstructed, thereby achieving high-precision trajectory tracking and robust control of unmanned trucks under time-varying and complex operating conditions.

[0006] The technical solution of the present invention is as follows: A method for unmanned truck trajectory tracking based on reinforcement learning-based dynamic correction MPC includes the following steps: Step S1: Obtain the real-time operating status and expected reference trajectory of the unmanned truck, and construct the current vehicle state vector and tracking error of the unmanned truck. Step S2: Input the current vehicle state vector into the adaptive agent based on deep reinforcement learning, input the expected reference trajectory and the current vehicle state vector into the MPC controller, the MPC controller calculates the prediction residual between its prior prediction value and the current vehicle state, and inputs the prediction residual into the adaptive agent; Step S3: Construct a comprehensive state vector based on the tracking error, the current vehicle state vector, and the prediction residual. The adaptive agent generates a correction gain based on the comprehensive state vector and inputs it into the MPC controller. Step S4: The MPC controller generates the current control command based on the desired reference trajectory, the current vehicle state vector, and the correction gain; Step S5: Input the current control command into the unmanned truck physical system to implement vehicle control; Step S6: Return to step S1 to achieve dynamic tracking of the unmanned truck trajectory.

[0007] Preferably, the current vehicle state vector Represented as:

[0008] in, The horizontal axis is... For the horizontal swing angle, and These represent the longitudinal and lateral speeds of the unmanned truck, respectively. ω represents the yaw rate.

[0009] Preferably, the desired reference trajectory Represented as:

[0010] in, To predict the time domain, To predict within the time domain k The expected vehicle state vector at time t.

[0011] Preferably, the tracking error includes lateral error. and yaw angle error .

[0012] Preferably, the prediction residual in step S2 is generated in the following manner: (1) Construct a dynamic model of the unmanned truck that includes longitudinal, lateral, and yaw motions:

[0013]

[0014] in, For the overall vehicle quality, These represent the longitudinal and lateral speeds of the unmanned truck, respectively. The yaw rate is angular velocity. For rotational inertia, The distance from the center of mass to the front and rear axles. This is the sum of the longitudinal and lateral forces of each axle in the vehicle coordinate system. The initial reference front axle lateral stiffness, The lateral force of the front tire. The lateral force of the rear tire. This refers to the front tire slip angle. This refers to the rear tire slip angle.

[0015] (2) The dynamic model of the unmanned truck is expressed as a continuous nonlinear state space form. The continuous state matrix is ​​obtained by expanding it using the first-order Jacobian matrix. With control matrix ; (3) Introducing sampling time The forward Euler method is used to discretize the continuous-time model, and a baseline discrete prediction model is established:

[0016] in, , , It is the identity matrix. For truncation error, This is the vehicle state vector from the previous moment. The control command from the previous moment; (4) Extract the prior prediction value of the MPC controller for the current time from the previous time step. Combined with the current vehicle state vector Calculate the predicted residual vector :

[0017] (5) From the predicted residual vector Extracting the lateral displacement residual Residual of yaw rate As the prediction residual.

[0018] Preferably, the integrated state vector is represented as:

[0019] Correction gain Represented as:

[0020] in, To compensate for the lateral stiffness of the front tires This is for rear tire lateral stiffness compensation.

[0021] Preferably, the current control command Represented as , This refers to the front wheel steering angle. This is the longitudinal acceleration.

[0022] Preferably, the adaptive agent based on deep reinforcement learning undergoes random mini-batch sampling training before use, including the following: (1) Constructing a dynamic model and experience playback pool for unmanned trucks; (2) Input the current control command into the dynamic model of the unmanned truck to perform simulation control and output a new vehicle state; calculate the tracking error based on the new vehicle state and the expected reference trajectory, and calculate the prediction residual based on the new vehicle state and the prior prediction value of the MPC controller at the previous moment. (3) The adaptive agent based on deep reinforcement learning includes an Actor network and a Critic network. The Actor network takes the comprehensive state vector as input and outputs a correction gain. The Critic network takes the comprehensive state vector and the correction gain as input and outputs a scalar Q value. (4) Construct a multi-objective reward function that includes trajectory tracking accuracy and model consistency rewards to guide the Actor network to output corrective gain; (5) Update the core dynamic parameters using the modified gain, recalculate the continuous state matrix and control matrix, and obtain the adaptive state matrix and control matrix by discretization, and establish the vehicle dynamics adaptive prediction equation that is dynamically matched with complex working conditions. (6) Based on the vehicle dynamics adaptive prediction equation, a quadratic performance index is constructed with the goal of minimizing tracking error and control cost. Combined with the physical hard constraints of the unmanned truck, a rolling quadratic optimization solution is performed to obtain new control commands. (7) Take the new control command as the current control command and return to step (2).

[0023] Preferably, the multi-objective reward function is expressed as:

[0024]

[0025]

[0026] in, For trajectory tracking penalty items, This is a model consistency penalty term. For the trajectory tracking term, the non-negative weight coefficients are... represents the non-negative weight coefficients of the model consistency term.

[0027] Preferably, the vehicle dynamics adaptive prediction equation is expressed as:

[0028] in, For the present moment and the future The predicted value of the vehicle state vector at time t. For the present moment and the future Predicted values ​​of control commands at any given time.

[0029] Preferably, step (6) specifically includes: a) Set the control time domain as ,and A quadratic objective function is constructed based on the vehicle dynamics adaptive prediction equation. :

[0030] in, These are the set state error weight matrix, control quantity weight matrix, and control increment weight matrix, respectively. To predict within the time domain The expected vehicle state vector at time t. The control increment is between adjacent time steps; These are the weighting coefficients. These are slack variables; b) Set physical hard constraints for control commands within the control time domain:

[0031] in, These are the upper and lower bounds of the physical extreme values ​​of the front wheel steering angle and longitudinal acceleration, respectively. These are the upper and lower limits of the extreme values ​​for the front wheel steering angle increment and the longitudinal acceleration increment, respectively. c) The quadratic objective function Combined with physical hard constraints, and transformed into a standard quadratic programming problem solution, the optimal control increment sequence is obtained through optimization calculation:

[0032] Extract the first element of the optimal control increment sequence. Combined with the control commands from the previous moment The new control command at the current moment is calculated. .

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention proposes a trajectory tracking method for unmanned trucks based on reinforcement learning-based dynamic correction MPC. It fully considers the characteristics of unmanned trucks, such as large mass, high center of gravity, and significant load changes with working conditions. It adopts a three-degree-of-freedom unmanned truck dynamic model, which can fully describe the coupling characteristics of longitudinal, lateral, and yaw three-dimensional motions while ensuring the model's computational efficiency meets the requirements of real-time control. This provides a reliable model foundation for subsequent reinforcement learning agents to accurately identify dynamic parameter mismatches.

[0034] 2. This invention proposes an unmanned truck trajectory tracking method based on reinforcement learning dynamic correction MPC. It uses a deep deterministic policy gradient algorithm to construct a parameter adaptive correction agent, which can output the lateral stiffness correction gain in the continuous action space, ensuring the accuracy and smoothness of parameter correction. At the same time, it achieves stable and efficient offline training through an experience playback mechanism.

[0035] 3. This invention proposes an unmanned truck trajectory tracking method based on reinforcement learning dynamic correction MPC, and constructs a multi-objective reward function that integrates trajectory tracking accuracy and model consistency penalty, thereby enabling the adaptive agent to output the optimal lateral stiffness correction gain under time-varying conditions.

[0036] 4. This invention proposes a trajectory tracking method for unmanned trucks based on reinforcement learning-based dynamic correction MPC. It constructs a quadratic objective function based on an adaptive prediction equation, transforms it into a standard quadratic programming problem by combining physical hard constraints, and solves the optimal control command through online rolling optimization to achieve high-precision trajectory tracking control of unmanned trucks under complex working conditions. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the unmanned truck trajectory tracking method based on reinforcement learning dynamic correction MPC proposed in this invention.

[0039] Figure 2 This is a schematic diagram of the operational logic of the unmanned truck trajectory tracking method of the present invention.

[0040] Figure 3 This is a schematic diagram of the interaction architecture between the adaptive agent and the MPC controller in this invention. Detailed Implementation

[0041] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0042] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0043] The unmanned truck trajectory tracking method based on reinforcement learning dynamic correction MPC proposed in this invention, such as... Figure 1-2 As shown, the main steps include: Step 1: Obtain the real-time operating status and expected reference trajectory of the unmanned truck, and construct the current vehicle state vector and control commands of the chassis actuators for the unmanned truck.

[0044] Step 1.1, in the control cycle The current vehicle state vector of the unmanned truck is obtained through onboard sensors. :

[0045] in, The horizontal axis is... For the horizontal swing angle, and These represent the longitudinal and lateral speeds of the unmanned truck, respectively. ω represents the yaw rate.

[0046] Step 1.2: Based on the global planning path, in the prediction time domain... Discretize along the inner path tangent direction to extract the desired reference trajectory. :

[0047] in, To predict the time domain, To predict within the time domain k The expected vehicle state vector at time t.

[0048] Step 1.3: Define the control commands for the chassis actuators. ,in This refers to the front wheel steering angle. For longitudinal acceleration. Calculate the current lateral tracking error. With yaw angle error These are used to construct the state features of the agent in subsequent operations.

[0049] Step 2: Construct a three-degree-of-freedom truck dynamics model and discretize it. Combine the actual observations from the sensors to extract the predicted residuals that characterize the model mismatch.

[0050] Step 2.1: Construct a three-degree-of-freedom truck dynamics model that includes longitudinal, lateral, and yaw motions. The core dynamic equations are as follows:

[0051] in, For the overall vehicle quality, These represent the longitudinal and lateral speeds of the unmanned truck, respectively. The yaw rate is angular velocity. For rotational inertia, The distance from the center of mass to the front and rear axles. This is the sum of the longitudinal and lateral forces of each axle in the vehicle coordinate system. The lateral force of the front tire. This refers to the lateral force of the rear tire.

[0052] Based on the linear tire model, the initial reference front axle lateral stiffness is set as follows: The initial reference rear axle lateral stiffness is The lateral forces of the front and rear tires can then be expressed as:

[0053] in, This refers to the slip angle of the front and rear tires.

[0054] Step 2.2: Represent the above truck dynamics model in a continuous nonlinear state-space form. At the work site The continuous state matrix is ​​obtained by expanding the first-order Jacobian matrix. With control matrix .

[0055] Step 2.3: Introduce sampling time The forward Euler method is used to discretize the continuous-time model, and a baseline discrete prediction model is established:

[0056] in, , , It is the identity matrix. For truncation error, This is the vehicle state vector from the previous moment. This refers to the control command from the previous moment.

[0057] Step 2.4: Extract the prior prediction value of the MPC controller for the current time step from the previous time step. Combined with the current vehicle state vector Calculate the predicted residual vector :

[0058] Step 2.5: From the predicted residual vector Extracting the lateral displacement residual Residual of yaw rate It is used to quantify the parameter mismatch of the current dynamic model caused by changes in operating conditions.

[0059] Step 3: Construct an adaptive agent based on a deep reinforcement learning algorithm. The adaptive agent includes a policy network (Actor) and a value network (Critic), and takes the comprehensive state vector formed by concatenating the tracking error and the prediction residual as input.

[0060] Step 3.1: Construct a parameter-adaptive agent using the Deep Deterministic Policy Gradient (DDPG) algorithm. The DDPG algorithm includes an Actor network for outputting continuous deterministic actions and a Critic network for evaluating the value of actions, with identical online and target networks configured for each. During offline training, the agent stores historical interaction data in an experience replay pool and performs mini-batch sampling to break data correlations. The parameters of the online Critic network are updated using temporal difference errors, while the parameters of the online Actor network are gradient-updated by maximizing the Q-value output of the online Critic network. The target network uses a smooth soft update mechanism, thereby ensuring stable convergence during training.

[0061] Step 3.2: Calculate the current lateral tracking error of the unmanned truck. Heading angle error and its corresponding first rate of change vehicle longitudinal speed yaw rate and the lateral displacement prediction residual extracted from step 2 and yaw rate prediction residual To splice and build The combined state vector at time step As shown in the following formula.

[0062]

[0063] Step 3.3: The policy network synthesizes the state vector. As input, the output is the correction gain for the underlying dynamic parameters, i.e., the action. The motion space is defined as the nonlinear compensation gain of the front and rear axle lateral stiffness. :

[0064] To ensure the fundamental stability of the physical system and prevent vehicle loss of control due to excessive parameter correction, physical boundary constraints are set for the network output layer to strictly limit the compensation gain within the safe envelope of the initial reference lateral stiffness:

[0065] In the formula, These are the initial reference lateral stiffnesses of the front and rear axles, respectively. The preset boundary ratio coefficient is set according to the actual physical characteristics and safety requirements of the vehicle to ensure that the corrected lateral stiffness is still within a reasonable physical range.

[0066] Step 4: Construct a multi-objective reward function that includes rewards for trajectory tracking accuracy and model consistency, which is used to guide the policy network to output the correction gain of model parameters.

[0067] Step 4.1: To guide the policy network towards optimizing for reduced tracking error and model mismatch, a multi-objective reward function is constructed, consisting of a trajectory tracking error penalty term and a model consistency penalty term.

[0068] Step 4.2: Based on the vehicle's lateral tracking error With heading angle error Constructing a trajectory tracking penalty term using a quadratic function The calculation formula is:

[0069] In the formula, Here are the non-negative weight coefficients for the trajectory tracking term. Using a quadratic function can impose a strict quadratic penalty on larger tracking errors, thus effectively incentivizing the agent to improve the truck's trajectory tracking accuracy under complex conditions.

[0070] Step 4.3: Predict residuals based on the dynamic model and construct a model consistency penalty term using a quadratic function. This is used to quantify and provide feedback on the compensation effect of the agent's output parameters on model mismatch. The calculation formula is as follows:

[0071] In the formula, This represents the non-negative weighting coefficient of the model consistency term. This term is calculated by predicting the residuals of lateral displacement. and yaw rate prediction residual Applying a squared penalty guides the correction parameters of the agent's output to minimize the deviation between the prediction equation and the actual physical system.

[0072] Step 4.4: In each environmental interaction during the offline training phase, the agent utilizes the aforementioned multi-objective reward function. The single-step feedback reward is calculated and stored in the experience replay pool. By continuously maximizing the accumulated expected reward, the parameters of the policy network and the value network are iteratively updated until convergence, thereby enabling the policy network to output the optimal lateral stiffness correction gain under time-varying conditions.

[0073] Step 5: Update the core dynamic parameters such as tire lateral stiffness using the modified gain, recalculate the continuous state matrix and control matrix, and obtain the adaptive state matrix and control matrix through discretization, and establish a prediction equation that dynamically matches the complex working conditions.

[0074] Step 5.1: Obtain the lateral stiffness correction gain output by the strategy network at the current moment, add it to the initial reference lateral stiffness obtained from the truck's offline calibration, calculate the equivalent front axle lateral stiffness and equivalent rear axle lateral stiffness under the current complex working condition, and update the core dynamic parameters:

[0075] Step 5.2: Calculate the equivalent lateral stiffness. and Substitute the values ​​back into the unmanned truck dynamics model described in step 2. At the current vehicle operating point, recalculate the Jacobian matrix using Taylor expansion to obtain the updated continuous system state matrix. With continuous control matrix Then the system sampling time was introduced. The forward Euler method is used for time discretization to calculate the adaptive discrete state transition matrix. With discrete control input matrix .

[0076] Step 5.3: Reconstructed Matrix and An adaptive prediction equation for vehicle dynamics that dynamically matches the current operating conditions is established. Within the defined prediction time domain... Within this system, the evolution of its future state can be represented as follows:

[0077] in, For the present moment and the future The predicted value of the vehicle state vector at time t. For the present moment and the future Predicted values ​​of control commands at any given time.

[0078] Step 6: Based on the adaptive prediction equation, construct a quadratic performance index with the goal of minimizing tracking error and control cost. Combine the vehicle's physical hard constraints to perform rolling quadratic optimization solution to achieve high-precision trajectory tracking control of the vehicle under complex working conditions.

[0079] Step 6.1: Set the control time domain as... ,and To minimize both trajectory tracking error and control cost, a quadratic objective function is constructed based on the aforementioned adaptive prediction equation. :

[0080] in, These are the set state error weight matrix, control quantity weight matrix, and control increment weight matrix, respectively. To predict within the time domain The expected vehicle state vector at time t. The control increment is between adjacent time steps; These are the weighting coefficients. These are slack variables.

[0081] Step 6.2: Taking into account the mechanical saturation characteristics of the truck chassis actuators, apply physical hard constraints to the control commands and their increments in the control time domain:

[0082] in, These are the upper and lower bounds of the physical extreme values ​​of the front wheel steering angle and longitudinal acceleration, respectively. These are the upper and lower limits of the extreme values ​​for the front wheel steering angle increment and the longitudinal acceleration increment, respectively. Step 6.3: Transform the quadratic objective function Combining the aforementioned physical hard constraints, this problem is transformed into a standard quadratic programming problem. An online quadratic programming solver is then used to perform optimization calculations on this convex optimization problem, yielding the optimal control increment sequence in the current time domain:

[0083] Extract the first element of the optimal sequence. Combined with the actual control commands from the previous moment Calculate the current Actual control commands at any time This information is then distributed to the underlying drive-by-wire actuators of the unmanned truck. The system time then progresses to... At any given moment, sensor data is reacquired and the process returns to step 1 to enter the next control cycle, thereby achieving high-precision trajectory tracking control of the vehicle under complex working conditions.

[0084] Example 1 The simulation was performed using a two-axle monolithic truck model built into TruckSim, and the results were verified using a co-simulation platform combining MATLAB / Simulink and TruckSim. MPC control cycle. =0.05s, prediction time domain =15 steps, controlling the time domain =5 steps.

[0085] In this embodiment, the real-time operating status of the unmanned truck is obtained through the on-board inertial navigation unit, wheel speed sensor and GPS / RTK differential positioning module, and the vehicle state vector as described in formula (1) is constructed. Based on the desired driving path output by the global path planning module, in the prediction time domain... Discretization along the inner path tangent direction extracts the desired reference trajectory as described in formula (2). Define control commands. And calculate the current lateral tracking error. With heading angle error .

[0086] In this embodiment, a three-degree-of-freedom vehicle single-track dynamics model including longitudinal, lateral, and yaw motions is constructed, and its core dynamic equation is shown in equation (3). Based on the linear tire model, the initial reference front axle lateral stiffness is set. and initial reference rear axle lateral stiffness After linearizing the dynamic system at the operating point, the forward Euler method is used with sampling time... Discretization is performed to establish a baseline discrete prediction model. For each control cycle, the prior prediction value of the MPC at the previous moment is extracted, and subtracted from the actual sensor observation value to obtain the prediction residual vector. The lateral displacement residual is then extracted from this vector. Residual of yaw rate This is used to quantify the parameter mismatch of the current model.

[0087] like Figure 3 As shown, in this embodiment, step 3 uses the DDPG algorithm to construct a parameter adaptive correction agent. Both the Actor network and the Critic network are configured with an online network and a target network, respectively. The target network employs a soft update mechanism to ensure training stability. The agent's state input consists of the lateral tracking error. Heading angle error and its corresponding first rate of change vehicle longitudinal speed yaw rate and the predicted residuals extracted in step 2 and The splicing forms an 8-dimensional integrated state vector as shown in formula (7). The Actor network outputs the correction gain for the underlying dynamic parameters as shown in Equation (8). And by using physical boundary constraints, its amplitude is limited to the safe envelope of the initial reference lateral stiffness, with the boundary scaling factor... The specific physical boundary limit rules are set according to the actual physical characteristics of the vehicle, as shown in formula (9).

[0088] In this embodiment, to ensure that those skilled in the art can reproduce the control effect of the intelligent agent, the specific network structure and hyperparameter settings of the DDPG algorithm are as follows. Both the Actor network and the Critic network employ a fully connected feedforward neural network (Multilayer Perceptron, MLP). The Actor network includes an input layer, three hidden layers, and an output layer. The number of neurons in each hidden layer is 256, 128, and 64, respectively, and the ReLU activation function is used. The output layer has two neurons, corresponding to the nonlinear compensation gain for the anterior and posterior axial lateral stiffness. The Tanh activation function is used to normalize the output, which is then mapped to the actual gain range by multiplying by the physical boundary coefficient. The Critic network employs a dual-stream input structure. The state input stream passes through a hidden layer with 256 neurons, while the action input stream is unprocessed. The two are concatenated and then passed through three hidden layers with 256, 128, and 64 neurons, respectively, ultimately outputting a scalar Q-value. Except for the output layer, all hidden layers employ the ReLU activation function. The learning rate for both the Actor and Critic networks is set to 0.0001. The experience replay pool size is set to 100,000, and the batch size is 256. The target network's soft update coefficients are... Set to 0.005, the discount factor for future rewards. The value was set to 0.99. Furthermore, to enhance the exploratory capabilities during the early stages of training, Ornstein-Uhlenbeck random noise was superimposed on the actions output by the Actor network, where the noise regression rate... Take 0.15, noise variance Take 0.2.

[0089] In this embodiment, step 4 constructs a trajectory tracking error penalty term. Consistency penalty term with model Together they form a multi-objective reward function. Based on the quadratic function of lateral tracking error and heading angle error, a quadratic penalty is applied to the larger tracking deviation, and its calculation formula is shown in Equation (11). Based on the construction of quadratic functions of the lateral displacement prediction residual and the yaw rate prediction residual, the agent is guided to eliminate model mismatch from the root. The calculation formula is shown in Equation (12). During the offline training phase, the agent interacts with the vehicle model in the simulation environment, stores the transfer tuples in the experience replay pool, updates the Critic network through temporal difference error, and updates the Actor network through maximizing the Q-value gradient. After sufficient training, the policy network converges. In this embodiment, in order to balance the safety of trajectory tracking and the stability of model correction, the weight coefficients of the multi-objective reward function are set as follows: the weight coefficient of the trajectory tracking error penalty term. Set to 0.3 The weight is set to 0.4, assigning extremely high penalty weights to lateral displacement and heading angle deviations to ensure driving safety; the weight coefficient of the model consistency penalty term. Set to 0.2. It is set to 0.1. This weighting ratio is designed to guide the agent to smoothly output dynamic parameter corrections while prioritizing ensuring that the vehicle does not deviate from the reference trajectory.

[0090] In this embodiment, the correction gain of the Actor network output is obtained in each control cycle. and The equivalent lateral stiffness, as shown in Equation (13), is obtained by superimposing the initial reference lateral stiffness. Substituting the equivalent lateral stiffness back into the dynamic equation, the Jacobian matrix is ​​recalculated and discretized at the current operating point to obtain the adaptive discrete state transition matrix. and control input matrix This leads to the establishment of an adaptive prediction equation that dynamically matches the current operating conditions. Since the lateral stiffness parameter in the system matrix is ​​corrected in real time by the agent in each control cycle, this prediction equation can adaptively match the actual dynamic characteristics of the vehicle under different loads and road surface conditions.

[0091] In this embodiment, a quadratic objective function is constructed based on the adaptive prediction equation, which includes a state error term, a control quantity term, and a control increment term. Relaxation variables are introduced to ensure feasibility under extreme conditions. The calculation formula of the quadratic objective function is shown in Equation (15). After applying physical hard constraints to the control input and its increment, the optimization problem is transformed into a standard quadratic programming format, and optimization is performed using an online QP solver. The first element of the optimal control increment sequence is extracted, and the current control command is calculated by combining it with the control quantity at the previous moment and sent to the underlying wire-controlled actuator. Subsequently, the system enters the next control cycle to achieve rolling optimization control.

[0092] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0093] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0094] In this invention, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for unmanned truck trajectory tracking based on reinforcement learning-based dynamic correction MPC, characterized in that, Includes the following steps: Step S1: Obtain the real-time operating status and expected reference trajectory of the unmanned truck, and construct the current vehicle state vector and tracking error of the unmanned truck. Step S2: Input the current vehicle state vector into the adaptive agent based on deep reinforcement learning, input the desired reference trajectory and the current vehicle state vector into the MPC controller, the MPC controller calculates the prediction residual based on the deviation between its previous time-aware prediction value and the current vehicle state vector, and inputs the prediction residual into the adaptive agent. Step S3: Construct a comprehensive state vector based on the tracking error, the current vehicle state vector, and the prediction residual. The adaptive agent generates a correction gain based on the comprehensive state vector and inputs it into the MPC controller. Step S4: The MPC controller generates the current control command based on the desired reference trajectory, the current vehicle state vector, and the correction gain; Step S5: Input the current control command into the unmanned truck physical system to implement vehicle control; Step S6: Return to step S1 to achieve dynamic tracking of the unmanned truck trajectory.

2. The unmanned truck trajectory tracking method according to claim 1, characterized in that, The current vehicle state vector Represented as: in, The horizontal axis is... For the horizontal swing angle, and These represent the longitudinal and lateral speeds of the unmanned truck, respectively. This refers to the yaw rate; The expected reference trajectory Represented as: in, To predict the time domain, To predict within the time domain k The expected vehicle state vector at time t.

3. The unmanned truck trajectory tracking method according to claim 2, characterized in that, The tracking error includes lateral error. and yaw angle error .

4. The unmanned truck trajectory tracking method according to claim 2, characterized in that, The prediction residual in step S2 is generated in the following manner: (1) Construct a dynamic model of the unmanned truck that includes longitudinal, lateral, and yaw motions: in, For the overall vehicle quality, These represent the longitudinal and lateral speeds of the unmanned truck, respectively. The yaw rate is angular velocity. For rotational inertia, The distance from the center of mass to the front and rear axles. This is the sum of the longitudinal and lateral forces of each axle in the vehicle coordinate system. The initial reference front axle lateral stiffness, The lateral force of the front tire. The lateral force of the rear tire. This refers to the front tire slip angle. This refers to the rear tire slip angle. (2) The dynamic model of the unmanned truck is expressed as a continuous nonlinear state space form. The continuous state matrix is ​​obtained by expanding it using the first-order Jacobian matrix. With control matrix ; (3) Introducing sampling time The forward Euler method is used to discretize the continuous-time model, and a baseline discrete prediction model is established: in, , , It is the identity matrix. For truncation error, This is the vehicle state vector from the previous moment. The control command from the previous moment; (4) Extract the prior prediction value of the MPC controller for the current time from the previous time step. Combined with the current vehicle state vector Calculate the predicted residual vector : (5) From the predicted residual vector Extracting the lateral displacement residual Residual of yaw rate As the prediction residual.

5. The unmanned truck trajectory tracking method according to claim 4, characterized in that, The integrated state vector is represented as follows: Correction gain Represented as: in, To compensate for the lateral stiffness of the front tires This is for rear tire lateral stiffness compensation.

6. The unmanned truck trajectory tracking method according to claim 1, characterized in that, The current control command Represented as , This refers to the front wheel steering angle. This is the longitudinal acceleration.

7. The unmanned truck trajectory tracking method according to claim 4, characterized in that, The adaptive agent based on deep reinforcement learning undergoes random mini-batch sampling training before use, including the following: (1) Constructing a dynamic model and experience playback pool for unmanned trucks; (2) Input the current control command into the dynamic model of the unmanned truck to perform simulation control and output a new vehicle state; calculate the tracking error based on the new vehicle state and the expected reference trajectory, and calculate the prediction residual based on the new vehicle state and the prior prediction value of the MPC controller at the previous moment. (3) The adaptive agent based on deep reinforcement learning includes an Actor network and a Critic network. The Actor network takes the comprehensive state vector as input and outputs a correction gain. The Critic network takes the comprehensive state vector and the correction gain as input and outputs a scalar Q value. (4) Construct a multi-objective reward function that includes trajectory tracking accuracy and model consistency rewards to guide the Actor network to output corrective gain; (5) Update the core dynamic parameters using the modified gain, recalculate the continuous state matrix and control matrix, and obtain the adaptive state matrix and control matrix by discretization, and establish the vehicle dynamics adaptive prediction equation that is dynamically matched with complex working conditions. (6) Based on the vehicle dynamics adaptive prediction equation, a quadratic performance index is constructed with the goal of minimizing tracking error and control cost. Combined with the physical hard constraints of the unmanned truck, a rolling quadratic optimization solution is performed to obtain new control commands. (7) Take the new control command as the current control command and return to step (2).

8. The unmanned truck trajectory tracking method according to claim 7, characterized in that, The multi-objective reward function is expressed as follows: in, For trajectory tracking penalty items, This is a model consistency penalty term. The non-negative weight coefficients for the trajectory tracking term. represents the non-negative weight coefficients of the model consistency term.

9. The unmanned truck trajectory tracking method according to claim 8, characterized in that, The adaptive prediction equation for vehicle dynamics is expressed as follows: in, For the present moment and the future The predicted value of the vehicle state vector at time t. For the present moment and the future Predicted values ​​of control commands at any given time.

10. The unmanned truck trajectory tracking method according to claim 9, characterized in that, Step (6) specifically includes: a) Set the control time domain as ,and A quadratic objective function is constructed based on the vehicle dynamics adaptive prediction equation. : in, These are the set state error weight matrix, control quantity weight matrix, and control increment weight matrix, respectively. To predict within the time domain The expected vehicle state vector at time t. The control increment is between adjacent time steps; These are the weighting coefficients. These are slack variables; b) Set physical hard constraints for control commands within the control time domain: in, These are the upper and lower bounds of the physical extreme values ​​of the front wheel steering angle and longitudinal acceleration, respectively. These are the upper and lower limits of the extreme values ​​for the front wheel steering angle increment and the longitudinal acceleration increment, respectively. c) The quadratic objective function Combined with physical hard constraints, and transformed into a standard quadratic programming problem solution, the optimal control increment sequence is obtained through optimization calculation: Extract the first element of the optimal control increment sequence. Combined with the control commands from the previous moment The new control command at the current moment is calculated. .

Images