Learning enhancement MPC-based steer-by-wire vehicle trajectory tracking control method

Abstract

The invention discloses a steer-by-wire vehicle trajectory tracking control method based on learning enhancement MPC. The method comprises the following steps: establishing a tracking error model combined with vehicle dynamics; a trajectory tracking controller based on MPC is designed, a tracking error model is used as a prediction model of an MPC algorithm, a prediction sequence matrix is constructed, a target trajectory is tracked, the tracking error approaches 0, and the optimal control quantity is obtained through optimization and solution; the MPC algorithm is enhanced by using a depth deterministic strategy gradient algorithm, a terminal cost function of the MPC algorithm is replaced, and an MPC trajectory tracking controller enhanced by using a DDPG algorithm is obtained through training. According to the method, the DDPG algorithm is utilized, the enhanced MPC trajectory tracking controller is obtained through pre-training and combined training, the vehicle can better track a target trajectory in a complex dynamic scene, the transverse instability risk is reduced, and the driving safety and the riding comfort are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a trajectory tracking control method for a steer-by-wire vehicle based on learning-enhanced MPC. Background Art

[0002] With the rapid development of autonomous driving technology, trajectory tracking control, as a core component of vehicle motion control, directly impacts driving safety and ride comfort. In complex dynamic scenarios, traditional model predictive control (MPC) methods demonstrate excellent handling of vehicle dynamics constraints through rolling horizon optimization strategies. However, their performance is highly dependent on accurate vehicle models and appropriate cost function design. In particular, the terminal cost function, a key parameter influencing long-term prediction accuracy, makes traditional empirical design methods difficult to adapt to changing driving environments, leading to decreased trajectory tracking accuracy and the risk of lateral instability.

[0003] Existing solutions based on a single control architecture have significant limitations: pure MPC methods often suffer from increased cumulative errors due to model mismatch when dealing with strongly nonlinear scenarios; while pure reinforcement learning methods, while environmentally adaptive, suffer from low training efficiency and insufficient real-time performance. Some improved solutions attempt to combine MPC and reinforcement learning in a cascade, such as using reinforcement learning to optimize MPC weight parameters or reference trajectories, but fail to achieve deep coupling between the two methods. This fragmented design makes it difficult for short-term optimization and long-term decision-making to achieve synergy. In scenarios such as high-speed emergency obstacle avoidance or continuous curves, safety hazards such as trajectory tracking lag and lateral acceleration exceeding the limit still exist. Therefore, there is an urgent need to build a new control framework that can integrate the advantages of model-driven and data-driven approaches, accurately quantify long-term control benefits through the value function of deep reinforcement learning, and thus improve trajectory tracking robustness and lateral stability margin under complex working conditions. Summary of the Invention

[0004] In order to overcome the above technical problems, the purpose of the present invention is to provide a steer-by-wire vehicle trajectory tracking control method based on learning-enhanced MPC. By using the DDPG algorithm, an enhanced MPC (RL-MPC) trajectory tracking controller is obtained through pre-training and combined training. The method enables the vehicle to better track the target trajectory in complex dynamic scenarios, reduce the risk of lateral instability, and improve driving safety and ride comfort.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A trajectory tracking control method for a steer-by-wire vehicle based on learning-enhanced MPC is described. The steps are as follows:

[0007] Step 1): Establish a tracking error model that incorporates vehicle dynamics;

[0008] Step 2): Design an MPC-based trajectory tracking controller. Use the tracking error model combined with vehicle dynamics established in step 1) as the prediction model of the MPC algorithm, construct a prediction sequence matrix, track the target trajectory, make the tracking error approach 0, and optimize to obtain the optimal control variable.

[0009] Step 3): The Deep Deterministic Policy Gradient (DDPG) algorithm is used to enhance the MPC algorithm by replacing its terminal cost function. This combines the short-term optimization and long-term prediction capabilities of the two algorithms to improve the robustness and control accuracy of the trajectory tracking controller. The DDPG-enhanced MPC (RL-MPC) trajectory tracking controller is obtained through training.

[0010] Furthermore, the step 1) specifically includes:

[0011] Let the state variables be:

[0012]

[0013] The control variables are:

[0014] u=δ f

[0015] Establish the state-space equations of the tracking error model incorporating vehicle dynamics:

[0016]

[0017] in,

[0018] Furthermore, the step 2) specifically includes: designing an MPC-based trajectory tracking controller.

[0019] The above state space equation is discretized using the forward Euler method:

[0020] ξ(k+1)=Aξ(k)+Bu(k)

[0021] Among them, A=I+T s A0; B = T s B0

[0022] Construct the prediction sequence matrix:

[0023]

[0024] in,

[0025] Design objective function:

[0026]

[0027] Where, Q = diag(q1,q2,q3,q4,q5); R = r

[0028] Convert the objective function into a standard quadratic form:

[0029]

[0030] Where H = 2(Θ T Q k Θ+R k ), f T =2(Ψξ) T Q k Θ

[0031] The optimal control sequence [u * (k)u * (k)+1)…u * (k+N c -1)] T The first element u of the optimal control sequence * (k) serves as the actual control input of the system.

[0032] Furthermore, the step 3) specifically includes: using DDPG to enhance the MPC algorithm, replacing its terminal cost function, and obtaining an MPC (RL-MPC) trajectory tracking controller enhanced by the DDPG algorithm through training.

[0033] The state and control quantities of DDPG are defined as:

[0034]

[0035] a=δ f

[0036] Design the reward function for the pre-training phase:

[0037]

[0038] After pre-training, the terminal cost term in the MPC objective function is replaced with the critic network used in the DDPG agent to approximate the value function:

[0039]

[0040] Design the reward function for the combined training phase:

[0041]

[0042] After training, an MPC trajectory tracking controller enhanced by the DDPG algorithm is obtained.

[0043] Beneficial effects of the present invention:

[0044] 1. This paper uses DDPG to replace the terminal cost function of MPC, integrating the advantages of model-driven and data-driven approaches. It not only leverages MPC's short-term optimization capabilities, but also uses DDPG to accurately quantify long-term control benefits. This improves the robustness and control accuracy of the trajectory tracking controller under complex working conditions, overcoming the limitations of traditional MPC's reliance on precise models and manually designed cost functions, as well as the low training efficiency and lack of real-time performance of pure reinforcement learning.

[0045] 2. The present invention uses the DDPG algorithm to obtain an enhanced MPC (RL-MPC) trajectory tracking controller through pre-training and combined training, which can enable the vehicle to better track the target trajectory in complex dynamic scenarios, reduce the risk of lateral instability, and improve driving safety and ride comfort.

[0046] The method of the present invention is simple and broadens the research ideas for solving the control of wire-controlled intelligent vehicle systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the vehicle's two-degree-of-freedom dynamic model.

[0048] Figure 2 Schematic diagram of the vehicle tracking error model.

[0049] Figure 3 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0050] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and drawings. The contents mentioned in the embodiments are not intended to limit the present invention.

[0051] like Figure 3 As shown in Figure 1, a trajectory tracking control method for a steer-by-wire vehicle based on learning-enhanced MPC is as follows:

[0052] Step 1): Build a vehicle dynamics model

[0053] like Figure 1 As shown, only the lateral and yaw motions of the vehicle are considered, and the two-degree-of-freedom differential equation of the vehicle is established:

[0054]

[0055] Where m is the vehicle mass; is the lateral acceleration; is the longitudinal velocity; is the yaw angular velocity; F yf and Fyr are the lateral forces of the front and rear tires respectively; δ f is the front wheel turning angle; I z is the moment of inertia about the z-axis; a and b are the distances from the center of mass to the front and rear axles of the vehicle, respectively.

[0056] Using the linear tire model, the tire lateral force is expressed as:

[0057]

[0058] Among them, C f and C r are the front and rear wheel cornering stiffnesses respectively; α f and α r are the front and rear wheel slip angles, respectively, and their expressions are:

[0059]

[0060] Then the above two-degree-of-freedom differential equation can be rewritten as:

[0061]

[0062] like Figure 2 As shown, a vehicle tracking error model is established.

[0063] Let the road curvature at the target point be ρ T , then the target yaw angular velocity is expressed as:

[0064]

[0065] The target lateral acceleration is expressed as:

[0066]

[0067] The lateral acceleration error is then expressed as:

[0068]

[0069] but

[0070] The yaw angle error is expressed as:

[0071]

[0072] in, is the yaw angle; is the target yaw angle.

[0073] The yaw rate error is expressed as:

[0074]

[0075] The lateral velocity error is expressed as

[0076]

[0077] Combining the tracking error model with vehicle dynamics, we can get:

[0078]

[0079] Let the state variables be:

[0080]

[0081] Among them, e y is the lateral position error; is the lateral velocity error; is the yaw angle error; is the yaw rate error; is the target yaw rate;

[0082] The control variables are:

[0083] u=δ f Among them, δ f is the front wheel turning angle;

[0084] Then the state space equation is expressed as:

[0085]

[0086] in,

[0087] Among them, C f and C r are the front and rear wheel cornering stiffnesses respectively; a and b are the distances from the center of mass to the front and rear axles of the vehicle respectively; m is the vehicle mass; is the longitudinal velocity; I z is the moment of inertia about the z-axis; k is the time step.

[0088] At this point, the tracking error model combined with vehicle dynamics is completed.

[0089] Step 2): Design an MPC-based trajectory tracking controller. Use the tracking error model combined with vehicle dynamics established in step 1) as the prediction model of the MPC algorithm, construct a prediction sequence matrix, track the target trajectory, make the tracking error approach 0, and optimize to solve the optimal control quantity.

[0090] T s is the discrete time step, and the forward Euler method is used to discretize the above state space equation:

[0091]

[0092] Then ξ(k+1)=(I+T s A0)ξ(k)+T s B0u(k)=Aξ(k)+Bu(k)

[0093] Among them, A=I+T s A0; B = T s B0; I is the identity matrix; T s is the discrete time step;

[0094] Let the prediction time domain and control time domain of MPC be N p and N c , and N p ≥N c , then the predicted sequence of state variables is expressed as:

[0095]

[0096] Defining the forecast sequence Control Sequence

[0097] N p is the prediction time domain step size; N c is to control the time domain step size;

[0098] Convert to matrix form:

[0099] ζ(k)=Ψξ(k)+ΘU(k)

[0100] in,

[0101] Considering the minimum tracking error and its rate of change, as well as the minimum control amount, the objective function is established:

[0102]

[0103] Among them, Q and R are the weight matrices corresponding to the state variables and control variables respectively, and their expressions are:

[0104] Q=diag(q1,q2,q3,q4,q5); R=r

[0105] Convert the objective function into matrix form:

[0106] J = ζ T Q k ζ+U T R k U=[Ψξ+ΘU] T Q k [Ψξ+ΘU]+U T R k U

[0107] in, represents the Kronecker product.

[0108] The standard form in which quadratic programming can be solved is:

[0109]

[0110] Convert the objective function into the above form:

[0111] J=[Ψξ+ΘU] T Q k [Ψξ+ΘU]+U T R k U

[0112] =[(Ψξ) T +U T Θ T ]Q k [(Ψξ)+ΘU]+U T R k U

[0113] =(Ψξ) T Q k (Ψξ)+(Ψξ) T Q k ΘU+U T Θ T Q k (Ψξ)+U T Θ T Q k ΘU+U T R k U

[0114] =U T (Θ T Q k Θ+R k )U+2(Ψξ) T Q k ΘU+(Ψξ) T Q k (Ψξ)

[0115] Then H=2(Θ T Q k Θ+R k ), f T =2(Ψξ) T Q k Θ.

[0116] Considering the influence of the tire's lateral characteristics in actual situations, the wheel angle and its rate of change should be limited to a reasonable range. Add the following constraints:

[0117]

[0118] Perform optimization and solve in each control cycle, and control the first element u of the sequence * (k) serves as the actual control input of the system.

[0119] At this point, the design of the MPC-based trajectory tracking controller is completed.

[0120] Step 3): DDPG is used to enhance the MPC algorithm by replacing its terminal cost function. This combines the short-term optimization and long-term prediction capabilities of the two algorithms, improving the robustness and control accuracy of the trajectory tracking controller. The DDPG-enhanced MPC (RL-MPC) trajectory tracking controller is obtained through training.

[0121] The DDPG algorithm is a reinforcement learning algorithm for solving continuous action space problems that combines deep neural networks with deterministic policy gradient methods. Its agent, based on an actor-critic framework, maximizes the expected cumulative long-term reward by searching for the optimal policy. To estimate the policy function and value function, the DDPG agent contains four function approximators:

[0122] The Actor Network is responsible for generating actions. It takes the state of the environment as input and outputs a deterministic action through the forward propagation of the neural network. The parameters of this network are denoted as θ μ , whose goal is to learn a strategy μ(s;θ μ ), that is, output an optimal action a according to the current state s.

[0123] The Critic Network is used to evaluate the value of an action. It takes state s and action a as input and outputs a value function Q(s,a;θ) through a neural network. Q ), represents the expected cumulative reward of performing action a in state s, and the parameter is θ Q Its role is to provide feedback to the actor network and help it adjust its strategy to obtain higher rewards.

[0124] In addition, to improve the stability of the algorithm, DDPG introduces a target actor network and a target critic network, whose structures are the same as the original actor network and critic network, but the parameters are updated slowly. The parameters θ of the target actor network are μ' and the parameters θ of the target critic network Q' The parameters θ of the original network are periodically μ and θ QTo update, a soft update is usually used, i.e., θ←τθ+(1-τ)θ', where τ is a small constant, such as 0.001. This can keep the parameters of the target network relatively stable over a period of time, avoiding drastic fluctuations during training.

[0125] At the same time, DDPG uses the experience replay buffer to store the experience samples (s t ,a t ,r t ,s t+1 ), where s t is the current state, a t is the action performed, r t is the reward obtained, s t+1 During the training process, randomly sampling a batch of experience samples from the buffer for network update can break the correlation between data and improve the stability and convergence of the algorithm.

[0126] For the update of the critic network, the collected experience samples (s t ,a t ,r t ,s t+1 ), calculate the target value according to the Bellman equation:

[0127] y t =r t +γQ'(s t+1 ,μ'(s t+1 θ μ' );θ Q' )

[0128] Here, γ is a discount factor that measures the importance of future rewards.

[0129] The parameters θ of the critic network are then updated by minimizing the mean squared error Q :

[0130]

[0131] Where N is the batch size of the samples. The minimization process is achieved by using an optimization algorithm such as stochastic gradient descent.

[0132] The actor network is updated based on the feedback provided by the critic network by computing the policy gradient:

[0133]

[0134] Then use the optimization algorithm to update the parameters θ of the actor network along the direction of gradient ascent μ , to maximize the Q value, so that the agent can choose the action that can obtain higher rewards.

[0135] By continuously interacting with the environment, storing experience samples, and using experience replay and the target network to update the actor network and the critic network, the DDPG algorithm can gradually learn the optimal strategy to achieve the goal of maximizing the cumulative reward in an environment with a continuous action space.

[0136] Design DDPG's state, action, and reward functions:

[0137] This paper uses DDPG to enhance the MPC algorithm by replacing its terminal cost function. This combines the short-term optimization and long-term prediction capabilities of the two algorithms to improve the robustness and control accuracy of the trajectory tracking controller. Therefore, the state and action quantities of DDPG should be consistent with those of MPC, namely:

[0138]

[0139] a=δ f

[0140] Considering that training directly from randomly initialized states may cause the DDPG algorithm to be unstable in the early stages, especially in complex environments, this paper proposes to pre-train the DDPG agent before combining MPC and DDPG into RL-MPC training, so that it can imitate the MPC strategy of generating optimal actions.

[0141] Specifically, before the merge training, the steering controller adopts the above-mentioned traditional MPC, and DDPG only imitates MPC to generate the optimal action, and the generated action will not be applied to the controlled vehicle.

[0142] Therefore, the reward function in the pre-training stage is designed as:

[0143]

[0144] Among them, δ MPC and δ DDPG Front wheel angle signals generated by MPC and DDPG respectively.

[0145] After pre-training, the terminal cost term in the MPC objective function is replaced by the critic network used in the DDPG agent to approximate the value function, that is:

[0146]

[0147] During combined training, DDPG aims to assist MPC in achieving higher-precision trajectory tracking control under the same prediction horizon. Therefore, with other conditions remaining unchanged, the reward function used to train RL-MPC is designed as:

[0148]

[0149] At this point, the design of the MPC trajectory tracking controller enhanced by the DDPG algorithm is completed.

[0150] The present invention uses the DDPG algorithm to obtain an enhanced MPC (RL-MPC) trajectory tracking controller through pre-training and combined training, which can enable the vehicle to better track the target trajectory in complex dynamic scenarios, reduce the risk of lateral instability, and improve driving safety and ride comfort.

Claims

1. A steer-by-wire vehicle trajectory tracking control method based on learning-enhanced MPC, characterized in that: Here are the steps: Step 1): Establish a tracking error model that incorporates vehicle dynamics; Step 2): Design an MPC-based trajectory tracking controller. Use the tracking error model combined with vehicle dynamics established in step 1) as the prediction model of the MPC algorithm, construct a prediction sequence matrix, track the target trajectory, make the tracking error approach 0, and optimize to obtain the optimal control variable. Step 3): Use the deep deterministic policy gradient algorithm to enhance the MPC algorithm and replace its terminal cost function to combine the short-term optimization and long-term prediction capabilities of the two, improve the robustness and control accuracy of the trajectory tracking controller, and obtain the MPC trajectory tracking controller enhanced by the DDPG algorithm through training.

2. The trajectory tracking control method of a steer-by-wire vehicle based on learning-enhanced MPC according to claim 1, characterized in that: The step 1) specifically includes: Let the state variables be: Among them, e y is the lateral position error; is the lateral velocity error; is the yaw angle error; is the yaw rate error; is the target yaw rate; The control variables are: u=δ f Among them, δ f is the front wheel angle; Establish the state-space equation of the tracking error model incorporating vehicle dynamics: in, Among them, C f and C r are the front and rear wheel cornering stiffnesses respectively; a and b are the distances from the center of mass to the front and rear axles of the vehicle respectively; m is the vehicle mass; is the longitudinal velocity; I z is the moment of inertia about the z-axis; k is the time step.

3. The trajectory tracking control method of a steer-by-wire vehicle based on learning-enhanced MPC according to claim 2, characterized in that: The step 2) specifically includes: discretizing the state space equation using the forward Euler method: ξ(k+1)=Aξ(k)+Bu(k) Among them, A=I+T s A0; B = T s B0; I is the identity matrix; T s is the discrete time step; Construct the prediction sequence matrix: in, N p is the prediction time domain step size; N c is to control the time domain step size; Design objective function: Among them, Q = diag (q1, q2, q3, q4, q5) and R = r are the weight matrices of state variables and control variables respectively; Convert the objective function into a standard quadratic form: where H = 2(Θ T Q k Θ + R k ), f T = 2(Ψξ) T Q k Θ The optimal control sequence [u * (k)u * (k+1)…u * (k+N c -1)] T The first element u of the optimal control sequence * (k) serves as the actual control input of the system.

4. The trajectory tracking control method for steer-by-wire vehicles based on learning-enhanced MPC according to claim 3, characterized in that: The step 3) specifically includes: The state and control quantities of DDPG are defined as: a=δ f Design the reward function for the pre-training phase: Among them, δ MPC and δ DDPG The front wheel angle signals generated by MPC and DDPG respectively; After pre-training, the terminal cost term in the MPC objective function is replaced by the critic network Q used in the DDPG agent to approximate the value function: Design the reward function for the combined training phase: After training, an MPC trajectory tracking controller enhanced by the DDPG algorithm is obtained.

Images