Four-wheel independent steering and driving vehicle path tracking control method based on compound control framework

Abstract

The invention relates to a four-wheel independent steering and driving vehicle path tracking control method based on a compound control framework, which comprises the following steps of: 1, establishing a seven-degree-of-freedom dynamical model of a four-wheel independent steering and driving vehicle, and realizing high-performance path tracking by reasonably distributing input parameters; step 2, establishing a deep reinforcement learning transition model, defining a vehicle state and an input parameter of an auxiliary controller, and designing a system reward function to optimize path tracking control performance; 3, selecting a twin delay depth deterministic strategy gradient algorithm to realize deep reinforcement learning, reducing the over-estimation problem of the algorithm, and improving the learning stability; 4, a double-layer control framework is adopted, an upper-layer controller calculates generalized control quantity based on nonlinear model predictive control, and a lower-layer controller distributes the generalized control quantity to each wheel through sequential quadratic programming so as to solve the transverse and longitudinal coupling problem of the vehicle in path tracking control; and step 5, the auxiliary controller based on deep reinforcement learning carries out real-time compensation on the control variable according to the external disturbance and the vehicle state, so that the control precision and stability of path tracking are improved. By means of the method, stable control over four-wheel independent steering and driving vehicle path tracking in a complex environment is achieved, and the adaptive capacity of the system to external interference is enhanced.

Description

Technical Field

[0001] The present invention relates to a path tracking control method for a four-wheel independent steering and drive vehicle, and is specifically designed to utilize an auxiliary controller based on deep reinforcement learning to implement a compensation control method for control variables. Background Art

[0002] Since the steering angle and torque of each wheel can be independently controlled, four-wheel independent steering and drive vehicles have extremely high path tracking flexibility, which greatly improves the adaptability and robustness in complex environments. However, the complex nonlinear characteristics of four-wheel independent steering and drive vehicles make it difficult to design a path tracking controller with fast response, high tracking accuracy and strong anti-interference ability. In recent years, control strategies and algorithms have been developed for the path tracking control of four-wheel independent steering and drive vehicles. The main control strategies include: first, lateral and longitudinal decoupling control, which achieves tracking of different targets through independent control lines; second, a hierarchical control structure is adopted, in which the upper controller is responsible for calculating the generalized forces required by the vehicle, and the lower controller accurately distributes these forces to each wheel. Commonly used control algorithms include sliding mode control, model predictive control, proportional-integral-differential control and their improved methods. Traditional path tracking control methods still have certain limitations when dealing with the complex nonlinear dynamics and lateral and longitudinal coupling characteristics of four-wheel independent steering and drive vehicles. First, model-based control methods have difficulty coping with external disturbances and drastic environmental changes; second, numerical errors may gradually amplify in recursive calculations. In the case of external disturbances, the error accumulation problem will make it difficult to ensure the stability of path tracking control. Summary of the invention

[0003] The present invention aims to overcome the above-mentioned shortcomings of the prior art and proposes a path tracking control method for a four-wheel independent steering and driving vehicle based on a composite control framework.

[0004] The path tracking control method of a four-wheel independent steering and driving vehicle based on a composite control framework of the present invention combines a model-based two-layer controller with a model-free deep reinforcement learning auxiliary controller, and compensates the control variables of the two-layer controller through the deep reinforcement learning auxiliary controller to achieve accurate tracking of the control target; the specific implementation steps include:

[0005] Step 1, establishing a dynamic model of a four-wheel independent steering and driving vehicle;

[0006] Step 2, establish a deep reinforcement learning transition model; the state vector s of the four-wheel independent steering and driving vehicle includes a measurable state o and a random disturbance d;

[0007] Step 3: Select the twin delayed deep deterministic policy gradient algorithm to implement deep reinforcement learning;

[0008] Step 4: A two-layer control architecture is used to solve the lateral and longitudinal coupling problem of the four-wheel independent steering and drive vehicle in path tracking control. The architecture consists of an upper-layer controller based on nonlinear model predictive control and a lower-layer controller based on sequential quadratic programming. The upper-layer controller calculates the current state error e of the vehicle through the ideal path tracking state information and the current state of the vehicle, and solves the optimal control increment through the nonlinear model predictive controller; the lower-layer controller solves the cost function through sequential quadratic programming and allocates the generalized control quantity u=u of the upper-layer controller c +a, to calculate the lateral force and longitudinal force of each wheel at the next moment;

[0009] Step 5: Use the composite control framework to control the vehicle through two control loops.

[0010] The proposed composite control framework controls the vehicle through two control loops: a two-layer control loop and a model-free deep reinforcement learning auxiliary control loop. The vehicle state information is passed to the upper-level controller to calculate the generalized force required at the next moment. At the same time, a deep reinforcement learning neural network is used as an auxiliary controller to generate auxiliary control items based on the vehicle state information and external disturbances to compensate for the upper-level control variables. The lower-level controller is responsible for distributing the generalized force transmitted by the upper-level controller, and finally gives the control variables under the composite control framework.

[0011] Compared with the prior art, the advantages of the present invention are: a composite control architecture based on a double-layer controller and a deep reinforcement learning auxiliary controller is proposed, the double-layer controller performs closed-loop control of the four-wheel independent steering and driving vehicle, and the deep reinforcement learning auxiliary controller assists in adjusting the wheel force according to external disturbances and vehicle status. On the basis of stable control of the four-wheel independent steering and driving vehicle path tracking, the composite control framework improves the adaptability of the control system to changes in the external environment and improves the control accuracy and stability of path tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Fig. 1(a) and Fig. 1(b) are schematic diagrams of the dynamic model of the four-wheel independent steering and driving vehicle of the present invention: Fig. 1(a) is the vehicle body dynamic model, and Fig. 1(b) is the tire dynamic model;

[0013] Figure 2 It is a schematic diagram of the composite control framework of the present invention;

[0014] Figure 3 It is a comparison chart of the path tracking mean absolute error (MAE) of the composite controller (TDC) and the dual-layer controller (MLC) based on the twin delayed deep deterministic policy gradient algorithm of the present invention;

[0015] Figure 4This is a comparison chart of the path tracking maximum error index (MAX) of the composite controller (TDC) and the dual-layer controller (MLC) based on the twin-delayed deep deterministic policy gradient algorithm;

[0016] Figure 5 It is a comparison chart of the path tracking performance of the composite controller (TDC) and the dual-layer controller (MLC) based on the twin delayed deep deterministic policy gradient algorithm and the reference path. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0018] The present invention is based on a composite control architecture based on a two-layer controller and a deep reinforcement learning auxiliary controller. The architecture uses a two-layer controller to perform closed-loop control of the four-wheel independent steering and drive vehicle, and uses a deep reinforcement learning controller to assist in adjusting the wheel force according to external disturbances and vehicle status. The composite control strategy improves the control system's ability to adapt to changes in the external environment while ensuring the stability of the four-wheel independent steering and drive vehicle path tracking controller, and further improves the path tracking control accuracy and stability under complex external interference conditions. The specific design steps are:

[0019] The four-wheel independent steering and driving vehicle path tracking control method based on the composite control framework of the present invention combines a model-based two-layer controller with a model-free deep reinforcement learning auxiliary controller, and compensates the control variables of the two-layer controller through the deep reinforcement learning auxiliary controller to achieve accurate tracking of the control target. The specific implementation steps include:

[0020] Step 1: Establish a dynamic model of a four-wheel independent steering and driving vehicle. As shown in Figure 1(a), OXYZ and O v X v Y v Z v They are respectively represented as the earth inertial reference system and the vehicle body inertial reference system. Assume that the vehicle is traveling on a flat road. The yaw motion of the vehicle body is mainly controlled by the longitudinal force and lateral force of each tire. For ease of identification, the four tires are indexed as i=fl, fr, rl, rr. In the tire coordinate system, the longitudinal force, lateral force, vertical force and steering angle of each tire are respectively represented by F l,i ,F L,i ,F N,i and δ i The dynamic equation of the vehicle body can be written as:

[0021]

[0022] In the formula, M and I γ represents the vehicle mass and moment of inertia, (Lbi ,l bi ) represents the distance between the tire and the vehicle's center of gravity, and They represent the vehicle's longitudinal acceleration, lateral acceleration and yaw angular acceleration respectively.

[0023] In the design process of the path tracking controller, the longitudinal force and lateral force of each tire are determined by adjusting the steering angle and wheel torque, as shown in Figure 1(b). In the tire coordinate system, the present invention uses the magic formula to represent the nonlinear characteristics of the tire model.

[0024] y=Dsin(Carctan{Hx-E[Hx-arctan(Hx)]}) (2)

[0025] Where y represents the longitudinal force or lateral force, x represents the longitudinal slip rate or slip angle, the peak coefficient D, the stiffness coefficient H, the shape coefficient C and the curvature coefficient E are the fitting coefficients of the empirical formula.

[0026] Longitudinal slip rate λ i The calculation formula is

[0027]

[0028] In the formula, R w and ω i represents the tire radius and angular velocity, u w,i The actual speed at the center of the tire. The tire speed can be calculated using the following formula

[0029]

[0030] The steering angle and torque of the tire can be expressed as

[0031]

[0032] Where, T i is the driving torque, f w is the rolling friction coefficient, I w is the moment of inertia of the tire.

[0033] Based on the above analysis, we established a vehicle dynamics model with seven degrees of freedom. By reasonably allocating input parameters, we hope to effectively control the vehicle state and achieve high-performance path tracking. The control input matrix of the vehicle dynamics model is

[0034] U=[T fl T fr T rl T rr δ fl δ fr δrl δ rr ] (6)

[0035] Wherein, the matrix U includes the driving torque and steering angle of the left front wheel, right front wheel, left rear wheel and right rear wheel of the vehicle. The vehicle state matrix is ​​defined as

[0036] X=[v x v y γ] (7)

[0037] In order to facilitate the design of auxiliary controllers based on deep reinforcement learning, the dynamic equations of the four-wheel independent steering and driving vehicle can be written in nonlinear matrix form

[0038]

[0039] Where U represents the control output matrix of the two-layer controller, represents the vehicle acceleration disturbance matrix, C represents the mapping matrix between the upper controller and the lower controller control quantity, and B represents the vehicle mass matrix. In order to truly reflect the impact of external disturbances on vehicle dynamics, we introduce the control-end disturbance matrix, which represents the lateral and longitudinal disturbance forces acting on each wheel.

[0040]

[0041] The external disturbance in equation (9) can be transferred to the body dynamics model through the tire dynamics model to determine the body acceleration disturbance matrix:

[0042] Step 2, establish a deep reinforcement learning transition model. The state vector s of the four-wheel independent steering and drive vehicle includes a measurable state o and a random disturbance d. This invention uses deep reinforcement learning as an auxiliary controller instead of directly controlling the vehicle end-to-end, so it is assumed that the disturbance d is known. In the deep reinforcement learning training process, the motion control process of the four-wheel independent steering and drive vehicle is written as: Since the auxiliary controller based on deep reinforcement learning needs to control the vehicle in real time according to the vehicle status, the following optimization problem can be constructed:

[0043]

[0044] In the formula, a=[F ax F ay M aφ ] is the auxiliary controller action based on deep reinforcement learning, which represents the auxiliary control input of the driving torque and steering angle of each wheel, and f H (·) represents the four-wheel independent steering and driving vehicle dynamics function, f a (·) represents the auxiliary controller, fuc (·) represents the upper controller, f l (·) represents the lower-level controller, X * represents the reference path state of the longitudinal velocity, lateral velocity, and yaw velocity obtained from the lateral displacement y and the yaw angle γ by differential operation.

[0045] The state of the deep reinforcement learning controller is designed as in Represents the error between the current actual vehicle state and the ideal state, represents the longitudinal velocity error, e Y represents the lateral displacement error, e φ Indicates the heading angle error, u c =[F cx F cy M cφ ] represents the generalized force of the control end.

[0046] The reward function of the system is designed as:

[0047]

[0048] In the formula, K r Represents the positive gain parameter of the path tracking error, K f Positive gain parameter representing continuous control smoothness, e ub Indicates the limit value of tracking error, K b It represents the penalty factor. It indicates the changing speed of the auxiliary control variable, and its purpose is to ensure that the control process has a certain smoothness.

[0049] Step 3, select the Twin Delayed Deep Deterministic Policy Gradient algorithm to implement deep reinforcement learning. Twin Delayed Deep Deterministic Policy Gradient (TD3) is a reinforcement learning algorithm based on the Deep Deterministic Policy Gradient (DDPG) algorithm, which is used to solve the problem of training instability in continuous action space. TD3 reduces the over-estimation problem in traditional algorithms and improves the stability and performance of learning by introducing mechanisms such as dual Q networks, delayed policy updates, and target action noise. The core idea is to use the experience replay pool to store interaction samples with the environment, break the time correlation through random sampling, and train the strategy and Q network more efficiently.

[0050] Before formal training begins, the agent interacts with the environment by randomly selecting actions to accumulate enough initial experience. This step ensures that there are enough diverse samples in the experience replay pool for subsequent training. t Next, randomly select action a t , record the immediate reward r after executing the action t and update the status s t+1 , and the new transfer sample (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool Ω until enough samples are stored in the experience pool. Initialize the policy network π θ , two Q networks and and its corresponding target network π θ '、 and The initial parameters of the target network are the same as those of the current network. The experience replay pool Ω has already started collecting samples in the first step, and these samples will be used in subsequent training stages.

[0051] Randomly sample a batch of samples (s) from the experience replay pool Ω t ,a t ,r t ,s t+1 ). The experience replay pool breaks the temporal correlation of samples by storage and random sampling, effectively reducing the problem of policy overfitting. Then, using the target policy network π θ 'Generate the next state s t+1 The target action a t+1 , and introduce noise N t Smoothing to prevent the policy network from overfitting the action

[0052] a t+1 =π θ ′(s t+1 )+N t (12)

[0053] Using Target Q Network and Calculate the target Q value and take its minimum value to reduce the overestimation problem of the Q value

[0054]

[0055] In the formula, y t is the target Q value, r t For immediate rewards.

[0056] The loss function is constructed as the mean square error between the Q-value estimate and the target Q-value: The stochastic gradient descent method is used to minimize the loss function of the two Q networks. Through the optimization of the loss function, TD3 will gradually adjust the parameters φ1 and φ2 of the Q network to improve the accuracy of the Q value estimation. TD3 introduces a delayed policy update mechanism, that is, the policy network π is updated once every two Q networks are updated. θ The goal of the policy network is to find the action that maximizes the Q value.

[0057] (14) In the formula, π θ TD3 can optimize the policy network based on a more stable Q value estimation by delaying the update, avoiding the policy network from relying on inaccurate Q values ​​too early, and using the soft update method to update the parameters of the target Q network and the target policy network, so that the two target networks are updated smoothly from the main network.

[0058]

[0059] Where τ is the soft update step size. After training, the policy network π θ It has been optimized to the optimal strategy. At this point, for a given state, the optimal action a=π can be directly calculated through the strategy network θ (s), the action a is used to guide the agent’s actual decision in the environment.

[0060] Step 4: In order to solve the lateral and longitudinal coupling problem of four-wheel independent steering and drive vehicles in path tracking control, the patent of this invention adopts a two-layer control architecture, which consists of an upper controller based on nonlinear model predictive control and a lower controller based on sequential quadratic programming. The upper controller calculates the current state error e of the vehicle through the ideal path tracking state information and the current state of the vehicle, and solves the optimal control increment through the nonlinear model predictive controller.

[0061]

[0062] In the formula, e(t k+n |t k ) represents the time t k The state error, Q (k) and R (k) Represent the error weight matrix and the control increment weight matrix respectively, and ε represents the relaxation factor to avoid being unable to find a feasible solution. The generalized control quantity is calculated by the cost function to calculate the state of the vehicle at the next moment

[0063] u c =[F cx F cy M cφ] (17)

[0064] In the formula, F cx 、F cy and M cφ They represent the generalized longitudinal force, lateral force and yaw moment of the vehicle body respectively.

[0065] To balance and minimize wheel utilization, the lower controller solves the cost function through sequential quadratic programming and allocates the generalized control quantity u = u of the upper controller. c +a, to calculate the lateral force and longitudinal force of each wheel at the next moment

[0066]

[0067] In the formula, w1 and w2 represent the tire utilization minimization weight coefficient and tire utilization balance weight coefficient respectively. The lateral force and longitudinal force of each wheel are calculated as the angle and torque respectively through the tire dynamics model, and the vehicle terminal control variable at the next moment is obtained, which is then passed to the vehicle body to realize the closed-loop control of path tracking.

[0068] Step 5. The composite control framework proposed in the present invention controls the vehicle through two control loops: a double-layer control loop and a model-free deep reinforcement learning auxiliary control loop. The vehicle state information is transmitted to the upper-level controller to calculate the generalized force required at the next moment. At the same time, the deep reinforcement learning neural network is used as an auxiliary controller to generate auxiliary control items based on the vehicle state information and external disturbances to compensate for the upper-level control variables. The lower-level controller is responsible for allocating the generalized force transmitted by the upper-level controller, and finally gives the control variables under the composite control framework.

[0069] The contents described in the implementation examples of this specification are only descriptions of the control framework and control process, and the protection scope of the present invention should not be considered to be limited to the specific forms and parameters described in the implementation examples. In addition, the protection scope of the present invention also includes branching algorithms of different deep reinforcement learning.

Claims

1. A path tracking control method for a four-wheel independent steering and driving vehicle based on a composite control framework, characterized in that: The model-based two-layer controller is combined with the model-free deep reinforcement learning-assisted controller. The deep reinforcement learning-assisted controller is used to compensate the control variables of the two-layer controller to achieve accurate tracking of the control target. The specific implementation steps include: Step 1, establishing a dynamic model of a four-wheel independent steering and driving vehicle; Step 2, establish a deep reinforcement learning transition model; the state vector s of the four-wheel independent steering and driving vehicle includes a measurable state o and a random disturbance d; Step 3: Select the twin delayed deep deterministic policy gradient algorithm to implement deep reinforcement learning; Step 4: A two-layer control architecture is used to solve the lateral and longitudinal coupling problem of the four-wheel independent steering and drive vehicle in path tracking control. The architecture consists of an upper-layer controller based on nonlinear model predictive control and a lower-layer controller based on sequential quadratic programming. The upper-layer controller calculates the current state error e of the vehicle through the ideal path tracking state information and the current state of the vehicle, and solves the optimal control increment through the nonlinear model predictive controller; the lower-layer controller solves the cost function through sequential quadratic programming and allocates the generalized control quantity u=u of the upper-layer controller c +a, to calculate the lateral force and longitudinal force of each wheel at the next moment; Step 5: Use the composite control framework to control the vehicle through two control loops.

2. A path tracking control method for a four-wheel independent steering and driving vehicle based on a composite control framework, characterized in that: Step 1 specifically includes: the four tires are indexed as i = fl, fr, rl, rr respectively; in the tire coordinate system, the longitudinal force, lateral force, vertical force and steering angle of each tire are respectively represented by F l,i ,F L,i ,F N,i and δ i Expressed; the dynamic equation of the car body is written as: In the formula, M and I γ represents the vehicle mass and moment of inertia, (L bi ,l bi ) represents the distance between the tire and the vehicle's center of gravity, and They represent the longitudinal acceleration, lateral acceleration and yaw angular acceleration of the vehicle respectively; In the design process of the path tracking controller, the longitudinal force and lateral force of each tire are determined by adjusting the steering angle and wheel torque; in the tire coordinate system, the magic formula is used to express the nonlinear characteristics of the tire model: y=Dsin(Carctan{Hx-E[Hx-arctan(Hx)]}) (2) In the formula, y represents the longitudinal force or lateral force, x represents the longitudinal slip rate or slip angle, the peak coefficient D, stiffness coefficient H, shape coefficient C and curvature coefficient E are the fitting coefficients of the empirical formula; Longitudinal slip rate λ i The calculation formula is: In the formula, R w and ω i represents the tire radius and angular velocity, u w,i The actual speed at the center of the tire. The tire speed can be calculated using the following formula The steering angle and torque of the tire can be expressed as Where, T i is the driving torque, f w is the rolling friction coefficient, I w is the moment of inertia of the tire; Based on the above analysis, a vehicle dynamics model with seven degrees of freedom is established. By reasonably allocating input parameters, the vehicle state is effectively controlled to achieve high-performance path tracking. The control input matrix of the vehicle dynamics model is: U=[T fl T fr T rl T rr d fl d fr d rl d rr ] (6) Wherein, the matrix U includes the driving torque and steering angle of the left front wheel, right front wheel, left rear wheel and right rear wheel of the vehicle; the vehicle state matrix is ​​defined as X=[v x v y γ] (7) In order to facilitate the design of auxiliary controllers based on deep reinforcement learning, the dynamic equations of the four-wheel independent steering and driving vehicle can be written in nonlinear matrix form Where U represents the control output matrix of the two-layer controller, represents the vehicle acceleration disturbance matrix, C represents the mapping matrix between the upper controller and the lower controller control quantity, and B represents the vehicle mass matrix; in order to truly reflect the impact of external disturbances on vehicle dynamics, the control end disturbance matrix is ​​introduced, which represents the lateral and longitudinal disturbance forces acting on each wheel. The external disturbance in equation (9) can be transferred to the body dynamics model through the tire dynamics model to determine the body acceleration disturbance matrix:

3. A path tracking control method for a four-wheel independent steering and driving vehicle based on a composite control framework, characterized in that: Step 2 specifically includes: During the deep reinforcement learning training process, the motion control process of the four-wheel independent steering and driving vehicle is written as: Since the auxiliary controller based on deep reinforcement learning needs to control the vehicle in real time according to the vehicle status, the following optimization problem is constructed: In the formula, a=[F ax F ay M aφ ] is the auxiliary controller action based on deep reinforcement learning, which represents the auxiliary control input of the driving torque and steering angle of each wheel, and f H (·) represents the four-wheel independent steering and driving vehicle dynamics function, f a (·) represents the auxiliary controller, f uc (·) represents the upper controller, f l (·) represents the lower-level controller, X * represents the reference path state of the longitudinal velocity, lateral velocity and yaw velocity obtained from the lateral displacement y and the yaw angle γ by differential operation; The state of the deep reinforcement learning controller is designed as in Represents the error between the current actual vehicle state and the ideal state, represents the longitudinal velocity error, e Y represents the lateral displacement error, e φ Indicates the heading angle error, u c =[F cx F cy M cφ ] represents the generalized force of the control end; The reward function of the system is designed as: In the formula, K r Represents the positive gain parameter of the path tracking error, K f Positive gain parameter representing continuous control smoothness, e ub Indicates the limit value of tracking error, K b It represents the penalty factor. It indicates the changing speed of the auxiliary control variable, and its purpose is to ensure that the control process has a certain smoothness.

4. A four-wheel independent steering and driving vehicle path tracking control method based on a composite control framework, characterized in that: Step 3 specifically includes: before the formal training begins, the agent interacts with the environment by randomly selecting actions to accumulate enough initial experience; this step ensures that there are enough diverse samples in the experience replay pool for subsequent training; for the initial time step t, the agent in each state s t Next, randomly select action a t , record the immediate reward r after executing the action t and update the status s t+1 , and the new transfer sample (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool Ω until enough samples are stored in the experience pool; initialize the policy network π θ , two Q networks and and its corresponding target network π′ θ , and The initial parameters of the target network are the same as those of the current network; the experience replay pool Ω has started collecting samples in the first step, and these samples will be used in the subsequent training phase; Randomly sample a batch of samples (s) from the experience replay pool Ω t ,a t ,r t ,s t+1 ); The experience replay pool breaks the temporal correlation of samples by storage and random sampling, effectively reducing the problem of policy overfitting; then, the target policy network π′ is used θ Generate the next state s t+1 The target action a t+1 , and introduce noise N t Smoothing to prevent the policy network from overfitting the action a t+1 =π′ θ (s t+1 )+N t (12) Using Target Q Network and Calculate the target Q value and take its minimum value to reduce the overestimation problem of the Q value In the formula, y t is the target Q value, r t For immediate rewards; The loss function is constructed as the mean square error between the Q-value estimate and the target Q-value: Use stochastic gradient descent to minimize the loss function of the two Q networks; through the optimization of the loss function, TD3 will gradually adjust the parameters φ1 and φ2 of the Q network to improve the accuracy of Q value estimation; TD3 introduces a delayed policy update mechanism, that is, the policy network π is updated once after every two Q networks are updated θ ; The goal of the policy network is to find the action that maximizes the Q value In the formula, through π θ TD3 uses delayed updates to optimize the policy network based on a more stable Q-value estimation, avoiding premature reliance on inaccurate Q-values ​​by the policy network, and using a soft update method to update the parameters of the target Q network and the target policy network, so that the two target networks are updated smoothly from the main network. Where τ is the soft update step size; after training, the policy network π θ It has been optimized to the optimal strategy; at this time, for a given state s, the optimal action a = π can be directly calculated through the strategy network θ (s), the action a is used to guide the agent’s actual decision in the environment.

5. A four-wheel independent steering and driving vehicle path tracking control method based on a composite control framework, characterized in that: The upper controller described in step 4 calculates the current state error e of the vehicle through the ideal path tracking state information and the current state of the vehicle, and solves the optimal control increment through the nonlinear model predictive controller, specifically including: In the formula, e(t k+n |t k ) represents the time t k The state error, Q (k) and R (k) Represent the error weight matrix and the control increment weight matrix respectively, and ε represents the relaxation factor to avoid being unable to find a feasible solution; the generalized control quantity is calculated by the cost function to calculate the state of the vehicle at the next moment u c =[F cx F cy M cφ ] (17) In the formula, F cx 、F cy and M cφ They represent the generalized longitudinal force, lateral force and yaw moment of the vehicle body respectively; To balance and minimize wheel utilization, the lower controller solves the cost function through sequential quadratic programming and allocates the generalized control quantity u = u of the upper controller. c +a, to calculate the lateral and longitudinal forces of each wheel at the next moment Where w1 and w2 represent the tire utilization minimization weight coefficient and tire utilization balance weight coefficient, respectively. The lateral force and longitudinal force of each wheel are calculated as the steering angle and torque, respectively, through the tire dynamics model to obtain the vehicle terminal control variable at the next moment, which is then passed to the vehicle body to realize closed-loop control of path tracking.

6. A path tracking control method for a four-wheel independent steering and driving vehicle based on a composite control framework, characterized in that: Step 5 specifically includes: a double-layer control loop and a model-free deep reinforcement learning auxiliary control loop; the vehicle state information is transmitted to the upper-level controller to calculate the generalized force required at the next moment; at the same time, the deep reinforcement learning neural network is used as an auxiliary controller to generate auxiliary control items according to the vehicle state information and external disturbances to compensate for the upper-level control variables; the lower-level controller is responsible for allocating the generalized force transmitted by the upper-level controller, and finally gives the control variables under the composite control framework.

Images