A vehicle platoon planning control method based on a deep reinforcement learning algorithm

Abstract

The application belongs to the technical field of automobile control, and discloses a vehicle formation planning control method based on a deep reinforcement learning algorithm, which comprises the following steps: S1. An oil consumption model considering the influence of vehicle road slope is constructed, and the instantaneous fuel consumption and expected speed of the vehicle are calculated according to the oil consumption model; S2. A 3-DOF vehicle dynamics model is combined with a lane keeping model to establish a vehicle queue model; S3. A local predictive controller is designed for each following vehicle according to the vehicle queue model; S4. Constraint conditions in the iteration process of the deep reinforcement learning algorithm and a reward function with the instantaneous fuel consumption and expected speed of the vehicle as parameters are constructed; and S5. The optimal control strategy of the local predictive controller is solved by using the deep reinforcement learning algorithm, and the optimal control strategy is applied to the target following vehicle through the local predictive controller. As a result, the application is more in line with the characteristics of the vehicle system, such as nonlinearity, multivariable and strong coupling.

Description

Technical Field

[0001] This invention belongs to the field of automotive control technology, specifically relating to a vehicle formation planning and control method based on deep reinforcement learning algorithms. Background Technology

[0002] In recent years, the number of commercial vehicles has surged due to the rapid development of the logistics industry. Statistics show that from January to May 2024, domestic automobile production and sales totaled 11.384 million and 11.496 million units respectively, representing year-on-year increases of 6.5% and 8.3%. Among these, commercial vehicle sales reached 1.731 million units, a year-on-year increase of 7.1%. With the increase in the number of commercial vehicles, fuel demand and exhaust emissions have also increased. Furthermore, vehicle platooning (especially heavy truck platooning) can reduce air resistance between vehicles, thereby reducing emissions and fuel consumption, and improving road capacity. It is precisely because of these potential benefits that vehicle platooning control has attracted increasing attention from scholars.

[0003] Energy-saving research in vehicle platooning mainly includes the vehicle spacing method, traffic oscillation method, and speed planning method. Among them, the vehicle spacing method mainly studies the impact of vehicle spacing strategies on energy saving; the traffic oscillation method aims to avoid frequent acceleration / deceleration and reduce fuel consumption under frequent braking; and speed planning plans the platoon speed according to road conditions to achieve energy-saving requirements.

[0004] The research on vehicle platooning mainly includes longitudinal control and lateral control. The objective of longitudinal control is to achieve the desired speed of the vehicles in the platoon while maintaining the desired distance between adjacent vehicles; the objective of lateral control is to ensure that the vehicles in the platoon travel along designated lanes.

[0005] Existing research on vehicle platooning control mostly employs decoupled control methods, that is, designing separate lateral and longitudinal controllers to achieve lateral lane keeping and longitudinal speed tracking. Specifically, the methods used for lateral control mainly include PID control, fuzzy control, and H-robust control, while the methods used for longitudinal control mainly include sliding mode control, adaptive control, and model predictive control.

[0006] In summary, although the decoupling strategy achieves a certain degree of vehicle formation control, as a nonlinear, multivariable, and strongly coupled system, the lateral and longitudinal coupling effects become particularly significant when the vehicle system is subjected to conditions with greater acceleration, greater lateral and longitudinal forces, or lower road adhesion coefficients. In such cases, the tracking performance of decoupling control may decrease, resulting in a significant reduction in control accuracy. This could lead to vehicle collisions or failure to form a platoon. Summary of the Invention

[0007] In view of this, in order to solve the problems mentioned in the background art, the purpose of this invention is to provide a vehicle formation planning and control method based on deep reinforcement learning algorithm.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A vehicle platooning planning and control method based on deep reinforcement learning algorithm includes the following steps:

[0010] S1. Construct a fuel consumption model that takes into account the influence of road slope on the vehicle, and calculate the instantaneous fuel consumption and desired speed of the vehicle based on the fuel consumption model;

[0011] S2. Combine the 3-DOF vehicle dynamics model with the lane keeping model to establish a vehicle queuing model;

[0012] S3. Design a local predictive controller for each following vehicle based on the vehicle queuing model;

[0013] S4. Construct the constraints in the iteration process of the deep reinforcement learning algorithm and the reward function with the instantaneous fuel consumption and expected speed of the vehicle as parameters;

[0014] S5. Solve the optimal control strategy of the local predictive controller using a deep reinforcement learning algorithm, and apply the optimal control strategy to the target following vehicle through the local predictive controller.

[0015] Preferably, the fuel consumption model is expressed as:

[0016] In the formula, E fuel P and α represent the vehicle's fuel consumption rate and mechanical power, respectively; α represents the fuel-to-air ratio; γ is the calorific value of the fuel; β, σ, ε, and τ are the engine's coefficient of friction, engine speed, displacement, and conversion factor, respectively; and η is the fuel-to-air ratio. f and η c These are the efficiency parameters for the engine and transmission system, respectively.

[0017] Preferably, the 3-DOF vehicle dynamics model is expressed as follows:

[0018]

[0019] In the formula, These are the longitudinal velocity, lateral velocity, and yaw rate, respectively. i x For longitudinal force, F i yf ,F i yr The lateral forces of the front and rear wheels are respectively, m i For vehicle quality, Let δ be the moment of inertia of the vehicle about the z-axis. i For the front wheel steering angle, a i ,b i These are the distances from the center of mass to the front and rear axles, respectively.

[0020] Preferably, the lane keeping model is expressed as follows:

[0021] In the formula, Let L be the desired angular velocity of the heading, and L be the aiming distance. For longitudinal vehicle spacing error, This refers to the lateral positional error between the vehicle and the lane lines. This represents the error between the vehicle's heading angle and the road tangent.

[0022] Preferably, the iterative process of the deep reinforcement learning algorithm includes alternating convergence iterations of the actor policy function neural network and the critic value function neural network.

[0023] Preferably, the actor policy function neural network is used to optimize the policy parameters of the control policy; the critic value function neural network is used to evaluate the merits of the current policy parameters optimized by the actor policy function neural network.

[0024] Preferably, the actor policy function neural network includes one input layer, four fully connected layers, four ReLU activation layers, one tanh layer, and one scaling layer, and the actor policy function neural network takes state s as input and action a as output.

[0025] Preferably, the critic value function neural network includes one input layer, four fully connected layers, four ReLU activation layers, one addition layer, and one output layer, and the critic value function neural network takes state s and action a as input and state-action value q(s, a) as output.

[0026] Preferably, the constraint condition in the iteration process of the deep reinforcement learning algorithm is to impose a constraint range on the output action 'a' of the actor policy function neural network, and the constraint range is expressed as:

[0027] In the formula, S max and S min These represent the maximum and minimum values ​​of state s, respectively.

[0028] Preferably, the reward function, with the vehicle's expected speed as a parameter, is expressed as follows during the iteration of the deep reinforcement learning algorithm:

[0029] In the formula, v err This indicates the error between the vehicle's actual speed and its expected speed.

[0030] Preferably, the reward function in the iterative process of the deep reinforcement learning algorithm, with the instantaneous fuel consumption of the vehicle as a parameter, is expressed as follows:

[0031] In the formula, e fuel This indicates the vehicle's instantaneous fuel consumption.

[0032] Preferably, the method of using deep reinforcement learning algorithms to solve for the optimal control strategy of the local predictive controller includes:

[0033] Initialize the neural network weights θ for the actor policy function and the neural network weights ω for the critic value function;

[0034] The actor policy function neural network obtains action a based on the current state s of the target vehicle being followed, and applies action a to the target vehicle to obtain a new state s and an immediate reward r.

[0035] The actor policy function neural network obtains a new action a based on the new state s;

[0036] The critic value function neural network evaluates and scores actions a and a' to obtain q(s, a) and q(s, a), and then calculates the predicted value q(s, a) and expected value y of the critic value function neural network according to the Bellman equation. t The error between them is TDerror: TDerror = y t -q(s,a), y t =r + γq(s, a);

[0037] The optimal control strategy is obtained by iteratively updating the weights θ and ω using gradient descent. And L(θ) = q(s,a),

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] (1) This invention completes the horizontal and vertical coupling modeling of vehicle formation. Compared with the traditional decoupling control method, this invention is more in line with the characteristics of nonlinearity, multivariable and strong coupling of vehicle system. In addition, under the distributed model predictive control framework, a local predictive controller is designed for each following vehicle, so as to transform the global optimization problem of vehicle formation into the local optimization problem of each following vehicle, thus avoiding the burden of centralized control.

[0040] (2) This invention uses a deep reinforcement learning algorithm for policy optimization. Specifically, it utilizes alternating critic training (policy evaluation) and actor training (policy improvement) to enable the actor network to output a deterministic optimal control policy. Compared with common policy-based learning methods, this invention can accelerate policy updates and thus reduce the number of learning iterations.

[0041] (3) This invention introduces a fuel consumption model and constructs constraints and a reward function based on the fuel consumption model during the iteration process of the deep reinforcement learning algorithm. This not only reduces the proportion of invalid training in the policy optimization process, but also improves the deep learning efficiency of the optimal policy. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the 3-DOF vehicle dynamics model of the present invention;

[0043] Figure 2 This is a control flowchart of the local predictive controller of the present invention;

[0044] Figure 3 This is an iterative flowchart of the deep reinforcement learning algorithm of the present invention;

[0045] Figure 4 This is a network structure diagram of the deep reinforcement learning algorithm of this invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] A vehicle platooning planning and control method based on deep reinforcement learning algorithm includes the following steps:

[0048] S1. Construct a fuel consumption model that takes into account the influence of road slope on the vehicle, and calculate the instantaneous fuel consumption and desired speed of the vehicle based on the fuel consumption model.

[0049] According to Newton's second law, the longitudinal dynamics model of a vehicle can be expressed as:

[0050]

[0051] Where: s0 and v0 represent the longitudinal position and longitudinal velocity of the vehicle, respectively, and F e,0 This indicates the force provided by the engine. This indicates the resistance experienced by the vehicle, and

[0052] Specifically:

[0053]

[0054] Where: C tf,0 C is the rolling resistance coefficient of the vehicle. aero,0 m0 and A represent the air drag coefficient, vehicle mass, and vehicle frontal area, respectively, while ρ, g, and θ represent air density, gravitational acceleration, and ramp inclination angle, respectively.

[0055] Generally, the mechanical power of a commercial vehicle needs to overcome the effects of gravity, and the vehicle's fuel consumption is closely related to its mechanical power. The expression for mechanical power is P = (F... e,0 +F tf,0 +F aero,0 +F gra,0 )v0.

[0056] In summary, the fuel consumption model relating vehicle fuel consumption rate and mechanical power is as follows:

[0057]

[0058] In the formula, E fuel P and α represent the vehicle's fuel consumption rate and mechanical power, respectively; α represents the fuel-to-air ratio; γ is the calorific value of the fuel; β, σ, ε, and τ are the engine's coefficient of friction, engine speed, displacement, and conversion factor, respectively; and η is the fuel-to-air ratio. f and η c These are the efficiency parameters for the engine and transmission system, respectively.

[0059] For the above fuel consumption model, the economic speed of the vehicle is planned by minimizing vehicle fuel consumption, and this speed is used as the expected speed of the fleet.

[0060] S2. Combine the 3-DOF vehicle dynamics model with the lane keeping model to establish a vehicle queuing model.

[0061] like Figure 1 As shown, the 3-DOF vehicle dynamics model is expressed as follows:

[0062]

[0063] In the formula, These are the longitudinal velocity, lateral velocity, and yaw rate, respectively. i x For longitudinal force, F i yf ,F i yr The lateral forces of the front and rear wheels are respectively, m iFor vehicle quality, Let δ be the moment of inertia of the vehicle about the z-axis. i For the front wheel steering angle, a i ,b i These are the distances from the center of mass to the front and rear axles, respectively.

[0064] The lane keeping model is expressed as follows:

[0065] In the formula, Let L be the desired angular velocity of the heading, and L be the aiming distance. For longitudinal vehicle spacing error, This refers to the lateral positional error between the vehicle and the lane lines. This represents the error between the vehicle's heading angle and the road tangent.

[0066] In summary, by combining the above 3-DOF vehicle dynamics model with the lane keeping model, the vehicle queuing model can be obtained as follows:

[0067]

[0068] Take state variables Control quantity Take sampling time T s Discretizing the above vehicle queue model yields its discrete form: x i (k+1)=f(x i (k), u i (k)); where x i (k) represents the state variable, u i (k) represents the input quantity.

[0069] S3. Design a local predictive controller for each following vehicle based on the vehicle queuing model;

[0070] S4. Construct the constraints in the iteration process of the deep reinforcement learning algorithm and the reward function with the instantaneous fuel consumption and expected speed of the vehicle as parameters;

[0071] (41) Constructing neural networks for actor policy function and critic value function.

[0072] like Figure 3 As shown, the iterative process of the deep reinforcement learning algorithm of the present invention specifically includes alternating convergence iterations of the actor policy function neural network and the critic value function neural network, wherein the actor policy function neural network is used to optimize the policy parameters of the control policy; and the critic value function neural network is used to evaluate the merits of the current policy parameters optimized by the actor policy function neural network.

[0073] according to Figure 4 The structure shown constructs an actor policy function neural network, which includes one input layer, four fully connected layers, four ReLU activation layers, one tanh layer, and one scaling layer. The actor policy function neural network takes state s as input and action a as output.

[0074] according to Figure 4 The structure shown constructs a critic value function neural network, which includes one input layer, four fully connected layers, four ReLU activation layers, one addition layer, and one output layer. The critic value function neural network takes state s and action a as inputs and state-action value q(s, a) as output.

[0075] (42) Set constraints and reward functions

[0076] Constraints:

[0077] To prevent the agent from blindly exploring, a constraint range is imposed on the output action 'a' of the actor policy function neural network, thereby ensuring that the output action of the overall network structure is within the specified range and thus reducing training time.

[0078] Define is_done as the training completion / termination symbol. During the interaction between the actor policy function neural network and the critic value function neural network, if the returned state is not within the expected range, then the current training process is terminated and training is restarted; otherwise, training continues.

[0079] From the above, the expression for the constraint condition can be defined as follows:

[0080]

[0081] In the formula, S max and S min These represent the maximum and minimum values ​​of state s, respectively.

[0082] reward function

[0083] Whenever a vehicle collision occurs, virtual reset technology is used, which means that the training does not end directly, but reverts to the previous safe state to continue learning. This not only reduces the proportion of ineffective training, but also effectively improves the efficiency of training and learning.

[0084] ① The vehicle needs to maintain the desired speed. The error between the actual speed and the desired speed is defined as v. errWhen the speed error is zero, the vehicle maintains the desired speed, and an additional reward value of 0 is given; otherwise, the reward value is zero. Thus, the reward function with the desired vehicle speed as the parameter is expressed as:

[0085]

[0086] ②Speed ​​control is not only related to the driving safety of the convoy, but also directly affects fuel economy. Therefore, e is defined as... fuel Let N represent the instantaneous fuel consumption of the vehicle. During training, when the vehicle speed meets the safe distance requirement, fuel consumption should be minimized as much as possible, and a positive reward of N should be given at this time; otherwise, the reward value is zero. Thus, the reward function with the instantaneous fuel consumption of the vehicle as a parameter is expressed as follows:

[0087]

[0088] ③ Define d in d represents the car-to-car distance within a convoy. des This represents the desired internal vehicle spacing, with a spacing error of d. err =d in -d des When d err When the value is zero, the desired car-to-car distance is maintained within the platoon, and an additional reward value K is given; otherwise, the reward value is zero. Thus, the reward function, with the car-to-car distance within the platoon as the parameter, is expressed as:

[0089]

[0090] ④ Define d pre d represents the distance between the convoy and the vehicle ahead. safe This represents the safe distance between the convoy and the vehicle ahead, when both meet condition d. pre ≥d safe When d is considered to be driving safely, the convoy will receive an additional bonus value L. pre <d safe When a collision is considered to be at risk, the convoy receives a negative reward L as a penalty. Thus, the reward function, with the distance between the convoy and the vehicle ahead as a parameter, is expressed as:

[0091]

[0092] ⑤ Define e y Indicates the lateral position deviation of the vehicle, when e y When the value is zero, the desired vehicle spacing is maintained within the platoon, and an additional reward value M is given; otherwise, the reward value is zero. Thus, the reward function, with the lateral position deviation of the vehicles as a parameter, is expressed as:

[0093]

[0094] In summary, the present invention can specifically provide a composite reward function, expressed as follows:

[0095] R = -(0.1d) err +0.5(d pre -d safe )+0.1e y +0.1e fuel +0.1v err )+K+L+M+N+O.

[0096] S5. Solve the optimal control strategy of the local predictive controller using a deep reinforcement learning algorithm, and apply the optimal control strategy to the target following vehicle through the local predictive controller.

[0097] according to Figure 2 and Figure 3 As shown, the method of using deep reinforcement learning algorithms to solve for the optimal control strategy of the local predictive controller includes:

[0098] Initialize the neural network weights θ for the actor policy function and the neural network weights ω for the critic value function;

[0099] The actor policy function neural network obtains action a based on the current state s of the target vehicle being followed, and applies action a to the target vehicle to obtain a new state s and an immediate reward r.

[0100] The actor policy function neural network obtains a new action a based on the new state s;

[0101] The critic value function neural network evaluates and scores actions a and a' to obtain q(s, a) and q(s, a), and then calculates the predicted value q(s, a) and expected value y of the critic value function neural network according to the Bellman equation. t The error between them is TDerror: TDerror = y t -q(s,a), y t =r + γq(s, a);

[0102] The optimal control strategy is obtained by iteratively updating the weights θ and ω using gradient descent. And L(θ) = q(s,a),

[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A vehicle platooning planning and control method based on deep reinforcement learning algorithm, characterized in that, Includes the following steps: S1. Construct a fuel consumption model that takes into account the influence of road slope on the vehicle, and calculate the instantaneous fuel consumption and desired speed of the vehicle based on the fuel consumption model; The fuel consumption model is expressed as follows: In the formula, and These represent the vehicle's fuel consumption rate and mechanical power, respectively. Indicates the ratio of fuel to air. It is the calorific value of fuel oil. , , and These are the engine's coefficient of friction, engine speed, displacement, and conversion factor. and These are the efficiency parameters for the engine and transmission system, respectively. S2. Combine the 3-DOF vehicle dynamics model with the lane keeping model to establish a vehicle queuing model; S3. Design a local predictive controller for each following vehicle based on the vehicle queuing model; S4. Construct the constraints in the iteration process of the deep reinforcement learning algorithm and the reward function with the instantaneous fuel consumption and expected speed of the vehicle as parameters; The iterative process of the deep reinforcement learning algorithm includes alternating convergence iterations of the actor policy function neural network and the critic value function neural network; the actor policy function neural network is state-based. Input quantity, action For output quantity; The constraint during the iteration process of the deep reinforcement learning algorithm is the output action of the actor policy function neural network. Apply a constraint range, and the constraint range is expressed as: In the formula, and Representing states respectively The maximum and minimum values; The error between the actual speed and the desired speed of a vehicle is defined as: The reward function, with the vehicle's expected speed as a parameter, is expressed as follows: ; definition The instantaneous fuel consumption of the vehicle is represented by the reward function, which takes the instantaneous fuel consumption of the vehicle as a parameter. ; definition Indicates the car-to-car distance within the convoy. This represents the desired internal vehicle spacing, with a spacing error of . The reward function, which takes the car-to-car distance within the convoy as a parameter, is expressed as: ; definition Indicates the distance between the convoy and the vehicles ahead. Let represent the safe distance between the convoy and the vehicle ahead. The reward function, which takes the distance between the convoy and the vehicle ahead as a parameter, is expressed as: ; definition The lateral position deviation of the vehicle is represented by the reward function, which takes the lateral position deviation of the vehicle as a parameter. ; Provide a composite reward function, expressed as: ; S5. Solve the optimal control strategy of the local predictive controller using a deep reinforcement learning algorithm, and apply the optimal control strategy to the target following vehicle through the local predictive controller.

2. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 1, characterized in that, The 3-DOF vehicle dynamics model is expressed as follows: In the formula, These are longitudinal velocity, lateral velocity, and yaw rate, respectively. For longitudinal force, , These are the lateral forces of the front and rear wheels, respectively. For vehicle quality, For vehicles to bypass Moment of inertia of the shaft, For the front wheel steering angle, , These are the distances from the center of mass to the front and rear axles, respectively.

3. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 2, characterized in that, The lane keeping model is expressed as follows: In the formula, For the desired heading angular velocity, Pre-aiming distance, For longitudinal vehicle spacing error, This refers to the lateral positional error between the vehicle and the lane lines. This represents the error between the vehicle's heading angle and the road tangent.

4. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 3, characterized in that: The actor policy function neural network is used to optimize the policy parameters of the control policy; The critic value function neural network is used to evaluate the quality of the current policy parameters optimized by the actor policy function neural network.

5. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 4, characterized in that: The actor policy function neural network includes one input layer, four fully connected layers, four ReLU activation layers, one tanh layer, and one scaling layer, and the actor policy function neural network is state-based. Input quantity, action For output quantity; The critic value function neural network includes one input layer, four fully connected layers, four ReLU activation layers, one addition layer, and one output layer, and the critic value function neural network is state-based. and actions Input quantity, in terms of state-action value This is the output quantity.

6. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 5, characterized in that, The method of using deep reinforcement learning algorithms to solve for the optimal control strategy of the local predictive controller includes: Initialize the neural network weights of the actor policy function. With the neural network weights of the critical value function ; The actor policy function neural network is based on the current state of the target following vehicle. Acquire action and the action The target follows the vehicle to obtain a new state. and instant rewards ; The actor policy function neural network adjusts according to the new state. Acquire new actions ; The critical value function neural network evaluates actions. and The evaluation and scoring were conducted to obtain and Then, the predicted value of the critic value function neural network is calculated according to the Bellman equation. Compared with expected value Error between : , ; The gradient descent method is used to minimize the iterative updates of the weights. and the weights To obtain the optimal control strategy ,and , .

Images