Vehicle formation planning control method based on deep reinforcement learning algorithm

Abstract

The invention belongs to the technical field of automobile control, and discloses a vehicle formation planning control method based on a deep reinforcement learning algorithm, which comprises the following steps: S1, constructing a fuel consumption model considering the influence of a vehicle road slope, and calculating the instantaneous fuel consumption and expected speed of a vehicle according to the fuel consumption model; s2, combining the 3-DOF vehicle dynamics model with the lane keeping model, and establishing a vehicle queue model; s3, designing a local prediction controller for each following vehicle according to the vehicle queue model; s4, constructing constraint conditions in the iteration process of the deep reinforcement learning algorithm and a reward function taking the instantaneous fuel consumption and the expected speed of the vehicle as parameters; and S5, solving an optimal control strategy of the local prediction controller by using a deep reinforcement learning algorithm, and acting the optimal control strategy on the target following vehicle through the local prediction controller. In conclusion, the method better conforms to the characteristics of nonlinearity, multivariate and strong coupling of a vehicle system.

Description

Technical Field

[0001] The present invention belongs to the field of automobile control technology, and specifically relates to a vehicle formation planning and control method based on a deep reinforcement learning algorithm. Background Art

[0002] In recent years, due to the rapid development of the logistics industry, the number of commercial vehicles has surged. According to statistics, from January to May 2024, the cumulative production and sales of domestic automobiles were 11.384 million and 11.496 million respectively, with a cumulative year-on-year increase of 6.5% and 8.3% respectively. Among them, the cumulative sales of commercial vehicles were 1.731 million, a cumulative year-on-year increase of 7.1%. With the increase in the number of commercial vehicles, fuel demand and exhaust emissions have also increased. In addition, vehicle platoons (especially heavy truck platoons) can reduce air resistance between vehicles, thereby reducing emissions and fuel consumption, and improving road capacity. It is precisely because of these potential benefits that vehicle platoon control has attracted more and more scholars.

[0003] The research on energy saving of vehicle formation mainly includes the vehicle spacing method, traffic oscillation method and speed planning method. Among them: the vehicle spacing method mainly studies the impact of the vehicle spacing strategy of vehicle formation on energy saving; the traffic oscillation method is to avoid frequent acceleration / deceleration and reduce fuel consumption under frequent braking; and speed planning is to plan the speed of the fleet according to the road conditions to achieve energy saving requirements.

[0004] The research content of vehicle platooning mainly includes longitudinal control and lateral control. The goal of longitudinal control is to achieve the desired speed of vehicles in the platoon and maintain the desired distance between adjacent vehicles; the task of lateral control is to ensure that vehicles in the platoon drive along the designated lane.

[0005] Existing research on vehicle platoon control mostly adopts decoupled control methods, that is, separately designing lateral controllers and longitudinal controllers to achieve lateral lane keeping and longitudinal speed tracking. Specifically, the methods used for lateral control mainly include PID control, fuzzy control, H robust control, etc., and the methods used for longitudinal control mainly include sliding mode control, adaptive control, model predictive control, etc.

[0006] In summary, although the decoupling strategy achieves a certain degree of vehicle formation control, the vehicle system is a nonlinear, multivariable, and strongly coupled system. When it comes to working conditions with greater acceleration, greater lateral and longitudinal forces, or lower road adhesion coefficients, its lateral and longitudinal coupling effects become particularly significant. At this time, the tracking performance of the decoupling control may be reduced, and the control accuracy will be significantly reduced accordingly, which may cause vehicle collisions and inability to achieve formation. Summary of the invention

[0007] In view of this, in order to solve the problems raised in the above background technology, the purpose of the present invention is to provide a vehicle formation planning and control method based on deep reinforcement learning algorithm.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A vehicle formation planning and control method based on a deep reinforcement learning algorithm comprises the following steps:

[0010] S1. Constructing a fuel consumption model that takes into account the influence of the road slope of the vehicle, and calculating the instantaneous fuel consumption and expected speed of the vehicle according to the fuel consumption model;

[0011] S2. Combine the 3-DOF vehicle dynamics model with the lane keeping model to establish a vehicle platoon model;

[0012] S3. Design a local predictive controller for each following vehicle based on the vehicle platoon model;

[0013] S4. Construct constraints in the iterative process of the deep reinforcement learning algorithm and a reward function with the vehicle's instantaneous fuel consumption and expected speed as parameters;

[0014] S5. Use a deep reinforcement learning algorithm to solve the optimal control strategy of the local predictive controller, and use the local predictive controller to apply the optimal control strategy to the target following vehicle.

[0015] Preferably, the fuel consumption model is expressed as:

[0016] In the formula, E fuel and P represent the fuel consumption rate and mechanical power of the vehicle respectively, α represents the ratio of fuel to air, γ is the calorific value of the fuel, β, σ, ε and τ are the friction coefficient, engine speed, exhaust volume and conversion coefficient of the engine respectively, and η f and η c are the efficiency parameters of the engine and transmission system respectively.

[0017] Preferably, the 3-DOF vehicle dynamics model is expressed as:

[0018]

[0019] In the formula, are longitudinal velocity, lateral velocity and yaw rate respectively, F i x is the longitudinal force, F i yf ,F i yr are the lateral forces of the front and rear wheels, m i is the vehicle mass, is the moment of inertia of the vehicle around the z-axis, δ i is the front wheel turning angle, a i ,b i are the distances from the center of mass to the front and rear axles, respectively.

[0020] Preferably, the lane keeping model is expressed as:

[0021] In the formula, is the expected heading angular velocity, L is the preview distance, is the longitudinal inter-vehicle distance error, is the lateral position error between the vehicle and the lane line, is the heading angle error between the vehicle heading angle and the road tangent.

[0022] Preferably, the iterative process of the deep reinforcement learning algorithm includes alternating convergence iterations of the actor strategy function neural network and the critic value function neural network.

[0023] Preferably, the actor strategy function neural network is used to optimize the strategy parameters of the control strategy; the critic value function neural network is used to evaluate the pros and cons of the current strategy parameters optimized by the actor strategy function neural network.

[0024] Preferably, the actor strategy function neural network includes 1 input layer, 4 fully connected layers, 4 ReLU activation layers, 1 tanhLayer and 1 scalingLayer, and the actor strategy function neural network takes state s as input and action a as output.

[0025] Preferably, the critic value function neural network includes 1 input layer, 4 fully connected layers, 4 ReLU activation layers, 1 addition layer and 1 output layer, and the critic value function neural network takes state s and action a as input and state-action value q(s, a) as output.

[0026] Preferably, the constraint condition in the iterative process of the deep reinforcement learning algorithm is to impose a constraint range on the output action a of the actor strategy function neural network, and the constraint range is expressed as:

[0027] In the formula, S max and S min Represent the maximum and minimum values ​​of state s respectively.

[0028] Preferably, the reward function with the expected speed of the vehicle as a parameter in the iterative process of the deep reinforcement learning algorithm is expressed as:

[0029] In the formula, v err Indicates the error between the actual speed of the vehicle and the expected speed;

[0030] Preferably, the reward function with the instantaneous fuel consumption of the vehicle as a parameter in the iterative process of the deep reinforcement learning algorithm is expressed as

[0031] In the formula, e fuel Indicates the instantaneous fuel consumption of the vehicle.

[0032] Preferably, the method of solving the optimal control strategy of the local predictive controller using a deep reinforcement learning algorithm includes:

[0033] Initialize the actor strategy function neural network weight θ and the critic value function neural network weight ω;

[0034] The actor strategy function neural network obtains action a according to the current state s of the target following vehicle, and applies action a to the target following vehicle to obtain a new state s and an immediate reward r;

[0035] The actor strategy function neural network obtains a new action a according to the new state s;

[0036] The critic value function neural network evaluates and scores actions a and a to obtain q(s, a) and q(s, a), and then calculates the predicted value q(s, a) and expected value y of the critic value function neural network according to the Bellman equation. t The error between TDerror: TDerror = y t -q(s,a),y t =r+γq(s,a);

[0037] The gradient descent method is used to minimize and iteratively update the weight θ and the weight ω to obtain the optimal control strategy And L(θ)=q(s,a),

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] (1) The present invention completes the lateral and longitudinal coupling modeling of vehicle formations. Compared with the traditional decoupling control method, the present invention is more in line with the nonlinear, multivariable, and strongly coupled characteristics of vehicle systems. In addition, under the distributed model predictive control framework, a local predictive controller is designed for each following vehicle, thereby converting the global optimization problem of the vehicle formation into a local optimization problem of each following vehicle, avoiding the burden of centralized control.

[0040] (2) The present invention adopts a deep reinforcement learning algorithm for strategy optimization. Specifically, it uses alternating critic training (strategy evaluation) and actor training (strategy improvement) to make the actor network output a deterministic optimal control strategy. Compared with common policy-based learning methods, the present invention can accelerate strategy updates and thus reduce the number of learning times.

[0041] (3) The present invention introduces a fuel consumption model and constructs constraints and a reward function based on the fuel consumption model during the iteration process of the deep reinforcement learning algorithm, thereby not only reducing the proportion of invalid training in the strategy optimization process, but also improving the deep learning efficiency of the optimal strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of the 3-DOF vehicle dynamics model of the present invention;

[0043] Figure 2 is a control flow chart of the local predictive controller of the present invention;

[0044] Figure 3 It is an iterative flow chart of the deep reinforcement learning algorithm of the present invention;

[0045] Figure 4 This is a network structure diagram of the deep reinforcement learning algorithm of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] A vehicle formation planning and control method based on a deep reinforcement learning algorithm comprises the following steps:

[0048] S1. Construct a fuel consumption model that takes into account the influence of the road slope of the vehicle, and calculate the instantaneous fuel consumption and expected speed of the vehicle based on the fuel consumption model.

[0049] According to Newton's second law, the longitudinal dynamics model of the vehicle can be expressed as:

[0050]

[0051] Where: s0 and v0 represent the longitudinal position and longitudinal velocity of the vehicle respectively, F e,0 represents the force provided by the engine, represents the resistance experienced by the vehicle, and

[0052] Specifically:

[0053]

[0054] Where: C tf,0 is the rolling resistance coefficient of the vehicle, C aero,0 , m0 and A represent the air resistance coefficient, vehicle mass and frontal area of ​​the vehicle respectively, ρ, g and θ represent the air density, gravitational acceleration and ramp inclination angle respectively.

[0055] Generally speaking, the mechanical power of a commercial vehicle needs to overcome the influence of gravity, and the fuel consumption of the vehicle is closely related to the mechanical power, where the expression of mechanical power is P = (F e,0 +F tf,0 +F aero,0 +F gra,0 )v0.

[0056] In summary, the fuel consumption model of the vehicle's fuel consumption rate and mechanical power is:

[0057]

[0058] In the formula, E fuel and P represent the fuel consumption rate and mechanical power of the vehicle respectively, α represents the ratio of fuel to air, γ is the calorific value of the fuel, β, σ, ε and τ are the friction coefficient, engine speed, exhaust volume and conversion coefficient of the engine respectively, and η f and η c are the efficiency parameters of the engine and transmission system respectively.

[0059] For the above fuel consumption model, the economic speed of the vehicle is planned by minimizing the vehicle fuel consumption, and this speed is used as the expected speed of the fleet.

[0060] S2. Combine the 3-DOF vehicle dynamics model with the lane keeping model to establish a vehicle platoon model.

[0061] like Figure 1 As shown, the 3-DOF vehicle dynamics model is expressed as:

[0062]

[0063] In the formula, are longitudinal velocity, lateral velocity and yaw rate respectively, F i x is the longitudinal force, F i yf ,F i yr are the lateral forces of the front and rear wheels, m iis the vehicle mass, is the moment of inertia of the vehicle around the z-axis, δ i is the front wheel turning angle, a i ,b i are the distances from the center of mass to the front and rear axles, respectively.

[0064] The lane keeping model is expressed as:

[0065] In the formula, is the expected heading angular velocity, L is the preview distance, is the longitudinal inter-vehicle distance error, is the lateral position error between the vehicle and the lane line, is the heading angle error between the vehicle heading angle and the road tangent.

[0066] In summary, by combining the above 3-DOF vehicle dynamics model with the lane keeping model, the vehicle platoon model can be obtained as follows:

[0067]

[0068] Take the state Control volume Take the sampling time T s , after discretizing the above vehicle platoon model, the discrete form of the vehicle platoon model can be obtained: i (k+1)=f(x i (k),u i (k)); where x i (k) is the state quantity, u i (k) is the input quantity.

[0069] S3. Design a local predictive controller for each following vehicle based on the vehicle platoon model;

[0070] S4. Construct constraints in the iterative process of the deep reinforcement learning algorithm and a reward function with the vehicle's instantaneous fuel consumption and expected speed as parameters;

[0071] (41) Build the actor strategy function neural network and the critic value function neural network

[0072] like Figure 3 As shown, the iterative process of the deep reinforcement learning algorithm of the present invention specifically includes the alternating convergence iteration of the actor policy function neural network and the critic value function neural network, and the actor policy function neural network is used to optimize the policy parameters of the control strategy; the critic value function neural network is used to evaluate the pros and cons of the current policy parameters optimized by the actor policy function neural network.

[0073] according to Figure 4 As shown in the structure, an actor strategy function neural network is constructed, and the actor strategy function neural network includes 1 input layer, 4 fully connected layers, 4 ReLU activation layers, 1 tanhLayer and 1 scalingLayer, and the actor strategy function neural network takes state s as input and action a as output.

[0074] according to Figure 4 As shown in the structure, a critic value function neural network is constructed, and the critic value function neural network includes 1 input layer, 4 fully connected layers, 4 ReLU activation layers, 1 addition layer and 1 output layer, and the critic value function neural network takes state s and action a as input and state-action value q(s, a) as output.

[0075] (42) Setting constraints and reward functions

[0076] Constraints:

[0077] In order to avoid blind exploration of the intelligent agent, a constraint range is imposed on the output action a of the actor strategy function neural network to ensure that the output action of the overall network structure is within the specified range, thereby reducing the training time.

[0078] Define is_done as the training completion / termination symbol. During the interaction between the actor strategy function neural network and the critic value function neural network, if the returned state is not within the expected range, the training process is terminated and restarted, otherwise the training continues.

[0079] From the above, the constraint condition can be defined as:

[0080]

[0081] In the formula, S max and S min Represent the maximum and minimum values ​​of state s respectively.

[0082] Reward Function

[0083] Whenever a vehicle collides, virtual reset technology is used, that is, the training does not end directly, but instead returns to the previous safe state to continue learning. This not only reduces the proportion of invalid training, but also effectively improves the training and learning efficiency.

[0084] ① The vehicle needs to maintain the desired speed, and the error between the actual speed and the desired speed is defined as v err, when the speed error is zero, the vehicle maintains the desired speed, and an additional reward value O is given, otherwise the reward value is zero. The reward function with the desired speed of the vehicle as a parameter is expressed as:

[0085]

[0086] ② Speed ​​control is not only related to the driving safety of the fleet, but also directly affects fuel economy. fuel Represents the instantaneous fuel consumption of the vehicle. During the training process, when the vehicle speed meets the safety distance, the fuel consumption is reduced as much as possible. At this time, a positive reward N is given, otherwise the reward value is zero. The reward function with the instantaneous fuel consumption of the vehicle as a parameter is expressed as:

[0087]

[0088] ③Define d in represents the vehicle-to-vehicle distance within the convoy, d des represents the desired internal vehicle spacing, and the vehicle spacing error is d err =d in -d des , when d err When is zero, the desired vehicle spacing is maintained within the convoy, and an additional reward value K is given. Otherwise, the reward value is zero. The reward function with the vehicle-to-vehicle distance within the convoy as a parameter is expressed as:

[0089]

[0090] ④Define d pre Indicates the distance between the convoy and the external vehicle in front, d safe Indicates the safe distance between the convoy and the external vehicle in front. When both meet the condition d pre ≥d safe When d pre <d safe When , the team is considered to have a potential collision risk and will receive a negative reward L as a penalty. The reward function with the distance between the team and the external vehicle in front as a parameter is expressed as:

[0091]

[0092] ⑤Define e y Indicates the lateral position deviation of the vehicle. When e y When is zero, the desired vehicle spacing is maintained within the convoy, and an additional reward value M is given. Otherwise, the reward value is zero. The reward function with the lateral position deviation of the vehicle as a parameter is expressed as:

[0093]

[0094] In summary, the present invention can specifically provide a composite reward function, which can be expressed as:

[0095] R=-(0.1d err +0.5(d pre -d safe )+0.1e y +0.1e fuel +0.1v err )+K+L+M+N+O.

[0096] S5. Use a deep reinforcement learning algorithm to solve the optimal control strategy of the local predictive controller, and use the local predictive controller to apply the optimal control strategy to the target following vehicle.

[0097] according to Figure 2 and Figure 3 As shown, the method of solving the optimal control strategy of the local predictive controller using a deep reinforcement learning algorithm includes:

[0098] Initialize the actor strategy function neural network weight θ and the critic value function neural network weight ω;

[0099] The actor strategy function neural network obtains action a according to the current state s of the target following vehicle, and applies action a to the target following vehicle to obtain a new state s and an immediate reward r;

[0100] The actor strategy function neural network obtains a new action a according to the new state s;

[0101] The critic value function neural network evaluates and scores actions a and a to obtain q(s, a) and q(s, a), and then calculates the predicted value q(s, a) and expected value y of the critic value function neural network according to the Bellman equation. t The error between TDerror: TDerror = y t -q(s,a),y t =r+γq(s,a);

[0102] The gradient descent method is used to minimize and iteratively update the weight θ and the weight ω to obtain the optimal control strategy And L(θ)=q(s,a),

[0103] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vehicle formation planning and control method based on deep reinforcement learning algorithm, characterized in that: The steps include: S1. Constructing a fuel consumption model that takes into account the influence of the road slope of the vehicle, and calculating the instantaneous fuel consumption and expected speed of the vehicle according to the fuel consumption model; S2. Combine the 3-DOF vehicle dynamics model with the lane keeping model to establish a vehicle platoon model; S3. Design a local predictive controller for each following vehicle based on the vehicle platoon model; S4. Construct constraints in the iterative process of the deep reinforcement learning algorithm and a reward function with the vehicle's instantaneous fuel consumption and expected speed as parameters; S5. Use a deep reinforcement learning algorithm to solve the optimal control strategy of the local predictive controller, and use the local predictive controller to apply the optimal control strategy to the target following vehicle.

2. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 1 is characterized in that: The fuel consumption model is expressed as: In the formula, E fuel and P represent the fuel consumption rate and mechanical power of the vehicle respectively, α represents the ratio of fuel to air, γ is the calorific value of the fuel, β, σ, ε and τ are the friction coefficient, engine speed, exhaust volume and conversion coefficient of the engine respectively, and η f and η c are the efficiency parameters of the engine and transmission system respectively.

3. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 2 is characterized in that: The 3-DOF vehicle dynamics model is expressed as: In the formula, are longitudinal velocity, lateral velocity and yaw rate respectively, F i x is the longitudinal force, F i yf ,F i yr are the lateral forces of the front and rear wheels, m i is the vehicle mass, is the moment of inertia of the vehicle around the z-axis, δ i is the front wheel turning angle, a i ,b i are the distances from the center of mass to the front and rear axles, respectively.

4. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 3 is characterized in that: The lane keeping model is expressed as: In the formula, is the expected heading angular velocity, L is the preview distance, is the longitudinal inter-vehicle distance error, is the lateral position error between the vehicle and the lane line, is the heading angle error between the vehicle heading angle and the road tangent.

5. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 4 is characterized in that: The iterative process of the deep reinforcement learning algorithm includes alternating convergence iterations of the actor strategy function neural network and the critic value function neural network.

6. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 5 is characterized in that: The actor strategy function neural network is used to optimize the strategy parameters of the control strategy; The critic value function neural network is used to evaluate the pros and cons of the current policy parameters optimized by the actor policy function neural network.

7. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 6 is characterized in that: The actor strategy function neural network includes 1 input layer, 4 fully connected layers, 4 ReLU activation layers, 1 tanhLayer and 1 scalingLayer, and the actor strategy function neural network takes state s as input and action a as output; The critic value function neural network includes 1 input layer, 4 fully connected layers, 4 ReLU activation layers, 1 addition layer and 1 output layer, and the critic value function neural network takes state s and action a as input and state-action value q((s,a)) as output.

8. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 7 is characterized in that: The constraint condition in the iterative process of the deep reinforcement learning algorithm is to impose a constraint range on the output action a of the actor strategy function neural network, and the constraint range is expressed as: In the formula, S max and S min They represent the maximum and minimum values ​​of state s respectively.

9. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 8 is characterized in that: During the iteration process of the deep reinforcement learning algorithm: The reward function with the desired speed of the vehicle as a parameter is expressed as: In the formula, v err Indicates the error between the actual speed of the vehicle and the expected speed; The reward function with the vehicle's instantaneous fuel consumption as a parameter is expressed as: In the formula, e fuel Indicates the instantaneous fuel consumption of the vehicle.

10. The vehicle formation planning and control method based on deep reinforcement learning algorithm according to claim 9 is characterized in that: The method of using a deep reinforcement learning algorithm to solve the optimal control strategy of the local predictive controller includes: Initialize the actor strategy function neural network weight θ and the critic value function neural network weight ω; The actor strategy function neural network obtains action a according to the current state s of the target following vehicle, and applies action a to the target following vehicle to obtain a new state s and an immediate reward r; The actor strategy function neural network obtains a new action a according to the new state s; The critic value function neural network evaluates and scores actions a and a to obtain q((s,a)) and q(s,a), and then calculates the predicted value q(s,a) and expected value y of the critic value function neural network according to the Bellman equation. t The error between TDerror: TDerror = y t -q(s,a),y t =r+γq(s,a); The gradient descent method is used to minimize and iteratively update the weight θ and the weight ω to obtain the optimal control strategy And L(θ)=q(s,a),

Images