Near-end strategy optimization for economical vehicle following control considering the impact of road gradient

Abstract

This invention discloses a near-end strategy optimization-based economic vehicle following control method considering the influence of road slope. The method includes: constructing a near-end strategy optimization network model; designing a reward function based on road slope characteristics, following distance, and vehicle speeds; training the near-end strategy optimization network model constructed in step one using a Carla / Python co-simulation platform. During training, the controlled vehicle continuously performs actions based on the current PPO (Progressive Point of Action) distribution in the simulation environment, interacting with the environment, and evaluating the actions based on the reward function designed in step two; updating and optimizing the parameters of the network model until a preset number of training steps are reached, at which point the trained economic following driving strategy considering the influence of slope is output. This invention designs a reward function that integrates the VT-Micro instantaneous energy consumption estimation model and designs a state space that incorporates road information such as slope, achieving economic following driving of the controlled vehicle while meeting the requirements of variable safety distance.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology and relates to a method for controlling the following of autonomous vehicles, specifically a near-end strategy optimization and economical following control method that takes into account the influence of road gradient. Background Technology

[0002] Following vehicles is a frequent driving scenario and a hot research topic in the field of autonomous driving. It has outstanding capabilities in improving driving comfort, alleviating traffic congestion, and reducing traffic accident rates.

[0003] Currently, numerous model-based traditional control algorithms have demonstrated excellent performance in vehicle following problems. Firstly, the proportional-integral-derivative (PID) controller can switch between different driving strategies based on the speed difference, relative distance, and the relationship between the vehicle's instantaneous speed and the relative distance between the two vehicles. PID-based following controllers not only outperform manual driving but also exhibit good instantaneous response performance. Building upon this, optimization algorithms such as particle swarm optimization are used to optimize the parameters of the PID controller for different objectives, resulting in varying degrees of performance improvement under different traffic conditions. Model predictive control theory has also been widely applied in this field. It can design control strategies for multiple objectives such as driving safety, ride comfort, and fuel economy. Using rolling optimization, it ensures the controlled vehicle meets multiple constraints while also considering personalized driving needs such as driver style to achieve vehicle following.

[0004] Deep reinforcement learning algorithms, as one of the representative algorithms of artificial intelligence, have achieved good performance in various application fields, including autonomous driving. In the application of autonomous driving following vehicles using deep reinforcement learning, a Markov decision process model was established based on the reinforcement Q-learning algorithm to address the vehicle following problem. An independent decision-making driving strategy was designed according to the characteristics of the vehicle following problem. The model-free optimal driving strategy based on Actor-Critic can learn online for different driving situations, which can reduce fuel consumption during vehicle operation, maintain a safe following distance, and has robustness.

[0005] While the aforementioned model-based traditional vehicle following control algorithms have achieved some success and application in industry, they rely on the establishment of the controlled model and the detailed development of the driving environment, resulting in poor adaptability to complex and ever-changing traffic conditions. Reinforcement learning, on the other hand, allows agents to learn in complex environments, ultimately leading to better strategies. Reinforcement learning has also made some progress in the current problem of autonomous vehicle following, but most of it addresses the case of zero gradient, i.e., following without considering road gradient. However, road gradient has a significant impact on fuel economy. Some current work explores following problems that consider road gradient using model-based reinforcement learning, but solving the error problem of model-based reinforcement learning remains a challenge. Summary of the Invention

[0006] To address the shortcomings of existing technologies, such as the poor adaptability of traditional model-based planning and control methods to complex traffic conditions, and the potential for suboptimal solutions due to hierarchical control strategies for different control objectives, this invention provides a proximal strategy optimization-based economic vehicle following control method that considers the impact of road gradient. For roads with gradients, this method utilizes a proximal strategy optimization algorithm based on model-free reinforcement learning, considers the influence of road gradient, designs a reward function that integrates the VT-Micro instantaneous energy consumption estimation model, and designs a state space that incorporates road information such as gradient. This achieves economical following of the controlled vehicle while meeting the requirement of variable safety distance.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A near-end strategy optimization method for economical following control that considers the influence of road slope includes the following steps:

[0009] Step 1: Build a near-end policy optimization network model;

[0010] Step 2: Design a reward function based on information such as road slope characteristics, following distance, and speed of both vehicles;

[0011] Step 3: Use the Carla / Python co-simulation platform to train the near-end policy optimization network model built in Step 1. During training, the controlled vehicle continuously makes actions in the simulation environment according to the current PPO policy distribution and interacts with the environment. The actions are evaluated according to the reward function designed in Step 2.

[0012] Step 4: Update and optimize the parameters of the network model until the preset number of training steps is reached, then output the trained economical following driving strategy that takes into account the impact of slope.

[0013] Furthermore, step two includes:

[0014] S21. Design the speed reward function r v ;

[0015] S22. Design the distance reward function r h ;

[0016] S23. Design the energy reward function r o ;

[0017] S24. Design the collision reward function r c With the endpoint reward function r end ;

[0018] S25. The total reward function r is the sum of the above reward functions, that is:

[0019] r = r v +r h +r o +r c +r end .

[0020] Preferably, step S21 includes:

[0021] The variable safety distance is set as the following distance that does not trigger the automatic emergency braking system. The calculation formula is as follows:

[0022]

[0023] In the above formula, v represents the instantaneous speed of the controlled vehicle; v p d represents the instantaneous speed of the vehicle in front; safe Represents a safe distance; T react This represents the allowable reaction time of the AEB system; a h,bmax and a p,bmax These represent the maximum braking deceleration of the main vehicle and the vehicle in front, respectively. "max" indicates selecting the larger of the two parameters in parentheses.

[0024] To guide the controlled vehicle to a speed close to that of the vehicle in front, a speed reward r is set. v :

[0025]

[0026] Preferably, step S22 includes:

[0027] To enable the lead vehicle to follow the vehicle in front, the headway (THW) is fused with the reward function. The calculation method for the headway and the distance reward function r are described below. h As shown below:

[0028]

[0029] r h =10|θ|e -THW

[0030] In the above formula, v represents the instantaneous speed of the controlled vehicle, d represents the real-time distance between the two vehicles, and θ represents the slope of the road.

[0031] Preferably, step S23 includes:

[0032] To enable the main vehicle to follow the driver's path economically, the VT-Micro fuel consumption estimation model is integrated into the reward function. The calculation formula for the VT-Micro fuel consumption estimation model is as follows:

[0033]

[0034] In the above formula, o represents the instantaneous energy consumption of the controlled vehicle during operation, and a n (t) and v n (t) represent the instantaneous acceleration and velocity of the vehicle during travel, respectively, and K ij (a n (t) is a coefficient that is positively or negatively correlated with the instantaneous acceleration of the vehicle;

[0035] Energy reward function r o As shown below:

[0036] r o =-500o.

[0037] Preferably, step S24 includes:

[0038] To avoid collisions between the controlled vehicle and the vehicle in front, and to reward the controlled vehicle for reaching a preset following endpoint, a collision reward function r is designed. c With the endpoint reward function r end As shown below:

[0039] r c =-750

[0040] r end =450.

[0041] Furthermore, step three includes:

[0042] Building the training environment: Designing a training environment based on the PPO economic car-following algorithm using the Carla / Python co-simulation platform;

[0043] Design motion space: Design discrete motion space, i.e. [-1,-0.5,0.0,0.2,0.4,0.6,0.8,1.0]; positive numbers in discrete motion space represent the opening degree of accelerator pedal, and negative numbers represent the opening degree of brake pedal;

[0044] Design state space: the instantaneous speed v of the controlled vehicle, the instantaneous speed v of the vehicle in front. p The real-time distance d between the two vehicles, the height h of the controlled vehicle's position, the road slope θ, and the indicator f indicating whether the controlled vehicle is traveling uphill or downhill are set as the state space, i.e., [v, v]. p [,d,h,θ,f]; If driving uphill, set f=0, and driving downhill, set f=1.

[0045] Furthermore, step four includes:

[0046] The proximal policy optimization network includes an Actor network and an Actor' network; wherein, the Actor network is the policy network, and the Actor' network is used to store the network parameters before the update;

[0047] When an Actor network interacts with its environment, the state s, action a, and reward r from start to finish time T are considered as a trajectory, i.e., a trajectory τ = {s1, a1, r1, s2, a2, r2, ..., s T ,a T ,r T};

[0048] When the parameters of the Actor network are θ, the probability of a trajectory being taken is set to p. θ (τ);

[0049] Therefore, the expected reward under parameter θ The calculation formula is:

[0050]

[0051] In the formula, N represents the total number of samples, and R(τ) (n) () is the sum of rewards for the nth trajectory in the sampling;

[0052] The near-end policy optimization algorithm uses importance sampling for updates, as shown in the following equation, where the expected value of the sampling trajectory τ under the policy network parameters θ is given by...

[0053]

[0054] The objective function for policy network updates is shown below:

[0055]

[0056] In the above formula, min represents taking the smaller term within the parentheses; A θ (s t ,a t) is the dominant function; clip means restricting the first parameter in the parentheses to between the second and third parameters; p θ (a t |s t ) and p θ' (a t |s t ) represent the probability distribution of state s taking action a at time t before and after parameter θ update; ε is a hyperparameter.

[0057] The present invention has the following beneficial effects:

[0058] The proximal policy optimization algorithm used in this invention belongs to the model-free reinforcement learning method, and takes into account the influence of road slope in the field of economical following driving application.

[0059] This invention designs a state space and action space that are adapted to the driving conditions of following another vehicle on a slope.

[0060] This invention designs a unique reward function that satisfies the following effect on slopes while also taking into account economical driving. Attached Figure Description

[0061] Figure 1 This is an overall flowchart of the near-end strategy optimization and economic vehicle following control method considering the influence of road slope as described in the embodiments of the present invention;

[0062] Figure 2 This is a simplified schematic diagram of the training environment described in an embodiment of the present invention;

[0063] Figure 3 This is a simplified diagram illustrating the near-end policy optimization network model and update process as described in an embodiment of the present invention.

[0064] Figure 4 This is a graph showing the reward function results as described in an embodiment of the present invention;

[0065] Figure 5 This is a schematic diagram illustrating the following effect of the controlled vehicle on an uphill slope according to an embodiment of the present invention.

[0066] Figure 5 In the image, (a) shows the vehicle speed following effect; (b) shows the vehicle distance following effect.

[0067] Figure 6 This is a schematic diagram illustrating the following effect of the controlled vehicle on a downhill slope according to an embodiment of the present invention.

[0068] Figure 6 In the image, (a) shows the vehicle speed following effect; (b) shows the vehicle distance following effect. Detailed Implementation

[0069] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments:

[0070] like Figure 1 As shown, a near-end strategy optimization method for economical following control considering the influence of road slope includes the following steps:

[0071] Step 1: Build the network model required for the Proximal Policy Optimization (PPO) algorithm.

[0072] Step 2: Design a reward function based on information such as road slope characteristics, following distance, and vehicle speeds.

[0073] In order to guide the controlled vehicle to achieve economical driving on the slope while meeting the constraints of variable safety distance, this embodiment designs a series of reward functions.

[0074] S21. Design the speed reward function r v

[0075] The variable safety distance is set as the following distance without triggering the automatic emergency braking (AEB) system, and its calculation formula is:

[0076]

[0077] In the above formula, v represents the instantaneous speed of the controlled vehicle, v p d represents the instantaneous speed of the vehicle in front. safe T represents a safe distance. react This represents the allowable response time of the AEB system, a h,bmax and a p,bmax These represent the maximum braking deceleration of the main vehicle and the vehicle in front, respectively. "max" indicates selecting the larger of the two parameters in parentheses.

[0078] To guide the controlled vehicle to a speed close to that of the vehicle in front, a speed reward r is set. v :

[0079]

[0080] S22. Design the distance reward function r h

[0081] To enable the lead vehicle to follow the vehicle in front, the headway (THW) is fused with the reward function. The calculation method for the headway and the distance reward function r are described below. h As shown below:

[0082]

[0083] r h =10|θ|e -THW (4)

[0084] In the above formula, v represents the instantaneous speed of the controlled vehicle, d represents the real-time distance between the two vehicles, and θ represents the slope of the road.

[0085] S23. Design the energy reward function r o

[0086] To enable the main vehicle to follow the driver's path economically, the VT-Micro fuel consumption estimation model is integrated into the reward function. The calculation formula for the VT-Micro fuel consumption estimation model is as follows:

[0087]

[0088] In the above formula, o represents the instantaneous energy consumption of the controlled vehicle during operation, and a n (t) and v n (t) represent the instantaneous acceleration and velocity of the vehicle during travel, respectively, and K ij (a n (t) is a coefficient that is positively or negatively correlated with the instantaneous acceleration of the vehicle.

[0089] Energy reward function r o As shown below:

[0090] r o =-500o (6)

[0091] S24. Design the collision reward function r c With the endpoint reward function r end

[0092] To avoid collisions between the controlled vehicle and the vehicle in front, and to reward the controlled vehicle for reaching the preset following endpoint, a collision reward function r is designed. c With the endpoint reward function r end As shown below:

[0093] r c =-750 (7)

[0094] r end =450 (8)

[0095] S25. The total reward function r is the sum of the above reward functions, that is:

[0096] r = r v +r h +r o +r c +r end (9)

[0097] Step 3: Use the Carla / Python co-simulation platform to train the near-end policy optimization network model built in Step 1. During training, the controlled vehicle continuously makes actions in the simulation environment according to the current PPO policy distribution and interacts with the environment. The actions are evaluated according to the reward function designed in Step 2.

[0098] Before training the proximal policy optimization network model built in step one using the Carla / Python co-simulation platform, the training environment and the action space and state space are designed as follows:

[0099] Setting up the training environment:

[0100] Since this invention is applicable to following a vehicle on a sloped road, a training environment based on the PPO (Progressive Vehicle Opportunity) following algorithm is designed using the Carla / Python co-simulation platform. This embodiment designs training environments for both uphill and downhill sections to achieve economical following, as follows: Figure 2 As shown.

[0101] Designing the action space:

[0102] To ensure that the controlled vehicle meets the real-time variable safety distance constraint on both uphill and downhill sections, and achieves economical following, a discrete action space was designed, namely [-1, -0.5, 0.0, 0.2, 0.4, 0.6, 0.8, 1.0]. Positive numbers in the discrete action space represent the accelerator pedal opening, and negative numbers represent the brake pedal opening.

[0103] Design state space:

[0104] In order to better perceive the dynamic information of the vehicle in front and the road characteristics, the instantaneous speed v of the controlled vehicle and the instantaneous speed v of the vehicle in front are... p The real-time distance d between the two vehicles, the height h of the controlled vehicle's position, the road slope θ, and the indicator f indicating whether the controlled vehicle is traveling uphill or downhill (f=0 for uphill travel and f=1 for downhill travel) are set as the state space, i.e., [v, v p ,d,h,θ,f).

[0105] Step 4: Update and optimize the parameters of the network model until the preset number of training steps is reached, then output the trained economical following driving strategy that takes into account the impact of slope.

[0106] like Figure 3As shown, the Actor-Critic network required to build the near-end policy optimization algorithm in this embodiment includes an Actor network and an Actor' network; wherein, the Actor network is used to interact with and update the simulation environment, that is, the Actor network is the policy network; the Actor' network is used to save the network parameters before the update;

[0107] When the Actor network interacts with the environment, the state s, action a, and reward r from the start to the final time T are considered as a trajectory, i.e., a trajectory τ = {s1, a1, r1, s2, a2, r2, ..., s T ,a T ,r T};

[0108] When the parameters of the Actor network are θ, the probability of a trajectory being taken is set to p. θ (τ); therefore, the expected reward under parameter θ. The calculation formula is:

[0109]

[0110] In the formula, N represents the total number of samples. R (τ (n) ) is the sum of rewards for the nth trajectory in the sampling.

[0111] After the policy network parameters are updated, resampling with the updated parameters θ' is required, rendering previously collected data invalid. To improve data utilization, the near-end policy optimization algorithm utilizes importance sampling, as shown in the following equation, where the expected value of the sampling trajectory τ under the policy network parameters θ is...

[0112]

[0113] Therefore, the objective function for policy network updates is as follows:

[0114]

[0115] In the above formula, min represents taking the smaller term within the parentheses; A θ (s t ,a t ) is the dominant function; clip means restricting the first parameter in the parentheses to between the second and third parameters; p θ (a t |s t ) and p θ' (a t |s t) represent the probability distribution of state s taking action a at time t before and after parameter θ update; ε is a hyperparameter, which takes the value of 0.2 in this embodiment.

[0116] Experimental and simulation results:

[0117] To demonstrate the effectiveness and stability of this invention, the experiment will be repeated four times, and the relationship between the average reward and the number of training steps will be plotted using the mean reward and its standard deviation, as shown in the figure below. Figure 4 As shown.

[0118] To demonstrate the following performance and economical driving capabilities of this invention, a speed-position dual PID algorithm was compared with the method proposed in this invention. The results are as follows: Figure 5 , Figure 6 As shown.

[0119] In experiments, although the speed difference between the present invention and the lead vehicle was greater than in the comparative experiment, the speed difference was no more than 2 m / s in both driving conditions, and the distance between the two vehicles was significantly smaller. Furthermore, the variable safety distance constraint was met, resulting in better following performance. In uphill and downhill driving conditions, the present invention achieved average instantaneous energy consumption of 0.001809 L / s and 0.001856 L / s, respectively. Compared to the comparative experiment's 0.001907 L / s and 0.001912 L / s, the average energy consumption was reduced by 5.147% and 2.939%, respectively, demonstrating economical driving performance.

Claims

1. A near-end strategy optimization and economical following control method considering the influence of road gradient, characterized in that, Includes the following steps: Step 1: Build a near-end policy optimization network model; Step 2: Design a reward function based on road gradient characteristics, following distance, and vehicle speeds; Step 2 includes: S21. Design the speed reward function ; S22. Design the distance reward function ,include: To enable the lead vehicle to follow the vehicle in front, the headway (THW) is integrated with the reward function. The calculation method for the headway and the distance reward function are as follows: As shown below: ； ； In the above formula, Represents the instantaneous speed of the controlled vehicle. This represents the real-time distance between the two vehicles. Represents the slope of the road; S23. Design the energy reward function ; S24. Design the collision reward function With the endpoint reward function ; S25. Total Reward Function The sum of the above reward functions is: ； Step 3: Use the Carla / Python co-simulation platform to train the near-end policy optimization network model built in Step 1. During training, the controlled vehicle continuously makes actions in the simulation environment according to the current PPO policy distribution and interacts with the environment. The actions are evaluated according to the reward function designed in Step 2. Step 4: Update and optimize the parameters of the network model until the preset number of training steps is reached, then output the trained economical following driving strategy that takes into account the impact of slope.

2. The near-end strategy optimization and economic vehicle following control method considering the influence of road slope as described in claim 1, characterized in that, Step S21 includes: The variable safety distance is set as the following distance that does not trigger the automatic emergency braking system. The calculation formula is as follows: ； In the above formula, Represents the instantaneous speed of the controlled vehicle; Represents the instantaneous speed of the vehicle in front; Represents a safe distance; This represents the allowable response time for the AEB system; and These represent the maximum braking deceleration of the main vehicle and the preceding vehicle, respectively. This indicates that the larger of the two parameters in parentheses is selected. To guide the controlled vehicle to maintain a speed close to that of the vehicle in front, a speed reward is set. : 。 3. The near-end strategy optimization and economic vehicle following control method considering the influence of road slope as described in claim 1, characterized in that, Step S23 includes: To enable the main vehicle to follow the driver's path economically, the VT-Micro fuel consumption estimation model is integrated into the reward function. The calculation formula for the VT-Micro fuel consumption estimation model is as follows: ； In the above formula, This represents the instantaneous energy consumption of the controlled vehicle while it is in motion. and These represent the instantaneous acceleration and velocity of the vehicle, respectively. It is a coefficient that has a positive or negative correlation with the instantaneous acceleration of the vehicle; Energy reward function As shown below: 。 4. The near-end strategy optimization and economic vehicle following control method considering the influence of road slope as described in claim 1, characterized in that, Step S24 includes: To prevent the controlled vehicle from colliding with the vehicle in front and to reward the controlled vehicle for reaching a preset following endpoint, a collision reward function is designed. With the endpoint reward function As shown below: ； 。 5. The near-end strategy optimization and economic vehicle following control method considering the influence of road slope as described in claim 1, characterized in that, Step three includes: Building the training environment: Designing a training environment based on the PPO economic car-following algorithm using the Carla / Python co-simulation platform; Design the motion space: Design the discrete motion space, i.e. In the discrete motion space, positive numbers represent the opening degree of the accelerator pedal, and negative numbers represent the opening degree of the brake pedal. Design state space: the instantaneous speed of the controlled vehicle The instantaneous speed of the car in front Real-time distance between the two vehicles Height of the location of the controlled vehicle Road gradient And signs indicating whether the vehicle being controlled is traveling uphill or downhill. Set as a state space, i.e. If driving uphill, set... Downhill driving settings .

6. The near-end strategy optimization and economical following control method considering the influence of road slope as described in claim 1, characterized in that, Step four includes: The proximal policy optimization network includes an Actor network and a... Network; where the Actor network is a policy network. The network is used to save network parameters before the update; During the interaction between the Actor network and its environment, from start to finish time... status ,action and rewards Considered as a trajectory, that is, a trajectory ; When the parameters of the Actor network are When, the probability of a trajectory being taken is set to ; Therefore, parameters Expected Rewards The calculation formula is: ； In the formula, Represents the total number of samples. It is the first in the sampling The sum of rewards for each trajectory; The near-end policy optimization algorithm updates the policy using importance sampling, as shown in the following equation, where the policy network parameters are set. The sampling trajectory below The expectation is : ； The objective function for policy network updates is shown below: ； In the above formula, min represents taking the smaller term within the parentheses; It is the dominant function; This means that the first parameter in the parentheses following it is restricted to between the second and third parameters; and They represent parameters respectively The state at time t before and after the update Take action The probability distribution; It's a hyperparameter.

Images