Construction method of deep reinforcement learning vehicle following model fusing driving style

Abstract

The application relates to a kind of construction methods of fusion driving style deep reinforcement learning vehicle following model, based on NGSIM project high-precision traffic flow data set, set corresponding rules to original data for screening, obtain qualified following pair data, the following pair data is divided into training data and verification data.Based on the analysis of the driving characteristics of the driver based on the training data, a driving style recognition model is built based on the support vector machine algorithm. Based on the analysis results of the following behavior of different characteristic drivers, the reinforcement learning reward function is designed by fusing the following distance of different driving styles to build a following model based on the deep deterministic policy gradient algorithm. The deep reinforcement learning following model of different driving styles is obtained by training the model in the built training environment. The simulation results based on the verification data show that the built following model can reflect different driving characteristics under the premise of ensuring safety and comfort.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent driving auxiliary systems, and relates to a construction method of a deep reinforcement learning vehicle following model fusing driving styles. BACKGROUND

[0002] Vehicle following driving is a common phenomenon in road traffic, especially in today's urban traffic where traffic jams occur frequently. Research on vehicle following behavior is an important research content for single vehicle following control and multi-vehicle platoon driving control in the field of intelligent driving.

[0003] As one of the most basic researches in traffic flow theory, the vehicle following model is indispensable for the research of single intelligent vehicle and intelligent network. At the same time, people no longer simply satisfy the following changes in the speed of the rear vehicle when the intelligent driving vehicle follows, but more require the vehicle to consider the driving style to realize the personification of the following to meet the needs of different drivers and passengers under the premise of ensuring safety and comfort. The patent is proposed in this background. SUMMARY

[0004] The application discloses a construction method of a deep reinforcement learning vehicle following model fusing driving styles. The application integrates driving styles into the research of the following model, builds a personalized following model considering the driving style of drivers, and makes the built following model meet the personalized needs of different drivers and passengers for intelligent driving, thereby improving the acceptance of the intelligent driving system.

[0005] The technical scheme adopted by the application is as follows:

[0006] The construction method of the deep reinforcement learning vehicle following model fusing driving styles comprises the following steps:

[0007] Step S1: Based on the high-precision traffic flow data set of the NGSIM (Next Generation Simulation) project, corresponding rules are set to process and screen the original data, and the following pair data meeting the conditions is obtained. 70% of the following pair data in the obtained data set is randomly selected as training data, and the remaining following pair data is used as test data.

[0008] Step S2: The relative speed, headway and absolute value of the following acceleration are used to represent the driving characteristics of the driver, the K-means clustering algorithm is used for driving style clustering, and the driving style is divided into two types of aggressive and conservative. Statistical analysis is performed on the following data of different styles to obtain the minimum following distance value under different driving styles. The driving style of the aggressive type is 4.09 m, and the driving style of the conservative type is 5.63 m.

[0009] Step S3: A driving style recognition model is built based on a support vector machine (SVM) algorithm;

[0010] Step S4: Based on the analysis results of the car-following behaviors of different characteristic drivers, a car-following model based on a deep deterministic policy gradient (DDPG) algorithm is built by designing a reinforcement learning reward function for the following distances of different driving styles;

[0011] Step S5: A simulation environment is built, and parameters are set. In the built simulation environment, the DDPG car-following model is trained based on the training data obtained in step 1. In the training process, each episode randomly selects a car-following pair in the training data to train the model. When a collision occurs or the car-following event ends, the current episode ends, and the next episode starts. Finally, the deep reinforcement learning car-following models of different driving styles are obtained.

[0012] Step S6: The model is tested based on test data.

[0013] Further, the processing operation of the NGSIM original data in step S1 is as follows: the data of the time period 04:00 p.m.-04:15 p.m. in the I-80 section is selected as the original data, the Savitzky-Golay filter algorithm is used to smooth the original data to remove noise, the window length of the Savitzky-Golay filter is selected as 21, and the polynomial order is selected as 3 to smooth the original data.

[0014] Further, the rules of the screening operation of the NGSIM original data in step S1 are as follows:

[0015] S1.1, the vehicle type is selected as a small car, i.e. v_Class=2;

[0016] S1.2, the duration of the car-following pair is greater than 15s;

[0017] S1.3, in the car-following pair, the headway between the target vehicle and the host vehicle is less than 150ft (feet), i.e. Space_Headway<=150.

[0018] S1.4, when following, the speed of the host vehicle and the target vehicle is greater than 0;

[0019] S1.5, in the I-80 detection section, 1-5 are driving lanes, 6 is a gathering and distributing lane, 7 is an entrance ramp, and 8 is an exit ramp. Only the car-following pairs on the driving lanes with Lane_ID of 1-5 are selected.

[0020] Furthermore, the specific operation of step S4 is as follows:

[0021] Step S4.1: Design the state space and action space for reinforcement learning. The state space is selected as the following distance between the two vehicles, the speed difference between the two vehicles, and the speed of the master vehicle, i.e. The motion space selection is the main vehicle acceleration, i.e. .

[0022] Step S4.2: Design of the reinforcement learning reward function. Reward function:

[0023]

[0024] in, To simulate the distance between the two vehicles during the algorithm's execution, The expected following distance is given by the formula. Collision penalty factor, This is a collision detection flag; a collision occurs during the algorithm's execution. hour, ;otherwise .

[0025] The expected following distance is calculated as follows:

[0026]

[0027] in, for k The expected following distance of the main vehicle at any given time. To achieve the desired headway, This is the minimum safe distance.

[0028] Step S4.3: Design of the environment update strategy. Treat the motion of the main vehicle as the motion of a point mass, and calculate the position and velocity of the main vehicle based on the following point mass kinematic model.

[0029]

[0030]

[0031] in, It is 0.1s. for t The location of the main vehicle at all times. for t The speed of the main vehicle at any given time, for t The acceleration of the main vehicle at any given time, with the initial position of the main vehicle set to 0;

[0032] Step S4.4: design the neural network structure of the Actor and Critic. The input of the Actor neural network is relative distance, relative speed and host vehicle speed, and the output is host vehicle acceleration, because the tanh activation function is applied to the intermediate layer and output layer, and the output is multiplied by the coefficient δ to obtain the simulated acceleration value. The input of the Critic network is relative distance, relative speed, host vehicle speed and the output of the Actor network, and the output is the Q value function. The intermediate layer of the Actor and Critic networks has 30 neurons, except that the intermediate layer and output layer of the Actor network apply the tanh activation function, and the ReLU activation function is used between the other layers;

[0033] A car following model framework based on DDPG is obtained.

[0034] Further, the specific operation of step S6 is as follows:

[0035] Step S6.1, model test: in order to compare the differences between the car following models of different driving styles, different styles of car following models are tested on the test data, and the distribution of the following distance, the following relative speed and the following acceleration of different styles of car following models on the test data is compared;

[0036] Step S6.2, randomly select a following pair in the test set to compare different car following models: input the selected following pair into the driving style recognition model established in step S3 to obtain the driving style, and compare the NGSIM driving data and the car following models of different styles. It can be seen that the aggressive car following model built in the application is closer to the NGSIM driving data in the following distance, the following relative speed and the following acceleration than the conservative car following model.

[0037] The application selects the NGSIM project real-time high-precision traffic flow data set which is widely used at home and abroad, first adopts the Savitzky-Golay filtering algorithm to filter the original data; next, according to the set data extraction rule, the NGSIM original data is screened to obtain the qualified car following pair data; finally, the car following pair data is statistically analyzed from the following distance, the following speed, the relative speed, the following acceleration and the headway, which provides data support for the following research.

[0038] Firstly, the IDM car-following model is labeled based on genetic algorithm using the root mean square percentage error of simulated acceleration and actual acceleration as the goodness-of-fit function; the MPC car-following model is designed considering safety, comfort and following; the three-layer BP neural network car-following model is built and the model training is completed; the reinforcement learning reward function is designed combined with the expected following distance, and the DDPG car-following model training environment is built to complete the training of the DDPG car-following model.

[0039] Firstly, the relative speed, headway and following acceleration absolute value parameters are selected to represent the driving characteristics of the driver when following, and the K-means clustering method is used to realize the clustering analysis of driving style, and the driving style is divided into two types of aggressive and conservative. Secondly, the driving style recognition model is built based on SVM algorithm. Best, the car-following model of different driving styles is built based on DDPG, and the simulation analysis of the car-following model of different driving styles is carried out, and the results show that the car-following model of different styles built in this paper can show different driving styles when following, and the performance is closer to the driving style recognition result.

[0040] The beneficial technical effects of the present application are:

[0041] 1) The reward function of deep reinforcement learning is simple in design, has fewer parameters, and is easy to set parameters, and at the same time, different driving styles are integrated into the reward function.

[0042] 2) The car-following model of different driving styles involved in the present application can continue to drive the vehicle with the current driver's driving style when the driver is not holding the steering wheel, which can meet the personalized needs of different drivers for automatic driving, thereby improving the acceptance of people to automatic driving. DETAILED DESCRIPTION

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description, and obviously, other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings;

[0044] Figure 1 It is the construction method flow chart of the deep reinforcement learning vehicle car-following model of different driving styles of the present application;

[0045] Figure 2 It is the smooth effect diagram of the longitudinal position of the test vehicle in a period of time;

[0046] Figure 3 It is the smooth effect diagram of the vehicle speed of the test vehicle in a period of time;

[0047] Figure 4 is a smooth plot of driving acceleration of the test vehicle in a period of time;

[0048] Figure 5 is a driving style clustering result plot;

[0049] Figure 6 is an Actor neural network structure schematic diagram;

[0050] Figure 7 is a Critic neural network structure schematic diagram;

[0051] Figure 8 is a DDPG-based car following model framework;

[0052] Figure 9 is a training reward change curve of an aggressive driving style model;

[0053] Figure 10 is a training reward change curve of a conservative driving style model;

[0054] Figure 11 is a following distance distribution plot of car following models of different styles;

[0055] Figure 12 is a relative speed distribution plot of car following models of different styles;

[0056] Figure 13 is an acceleration distribution plot of car following models of different styles;

[0057] Figure 14 is a following headway change curve of car following models of different styles;

[0058] Figure 15 is a relative speed change curve of car following models of different styles;

[0059] Figure 16 is an acceleration change curve of car following models of different styles. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0061] The construction method of the deep reinforcement learning vehicle car following model fusing driving styles is specifically as shown in Figure 1 .

[0062] Step S1: Based on the high-precision traffic flow data set of the NGSIM (Next Generation Simulation) project, the corresponding rules are set to process and screen the original data, and the qualified car-following pair data is obtained. 70% of the car-following pair data in the obtained data set is randomly selected as training data, and the remaining car-following pair data is used as test data;

[0063] Step S2: The driving characteristics of drivers are represented by relative speed, time headway and absolute value of following acceleration. K-means clustering algorithm is used to cluster driving styles into two categories: aggressive and conservative. Statistical analysis is performed on the car-following data of different styles to obtain the minimum following distance value under different driving styles. The aggressive driving style is 4.09 m, and the conservative driving style is 5.63 m;

[0064] Step S3: A driving style recognition model is built based on the support vector machine (SVM) algorithm;

[0065] Step S4: Based on the analysis results of the car-following behavior of drivers with different characteristics, the following distance of different driving styles is fused to design a reinforcement learning reward function to build a car-following model based on the deep deterministic policy gradient (DDPG) algorithm;

[0066] Step S5: Build a simulation environment and set parameters. In the simulation environment built, the DDPG car-following model is trained based on the training data obtained in step 1. In the training process, the car-following pair in the training data is randomly selected to train the model. When a collision occurs or the car-following event ends, the current scene ends and the next scene begins. The deep reinforcement learning car-following model of different driving styles is obtained.

[0067] Step S6: Test the model based on the test data.

[0068] In this embodiment, the detailed method steps are as follows:

[0069] Step S1.1: Select the data of time period 04:00 p.m.-04:15 p.m. in I-80 section for analysis. Since the original data in NGSIM is obtained by video processing, there are some abnormal values and measurement errors. In order to obtain more accurate vehicle driving data, the original data needs to be processed. In this embodiment, Savitzky-Golay filter algorithm is used to smooth the original data for the noise in the original data. The window length of Savitzky-Golay filter is selected as 21, and the polynomial order is selected as 3 to smooth the original data. Figure 2 、 3 , 4 respectively show the smooth effect diagram of the longitudinal position, vehicle speed and driving acceleration of the test vehicle in a period of time.

[0070] Step S1.2: The NGSIM data set contains the driving information of all vehicles in the measurement period. In order to obtain the required car-following data, the original data needs to be screened, and the following data screening rules are defined:

[0071] (1) The vehicle type is selected as small car, i.e. v_Class=2; The original data set contains the driving data of small cars, large cars and motorcycles. The car-following behavior of different types of vehicles may not be completely the same. Through observation, it is found that the data of small cars in the data set accounts for the majority, so small cars are selected as the research object to ensure that enough data can be screened.

[0072] (2) The duration of the car-following pair is greater than 15s; This rule is to ensure that the host vehicle is in a stable car-following state for a certain period of time, so as to obtain sufficient car-following data.

[0073] (3) In the car-following pair, the headway between the target vehicle and the host vehicle is less than 150ft (feet), i.e. Space_Headway<=150, which is about 45.72m; This rule ensures that the host vehicle is in a car-following state and will not be in a free driving state due to the large distance between the two vehicles.

[0074] (4) When car-following, the speed of the host vehicle and the target vehicle is greater than 0; This ensures that the vehicles are in driving state and will not stop in the middle of the road, affecting the analysis of car-following behavior.

[0075] (5) Only car-following pairs on the driving lane are selected; In I-80 detection section, 1-5 are driving lanes, 6 is a gathering and distributing lane, 7 is an entrance ramp, and 8 is an exit ramp. In order to ensure that the vehicles studied have similar driving behavior, only the car-following data of Lane_ID 1-5 is selected.

[0076] Randomly select 70% of the obtained data set as training data, and the remaining follow-up pair data as verification data.

[0077] Step S2.1: In this study, the relative speed, time headway and the absolute value of the following acceleration are used to represent the driving characteristics of the driver. The K-means clustering algorithm is used to cluster the driving styles into two categories: aggressive and conservative. The clustering results are shown in the following figure Figure 2 , and the driving style clustering results are shown in Table 1.

[0078]

[0079] Step S2.2: Statistical analysis is performed on the follow-up data of different styles to obtain the minimum following distance value under different driving styles. The aggressive driving style is 4.09m, and the conservative driving style is 5.63m.

[0080] Step S3: A driving style recognition model is built based on SVM, and the model is trained based on the training data.

[0081] Step S4.1: The specific operation of step S4 is as follows:

[0082] Step S4.1: Reinforcement learning state space and action space design, state space selection is the following distance between two vehicles, the speed difference between two vehicles and the host vehicle speed, that is ; the action space is selected as the host vehicle acceleration, that is .

[0083] Step S4.2: Design of reinforcement learning reward function, reward function:

[0084]

[0085] wherein, is the simulated distance between two vehicles during the algorithm running process, is the expected following distance shown in the formula, is the collision penalty factor, is the collision detection flag, when collision occurs during the algorithm running process, i.e. , ; otherwise .

[0086] The calculation of the expected following distance is as follows:

[0087]

[0088] wherein, is the expected following distance of the host vehicle at time k , is the expected time headway, is the minimum safety distance.

[0089] Step S4.3: Design of environment update strategy, take the host vehicle motion as the motion of a particle, calculate the host vehicle position and host vehicle speed according to the particle kinematics model as follows.

[0090]

[0091]

[0092] wherein, is 0.1s, is t is the host vehicle position at time t, is t is the host vehicle speed at time t, is t is the host vehicle acceleration at time t, the host vehicle position at the initial time is set to 0;

[0093] Step 4.4: Neural network structure: design the neural network structure of Actor and Critic as shown in Figures 6-7 .

[0094] The input of the Actor network is the relative distance, relative speed and host vehicle speed, and the output is the host vehicle acceleration, because the tanh activation function is applied to the intermediate layer and the output layer, the output is multiplied by the coefficient δ to obtain the simulated acceleration value.

[0095] The input of the Critic network is the relative distance, relative speed, host vehicle speed and the output of the Actor network, and the output is the Q value function. The intermediate layer of the Actor and Critic networks has 30 neurons, except that the tanh activation function is applied to the intermediate layer and the output layer of the Actor network, and the ReLU activation function is used between the layers of the other layers.

[0096] The DDPG-based car following model framework is obtained as shown in Figure 8 .

[0097] Step S5: Train the DDPG car following model based on the training data obtained in step S1, randomly select the car following pairs in the training data for each episode in the training process to train the model, when a collision occurs or the car following event ends, the current episode ends, and the training of the next episode starts. The parameter settings in the whole training process are shown in Table 2. The average reward change curves of the two styles of car following models in the training process are shown in Figure 9 and Figure 10 .

[0098]

[0099] Step S6.1: Model test: In order to compare the difference between different driving style car-following models, different driving style car-following models are tested on the test data. The distribution of following distance, relative speed and acceleration of different driving style car-following models on the test data are compared. It can be seen from Figures 11-13 that the following distance of aggressive driving style is smaller, with the mean value of 19.96 m, while the following distance of conservative driving style is 30.45 m; the relative speed of aggressive driving style is smaller, with the mean value of 0.13 m / s, while the relative speed of conservative driving style is 0.45 m / s; the absolute value of acceleration of aggressive driving style is larger, with the mean value of 0.43 m / s 2 , while the mean value of acceleration of conservative driving style is 0.40 m / s 2 .

[0100] Step S6.2: Randomly select a following pair in the test set to compare different car-following models. The selected following pair is input into the driving style recognition model established in step S3, and the driving style is determined to be aggressive, see Figures 14-16 . By comparing the NGSIM driving data and different driving style car-following models, it can be seen that the aggressive car-following model established in this paper is closer to the NGSIM driving data in the performance of following distance, relative speed and acceleration compared with the conservative car-following model.

Claims

1. A method for constructing a deep reinforcement learning vehicle car-following model that integrates driving styles, characterized by: The steps are as follows: Step S1: Based on the high-precision traffic flow dataset measured by the NGSIM (Next Generation Simulation) project, set corresponding rules to process and filter the original data to obtain car-following pair data that meet the conditions. Randomly select 70% of the car-following pair data in the processed dataset as training data, and use the remaining car-following pair data as test data. Step S2: The driving characteristics of the driver are characterized by relative speed, headway, and absolute value of following acceleration. The K-means clustering algorithm is used to cluster driving styles into two categories: aggressive and conservative. Statistical analysis is performed on the following data of different styles to obtain the minimum following distance value under different driving styles. The minimum following distance value is 4.09m for the aggressive driving style and 5.63m for the conservative driving style. Step S3: A driving style recognition model was built based on the Support Vector Machine (SVM) algorithm; Step S4: Based on the analysis results of car-following behavior of drivers with different characteristics, design a reinforcement learning reward function by integrating the following distance of different driving styles, and build a car-following model based on the Deep Deterministic Policy Gradient (DDPG) algorithm; Step S5: Set up the simulation environment and set the parameters. In the simulation environment, train the DDPG car-following model based on the training data obtained in Step 1. During the training process, randomly select the car-following pairs in the training data for each scene to train the model. When a collision occurs or the car-following event ends, the current scene ends and the training of the next scene begins again; thus obtaining deep reinforcement learning car-following models with different driving styles. Step S6: Test the model based on the test data; The specific operation of step S4 is as follows: Step S4.1: Design the state space and action space for reinforcement learning. The state space is selected as the following distance between the two vehicles, the speed difference between the two vehicles, and the speed of the master vehicle, i.e. ; The action space selection is the main vehicle acceleration, i.e. ; Step S4.2: Design of the reinforcement learning reward function. Reward function: in, To simulate the distance between the two vehicles during the algorithm's execution, The expected following distance is given by the formula. Collision penalty factor, This serves as a collision detection marker, which is used immediately upon a collision occurring during algorithm execution. ;otherwise ; The expected following distance is calculated as follows: in, for k The expected following distance of the main vehicle at any given time. To achieve the desired headway, Minimum safe distance; Step S4.3: Design of the environment update strategy, treating the motion of the main vehicle as the motion of a point mass, and calculating the position and velocity of the main vehicle based on the following point mass kinematic model; in, It is 0.1s. for t The location of the main vehicle at all times. for t The speed of the main vehicle at any given time, for t The acceleration of the main vehicle at any given time, with the initial position of the main vehicle set to 0; Step S4.4: Design the neural network structure for Actor and Critic: The input to the Actor neural network is the relative distance, relative speed, and vehicle speed, and the output is the vehicle acceleration. Because the intermediate layer and the output layer apply the tanh activation function, the output is multiplied by the coefficient δ to obtain the simulated acceleration value. The inputs to the Critic network are relative distance, relative speed, and vehicle speed, and the output of the Actor network is the Q-value function. Both the Actor and Critic networks have 30 neurons in their intermediate layers. Except for the intermediate and output layers of the Actor network, which use the tanh activation function, the other layers use the ReLU activation function. A car-following model framework based on DDPG was obtained.

2. The method for constructing a deep reinforcement learning vehicle car-following model that integrates driving styles according to claim 1, characterized in that: The processing operation of the NGSIM raw data in step S1 is as follows: Select the data from 04:00 p.m. to 04:15 pm in the I-80 segment as the raw data, and use the Savitzky-Golay filtering algorithm to smooth the raw data for noise. Select the window length of Savitzky-Golay filter as 21 and the polynomial order as 3 to smooth the raw data.

3. The method for constructing a deep reinforcement learning vehicle car-following model that integrates driving styles according to claim 1, wherein the rules for filtering the NGSIM raw data in step S1 are as follows: S1.1 Select the vehicle type as small car, i.e., v_Class=2; S1.2, The duration of following the vehicle is greater than 15 seconds; S1.3, When centering with the target vehicle, the headway between the target vehicle and the main vehicle should be less than 150 ft (feet), i.e., Space_Headway (headway) <= 150; S1.4 When following another vehicle, the speeds of both the lead vehicle and the target vehicle must be greater than 0. In the S1.5 and I-80 detection sections, lanes 1-5 are driving lanes, lane 6 is the collector / distributor lane, lane 7 is the entrance ramp, and lane 8 is the exit ramp. Only the following pairs with Lane_ID 1-5 are selected.

4. The method for constructing a deep reinforcement learning vehicle car-following model that integrates driving styles according to claim 1, characterized in that, The specific operation of step S6 is as follows: Step S6.1, Model Testing: In order to compare the differences between car-following models with different driving styles, different car-following models were tested on the test data. Specifically, the distribution of following distance, relative speed and acceleration of different car-following models on the test data were compared. Step S6.2: Randomly select car-following pairs in the test set to compare different car-following models: Input the selected car-following pairs into the driving style identification model established in step S3 to obtain their driving styles. By comparing the NGSIM driving data and car-following models of different styles, it can be seen that the aggressive car-following model is closer to the NGSIM driving data in terms of following distance, relative speed, and acceleration compared to the conservative car-following model.

Images