Vehicle following speed control method based on deep reinforcement learning

Abstract

The application provides a vehicle following speed control method based on deep reinforcement learning, comprising the following steps: based on a real driving data set, extracting automatic driving following pairs in a "leading vehicle-automatic driving vehicle-following vehicle" three-vehicle mode; designing a Markov reward process; based on the requirements of vehicle safety and comfort in the three-vehicle mode, designing a reinforcement learning reward function; constructing a vehicle following speed control network framework based on deep reinforcement learning; training the obtained network to obtain a vehicle following speed control model based on deep reinforcement learning; and testing the obtained network model to verify the safety and comfort of the model. The application alleviates the contradiction between the traditional deep reinforcement learning "trial and error" and the safety of automatic driving.

Description

Technical Field

[0001] This invention belongs to the field of autonomous vehicle car-following technology for micro-traffic flow, specifically involving a vehicle car-following speed control method based on deep reinforcement learning. Background Technology

[0002] In recent years, my country's urbanization has been advancing rapidly, leading to a significant increase in the travel demands of urban residents. This increase in travel volume and the number of motor vehicles has placed immense pressure on urban road safety and energy issues. Therefore, alleviating traffic congestion and improving the safety of transportation systems are urgent problems that need to be addressed. Faced with these complex traffic scenarios, the emergence of autonomous driving technology offers a new approach to solving these problems. Compared to human drivers, autonomous vehicles have advantages in reducing traffic accidents, alleviating traffic pressure, better adapting to different population groups, reducing air pollution, and lowering driving labor and time costs. Autonomous driving technology is a complex multidisciplinary technology, and many scholars have attempted to use car-following models as an adaptive cruise control strategy. Deep reinforcement learning breaks through the traditional rule-based behavioral decision-making methods. Decisions are made by the agent interacting with the environment to obtain the maximum cumulative reward value. The Deep Deterministic Policy Gradient (DDPG) algorithm is a classic algorithm in deep reinforcement learning used to solve continuous control problems. It adopts a deterministic policy gradient descent approach to find the optimal parameters of the neural network. While this exploration method accelerates the convergence speed of the algorithm, it still suffers from overestimation in the value network. Summary of the Invention

[0003] The purpose of this invention is to propose a vehicle following speed control method based on deep reinforcement learning, which alleviates the overestimation problem in deep deterministic policy gradient algorithms by using a dual-value network module. Furthermore, a reward function based on the autonomous driving risk level is designed for a three-vehicle mode to improve driving safety and efficiency.

[0004] The technical solution adopted in this invention is a vehicle following speed control method based on deep reinforcement learning, and the specific steps are as follows:

[0005] Step 1: Taking a sedan as the research object, based on the real dataset HighD, extract autonomous driving car-following pairs in three vehicle modes: lead car, autonomous vehicle, and follow car.

[0006] Step 2: Design a Markov reward process;

[0007] Step 3: Construct a vehicle car-following speed control network framework based on deep reinforcement learning, which mainly consists of four parts: environment construction, experience pool, policy network, and value network under three-vehicle mode.

[0008] Step 4: Train the network from Step 3 to obtain a vehicle following speed control model based on deep reinforcement learning;

[0009] Step 5: Test the control model after step 4 to verify the model's safety and comfort.

[0010] The invention is further characterized in that,

[0011] Step 1 is as follows:

[0012] Step 1.1: Select the lead vehicle, and the following vehicle should maintain its following status without changing lanes;

[0013] Step 1.2: Select a following trajectory segment with a length of not less than 15 seconds.

[0014] Step 2 is implemented as follows:

[0015] Step 2.1: Design and construct the Markov reward process. The reward function consists of a quadruple: M =<S,A,R,γ> , where M represents a Markov decision process, S represents the state space, A represents the action space, R represents the reward function, γ represents the discount factor, and γ∈[0,1];

[0016] Step 2.2: Design the state space S of the Markov reward process. The state space of the autonomous vehicle's following process is defined as a vector, S = [Δv] ls ,Δv fs ,Δy ls ,Δy fs ],

[0017] Where, Δv ls Δv represents the speed difference between the autonomous vehicle and the lead vehicle. fs Δy represents the speed difference between the autonomous vehicle and the following vehicle. ls Δy represents the relative distance between the autonomous vehicle and the guide vehicle. fs Represents the relative distance between the autonomous vehicle and the following vehicle;

[0018] Step 2.3: Design the action space A of the Markov reward process. Since the longitudinal speed control of the autonomous vehicle is mainly based on acceleration a, A = a.

[0019] Step 2.4: Design the reward function for the Markov reward process.

[0020] The design of the reward function for the Markov reward process in step 2.4 is as follows:

[0021] Step 2.4.1: The safest situation is when the speed of the autonomous vehicle is neither higher than the speed of the vehicle in front nor lower than the speed of the vehicle behind. Its reward function is:

[0022] R = 5

[0023] Step 2.4.2: When the speed of the autonomous vehicle is no higher than the speed of the vehicle in front and lower than the speed of the vehicle behind, its reward function is:

[0024]

[0025] Where, d sf x is the stopping distance between an autonomous vehicle and the vehicle behind it. s x represents the location of the autonomous vehicle. f Represents the position of the following vehicle, x l Represents the position of the guide vehicle, a max This is the maximum acceleration;

[0026] Step 2.4.3: When the speed of the autonomous vehicle is higher than the speed of the vehicle in front and not lower than the speed of the vehicle behind, its reward function is:

[0027]

[0028] Where, d ls The parking distance between an autonomous vehicle and the vehicle behind it;

[0029] Step 2.4.4: When the speed of the autonomous vehicle is higher than the speed of the vehicle in front but lower than the speed of the vehicle behind, and only when the distance between the autonomous vehicle and the vehicle behind is greater than the stopping distance, the reward function is:

[0030]

[0031] Step 2.4.5: When the speed of the autonomous vehicle is higher than the speed of the vehicle in front but lower than the speed of the vehicle behind, and only when the distance between the autonomous vehicle in front and the autonomous vehicle is greater than the stopping distance, its reward function is:

[0032]

[0033] Step 2.4.6: When the speed of the autonomous vehicle is higher than the speed of the vehicle in front but lower than the speed of the vehicle behind, and the distance between the autonomous vehicle in front and the vehicle behind is greater than the stopping distance, the collision avoidance priority is determined based on the quotient of the relative distance and the relative speed.

[0034] ①|x s -x f | / |v s -v f |>|x s -x l | / |vs -v l When |, the reward function is:

[0035]

[0036] ②|x s -x f | / |v s -v f |≤|x s -x l | / |v s -v l When |, the reward function is:

[0037]

[0038] Among them, v s v represents the speed of autonomous vehicles l v represents the speed of the lead vehicle. f It represents the speed of the car following it.

[0039] The specific implementation steps of step 3 are as follows:

[0040] Step 3.1: Establish the experience pool Replay Buffer;

[0041] Step 3.1.1: Establish an experience pool Replay Buffer to decompose the data of autonomous vehicle interaction with the environment into trajectories (s t ,a t ,r t ,s t+1 ) Stored in the experience pool; where s t a represents the state of the autonomous vehicle at time t. t r represents the action of the autonomous vehicle at time t. t s represents the reward for the autonomous vehicle at time t. t+1 This represents the state of the autonomous vehicle at time t+1;

[0042] Step 3.1.2: Subsequently, for each training session, a small batch of data is randomly sampled from the experience pool for network training;

[0043] Step 3.2: Establish the policy network, i.e., the Actor network;

[0044] Step 3.2.1: Establish the current policy network with network parameter π. φ ;

[0045] Step 3.2.2: Establish the target policy network and initialize the network parameters φ′←φ;

[0046] Step 3.2.3: Input a small batch of randomly sampled data from the experience pool into the policy network and smooth the target policy; introduce a regularization term, i.e., Gaussian noise, to smooth the target policy network, as specifically implemented below:

[0047] a′=π φ′ (s′)+∈

[0048] Where a′ is the acceleration at the next moment, and s′ is the state at the next moment;

[0049] Step 3.3: Establish a value network, i.e., a Critic network;

[0050] Step 3.3.1: Establish two independent current value networks The parameters of the two networks are θ1 and θ2;

[0051] Step 3.3.2: Establish two independent target value networks and initialize the network parameters θ1′←θ1, θ2←θ2;

[0052] Step 3.3.3: Input the small batch of data randomly sampled from the experience pool. and The network is trained and the target is updated. When updating the target, a smaller target in the target network is selected for updating, in order to mitigate the overestimation caused by the value network. The updated target is shown below:

[0053]

[0054] Where r is the reward for autonomous vehicles;

[0055] Step 3.3.4: Use the above formula with... and The mean squared error is calculated and then used as the loss function to update the current value network. (Loss function) As shown below:

[0056]

[0057] Where s represents the current state of the autonomous vehicle; N represents the number of policy network updates;

[0058] Step 3.4: Delay target network update;

[0059] Step 3.4.1: When the valuation of the value network is inaccurate, the strategy will update in the wrong direction. Therefore, when updating the network, the strategy network update speed is set to be slower than the value. That is, the current strategy network, the target strategy network, and the target value network are updated only after the current value network is updated twice.

[0060] Step 3.4.2: Update the current policy network using the deterministic policy gradient. The deterministic policy gradient is shown below:

[0061]

[0062] Step 3.4.3: Perform a soft update on the target network to stabilize its training. The update method is as follows:

[0063] θ′ i＝1,2 ←τθ i＝1,2 +(1-τ)θ i ′

[0064] φ′←τφ+(1-τ)φ′

[0065] Where, θ i＝1,2 Let θ′ be the parameters of the two current target value networks. i＝1,2 Let φ be the parameters of the two target value networks at the next time step, φ be the parameters of the current target policy network, φ′ be the parameters of the two target policy networks at the next time step, and τ be the soft update parameter τ∈[0,1].

[0066] The specific implementation steps of step 4 are as follows:

[0067] Step 4.1: Extract the three-vehicle mode car-following pairs of the guide vehicle, autonomous vehicle and guide vehicle, and randomly sample 80% of the extracted car-following pairs as the training set and 20% as the test set;

[0068] Step 4.2: Set the Adam optimizer, learning rate to 3e-4, discount reward rate to 0.99, experience pool size to 10e6, and mini-batch size to 256;

[0069] Step 4.3: Input the extracted car-following pairs into the network, train the network, and calculate the collision avoidance rate and average reward of the current autonomous vehicle after every 200 rounds.

[0070] Step 4.4: Save the model after training is complete.

[0071] The specific implementation steps of step 5 are as follows:

[0072] Step 5.1: Load the trained model and read the test dataset;

[0073] Step 5.2: Calculate the collision avoidance rate and average reward of the autonomous vehicle on the test dataset;

[0074] Step 5.3: Verify the Time-to-Collision (TTC) of the autonomous vehicle and compare it with that of a current human driver. The formula for calculating the TTC is as follows:

[0075]

[0076] Where, x n+1 The position of the vehicle in front, x n For the position of the rear vehicle, v n v is the speed of the following vehicle. n+1 The speed of the vehicle in front;

[0077] Step 5.4: To demonstrate the driving efficiency of the model, the headway TH of the autonomous vehicle is tested and compared with that of a current human driver. The formula for calculating TH is as follows:

[0078] TH=(x n+1 -x n ) / v n .

[0079] The beneficial effects of this invention are:

[0080] Compared to existing technologies, this invention proposes an autonomous vehicle following method based on a dual-value neural network module, which reduces the potential overestimation problem in the value network. Furthermore, it designs a reinforcement learning reward function based on safety levels determined by vehicle speed and distance in a three-vehicle mode ("leader vehicle—autonomous vehicle—follower vehicle"), making the driver more attentive to changes in the surrounding vehicle states and alleviating the conflict between the "trial and error" approach of traditional reinforcement learning and the safety of autonomous driving. Compared to previous reinforcement learning methods, this invention's method also demonstrates better generalization ability for driving scenarios not experienced during training. Attached Figure Description

[0081] Figure 1 This is a flowchart of the vehicle following speed control method based on deep reinforcement learning according to the present invention;

[0082] Figure 2 It is a diagram of a three-car model;

[0083] Figure 3 This is a network framework diagram of the vehicle following speed control method based on deep reinforcement learning in this invention;

[0084] Figure 4 It is the model collision avoidance rate during model training. Detailed Implementation

[0085] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0086] Example 1

[0087] See appendix Figure 1 This invention relates to a vehicle following speed control method based on deep reinforcement learning, which specifically includes the following steps:

[0088] Step 1: Taking a sedan as the research object, based on the real dataset HighD, extract autonomous driving car-following pairs in three vehicle modes: lead car, autonomous vehicle, and follow car.

[0089] Step 2: Design a Markov reward process;

[0090] Step 3: Construct a vehicle car-following speed control network framework based on deep reinforcement learning, which mainly consists of four parts: environment construction, experience pool, policy network, and value network under three-vehicle mode.

[0091] Step 4: Train the network from Step 3 to obtain a vehicle following speed control model based on deep reinforcement learning;

[0092] Step 5: Test the control model after step 4 to verify the model's safety and comfort.

[0093] Example 2

[0094] The present invention provides a vehicle car-following speed control method based on deep reinforcement learning, as detailed below:

[0095] Step 1: Based on the German real-world driving dataset HighD, extract autonomous driving follow pairs in a three-vehicle mode: "leader vehicle - autonomous vehicle - follower vehicle". The three-vehicle modes are shown in the attached figure. Figure 2 As shown;

[0096] Step 1.1: Based on the German real-world driving dataset HighD, select passenger cars as the research object;

[0097] Step 1.2: Select a lead vehicle, follow vehicle, etc., and maintain a following state without changing lanes;

[0098] Step 1.3: Select a following trajectory segment with a length of not less than 15 seconds.

[0099] Step 2: Design a Markov reward process;

[0100] Step 2.1: Design and construct the Markov reward process, using a simplified Markov reward process consisting of a quadruple: M =<S,A,R,γ> , where M represents a Markov decision process, S represents the state space, A represents the action space, R represents the reward function, γ represents the discount factor, and γ∈[0,1];

[0101] Step 2.2: Design the state space S of the Markov reward process. Since the interaction between the three vehicle modes is mainly based on kinematic processes, the state space of the autonomous vehicle's following process is defined as a vector, S = [Δv]. ls ,Δv fs ,Δx ls ,Δxfs ], where Δv ls Δv represents the speed difference between the autonomous vehicle and the lead vehicle. fs Δx represents the speed difference between the autonomous vehicle and the following vehicle. ls Δx represents the relative distance between the autonomous vehicle and the guide vehicle. fs Represents the relative distance between the autonomous vehicle and the following vehicle;

[0102] Step 2.3: Design the action space A of the Markov reward process. Since the longitudinal speed control of the autonomous vehicle is mainly based on acceleration a, A = a.

[0103] Step 2.4: Design the reward function for the Markov reward process.

[0104] Step 2.4.1: The safest situation is when the speed of the autonomous vehicle is neither higher than the speed of the vehicle in front nor lower than the speed of the vehicle behind. Its reward function is:

[0105] R = 5

[0106] Step 2.4.2: When the speed of the autonomous vehicle is no higher than the speed of the vehicle in front and lower than the speed of the vehicle behind, its reward function is:

[0107]

[0108] Where, d sf x represents the stopping distance between the autonomous vehicle and the vehicle behind it. s x represents the location of the autonomous vehicle. f Represents the position of the following vehicle, x l Represents the position of the guide vehicle, a max This is the maximum acceleration;

[0109] Step 2.4.3: When the speed of the autonomous vehicle is higher than the speed of the vehicle in front and not lower than the speed of the vehicle behind, its reward function is:

[0110]

[0111] Where, d ls The parking distance between an autonomous vehicle and the vehicle behind it;

[0112] Step 2.4.4: When the speed of the autonomous vehicle is higher than the speed of the vehicle in front but lower than the speed of the vehicle behind, and only when the distance between the autonomous vehicle and the vehicle behind is greater than the stopping distance, the reward function is:

[0113]

[0114] Step 2.4.5: When the speed of the autonomous vehicle is higher than the speed of the vehicle in front but lower than the speed of the vehicle behind, and only when the distance between the autonomous vehicle in front and the autonomous vehicle is greater than the stopping distance, its reward function is:

[0115]

[0116] Step 2.4.6: When the speed of the autonomous vehicle is higher than the speed of the vehicle in front but lower than the speed of the vehicle behind, and the distance between the autonomous vehicle in front and the vehicle behind is greater than the stopping distance, the collision avoidance priority is determined based on the quotient of the relative distance and the relative speed.

[0117] ①|x s -x f | / |v s -v f |>|x s -x l | / |v s -v l When |, the reward function is:

[0118]

[0119] ②|x s -x f | / |v s -v f |≤|x s -x l | / |v s -v l When |, the reward function is:

[0120]

[0121] Among them, v s v represents the speed of autonomous vehicles l v represents the speed of the lead vehicle. f Represents the speed of the car following the vehicle;

[0122] In this method, the autonomous vehicle needs to make a balanced decision regarding collision avoidance between the lead vehicle and the following vehicle, namely, how to accelerate and decelerate to maintain a three-vehicle mode without a collision. Rewards are given based on different states and chosen actions, and the reduction in rewards is set according to the degree of decrease in safety between the vehicles.

[0123] Step 3: Construct a vehicle car-following speed control network framework based on deep reinforcement learning, as shown in the attached figure. Figure 3 As shown, it mainly consists of four parts: environment construction, experience pool, policy network, and value network under the three-vehicle mode.

[0124] Step 3.1: Establish the experience pool (Replay Buffer);

[0125] Step 3.1.1: Establish an experience pool, decomposing the data of autonomous vehicle interaction with the environment into trajectories (s t ,a t ,r t ,s t+1 ) Stored in the experience pool, where s t a represents the state of the autonomous vehicle at time t. t r represents the action of the autonomous vehicle at time t. t s represents the reward for the autonomous vehicle at time t. t+1 This represents the state of the autonomous vehicle at time t+1;

[0126] Step 3.1.2: Subsequently, for each training session, a small batch of data is randomly sampled from the experience pool and input into the policy network for training;

[0127] Step 3.2: Establish the policy network, i.e., the Actor network;

[0128] Step 3.2.1: Establish the current policy network with network parameter π. φ ;

[0129] Step 3.2.2: Establish the target policy network and initialize the network parameters φ′←φ;

[0130] Step 3.2.3: Input the mini-batch data randomly sampled from the experience pool into the updated policy network and smooth the target policy. Because the policy network is prone to overfitting, a regularization term, namely Gaussian noise, is introduced to smooth the target policy network. The specific implementation is as follows:

[0131] a′=π φ′ (s′)+∈

[0132] Where a′ is the acceleration at the next moment, and s′ is the state at the next moment;

[0133] Step 3.3: Establish a value network;

[0134] Step 3.3.1: Establish two independent current value networks The parameters of the two networks are θ1 and θ2;

[0135] Step 3.3.2: Establish two independent target value networks and initialize the network parameters θ1′←θ1, θ′2←θ2;

[0136] Step 3.3.3: Input the small batch of data randomly sampled from the experience pool. and The network is trained and the target is updated. Considering that the value network might overestimate the value in practical applications, a smaller target from the target network is selected for updating to mitigate this overestimation. The updated target is shown below:

[0137]

[0138] Where r is the reward for autonomous vehicles;

[0139] Step 3.3.4: Use the above formula with... and The mean squared error is calculated and then used as the loss function to update the current value network. (Loss function) As shown below:

[0140]

[0141] Where s is the current state of the autonomous vehicle, and N is the number of policy network updates;

[0142] Step 3.4: Delay target network update;

[0143] Step 3.4.1: When the value valuation is inaccurate, the strategy will update in the wrong direction. Therefore, when updating the network, the strategy network update speed should be set slower than the value network. That is, the current strategy network, target strategy network, and target value network should only be updated after the current value network has been updated twice.

[0144] Step 3.4.2: Update the current policy network using the deterministic policy gradient. The deterministic policy gradient is shown below:

[0145]

[0146] Step 3.4.3: Perform a soft update on the target network to stabilize its training. The update method is as follows:

[0147] θ′ i＝1,2 ←τθ i＝1,2 +(1-τ)θ i ′

[0148] φ′←τφ(1-τ)φ′

[0149] Where, θ i＝1,2 Let θ′ be the parameters of the two current target value networks. i＝1,2 Let φ be the parameters of the two target value networks at the next time step, φ be the parameters of the current target policy network, φ′ be the parameters of the two target policy networks at the next time step, and τ be the soft update parameter τ∈[0,1].

[0150] Step 4: Train the network from Step 3 to obtain a vehicle following speed control model based on deep reinforcement learning;

[0151] Step 4.1: Extract car-following pairs using the "guide vehicle - autonomous vehicle - guide vehicle" three-vehicle pattern. Randomly sample 80% of the extracted car-following pairs as the training set and 20% as the test set.

[0152] Step 4.2: Set the Adam optimizer, learning rate to 3e-4, discount reward rate to 0.99, experience pool size to 10e6, and mini-batch size to 256;

[0153] Step 4.3: Input the extracted car-following pairs into the network, train the network, and calculate the collision avoidance rate and average reward of the current autonomous vehicle after every 200 car-following cycles.

[0154] Step 4.4: Save the model after training. The collision avoidance rate of the model during training is shown in the attached figure. Figure 4 As shown, the x-axis represents the number of training rounds, and the y-axis represents the cumulative collision avoidance rate of the autonomous vehicle during the following process. It can be seen that after more than two thousand training sessions, the model's collision avoidance rate can exceed 90%, proving the effectiveness and safety of the model.

[0155] Step 5: Test the network model after step 5.

[0156] Example 3

[0157] The difference from Example 2 is that,

[0158] Step 5.1: Load the trained model and read the test dataset;

[0159] Step 5.2: Calculate the collision avoidance rate and average reward of the autonomous vehicle on the test dataset;

[0160] Step 5.3: To demonstrate the driving safety of the model, the time-to-collision (TTC) of the autonomous vehicle is tested and compared with that of a current human driver. The formula for calculating the time-to-collision is as follows:

[0161]

[0162] Where, x n+1 The position of the vehicle in front, x n For the position of the rear vehicle, v n v is the speed of the following vehicle. n+1 The speed of the vehicle in front.

[0163] Step 5.4: To demonstrate the model's driving efficiency, the headway (TH) of the autonomous vehicle is tested and compared with that of a current human driver. The headway calculation formula is as follows:

[0164] TH=(x n+1 -x n ) / v n

[0165] Where, x n+1 The position of the vehicle in front, x n For the position of the rear vehicle, v n The speed of the following vehicle.

Claims

1. A vehicle following speed control method based on deep reinforcement learning, characterized in that, The specific steps are as follows: Step 1: Taking a sedan as the research object, based on the real dataset HighD, extract autonomous driving car-following pairs in three vehicle modes: lead car, autonomous vehicle, and follow car. Step 2: Design a Markov reward process; The specific steps are as follows: Step 2.1: Design and construct the Markov reward process, where the reward function consists of a quadruple: ,in Representing the Markov decision-making process, Represents the state space. Represents the action space, Represents the reward function. Represents the discount factor. ; Step 2.2: Design the state space of the Markov reward process The state space of an autonomous vehicle following another vehicle is defined as a vector. , in, This represents the speed difference between the autonomous vehicle and the lead vehicle. This represents the speed difference between the autonomous vehicle and the car following it. This represents the relative distance between the autonomous vehicle and the guide vehicle. Represents the relative distance between the autonomous vehicle and the following vehicle; Step 2.3: Design the action space of the Markov reward process Since the longitudinal speed control of autonomous vehicles is mainly based on acceleration ,therefore ; Step 2.4: Design the reward function for the Markov reward process; the specific steps are as follows: Step 2.4.1: The safest situation is when the speed of the autonomous vehicle is neither higher than the speed of the lead vehicle nor lower than the speed of the following vehicle. Its reward function is: Step 2.4.2: When the speed of the autonomous vehicle is no higher than the speed of the lead vehicle and lower than the speed of the following vehicle, its reward function is: in, The stopping distance between autonomous vehicles and following vehicles. Represents the location of autonomous vehicles. Represents the position of the following car. Indicates the position of the guide vehicle. This is the maximum acceleration; Step 2.4.3: When the speed of the autonomous vehicle is higher than the speed of the lead vehicle and not lower than the speed of the following vehicle, its reward function is: in, The stopping distance between autonomous vehicles and following vehicles; Step 2.4.4: When the speed of the autonomous vehicle is higher than the speed of the lead vehicle but lower than the speed of the following vehicle, and only when the distance between the following vehicle and the autonomous vehicle is greater than the stopping distance, the reward function is: Step 2.4.5: When the speed of the autonomous vehicle is higher than the speed of the lead vehicle but lower than the speed of the following vehicle, and only when the distance between the lead vehicle and the autonomous vehicle is greater than the stopping distance, the reward function is: Step 2.4.6: When the speed of the autonomous vehicle is higher than that of the lead vehicle but lower than that of the following vehicle, and the distance between the lead vehicle and the autonomous vehicle and the distance between the following vehicle and the autonomous vehicle are both greater than the stopping distance, the collision avoidance priority is determined based on the quotient of the relative distance and the relative speed. ① At that time, the reward function is: ② At that time, the reward function is: in, Represents the speed of autonomous vehicles. Represents the speed of the lead vehicle. Represents the speed of the following vehicle. Step 3: Construct a vehicle car-following speed control network framework based on deep reinforcement learning, which mainly consists of four parts: environment construction, experience pool, policy network, and value network under three-vehicle mode. Step 4: Train the network from Step 3 to obtain a vehicle following speed control model based on deep reinforcement learning; Step 5: Test the control model after step 4 to verify the model's safety and comfort.

2. The vehicle following speed control method based on deep reinforcement learning according to claim 1, characterized in that, The specifics of step 1 are as follows: Step 1.1: Select the lead vehicle, and the following vehicle should maintain its following status without changing lanes; Step 1.2: Select a following trajectory segment with a length of not less than 15 seconds.

3. The vehicle following speed control method based on deep reinforcement learning according to claim 1, characterized in that, The specific implementation steps of step 3 are as follows: Step 3.1: Establish the experience pool Replay Buffer; Step 3.1.1: Establish an experience pool Replay Buffer to decompose the data of autonomous vehicle interaction with the environment into trajectories. Stored in the experience pool; This represents the state of the autonomous vehicle at time t. This represents the actions of the autonomous vehicle at time t. This represents the reward for the autonomous vehicle at time t. This represents the state of the autonomous vehicle at time t+1; Step 3.1.2: Subsequently, for each training session, a small batch of data is randomly sampled from the experience pool for network training; Step 3.2: Establish the policy network, i.e., the Actor network; Step 3.2.1: Establish the current policy network with the following network parameters: ; Step 3.2.2: Establish the target policy network and initialize the network parameters. ; Step 3.2.3: Input a small batch of randomly sampled data from the experience pool into the policy network and smooth the target policy; introduce a regularization term, i.e., Gaussian noise. To smooth the target policy network, the specific implementation is as follows: in, For the acceleration in the next moment, The state at the next moment; Step 3.3: Establish a value network, i.e., a Critic network; Step 3.3.1: Establish two independent current value networks The parameters of two of the networks are and ; Step 3.3.2: Establish two independent target value networks and initialize the network parameters. , ; Step 3.3.3: Input the small batch of data randomly sampled from the experience pool. and The network is trained and the target is updated. When updating the target, a smaller target in the target network is selected for updating, in order to mitigate the overestimation caused by the value network. The updated target is shown below: in, r It is a reward for autonomous vehicles; Step 3.3.4: Use the above formula with... and The mean squared error is calculated and then used as the loss function to update the current value network. (Loss function) As shown below: in, This represents the current state of the autonomous vehicle. For the number of times the policy network is updated; Step 3.4: Delay target network update; Step 3.4.1: When the valuation of the value network is inaccurate, the strategy will update in the wrong direction. Therefore, when updating the network, the strategy network update speed is set to be slower than the value. That is, the current strategy network, the target strategy network, and the target value network are updated only after the current value network is updated twice. Step 3.4.2: Update the current policy network using the deterministic policy gradient. The deterministic policy gradient is shown below: Step 3.4.3: Perform a soft update on the target network to stabilize its training. The update method is as follows: in, The parameters of the two current target value networks are... The parameters of the two target value networks at the next time step. These are the parameters of the current target policy network. Let these be the parameters of the two target policy networks at the next time step. For soft update parameters .

4. The vehicle following speed control method based on deep reinforcement learning according to claim 1, characterized in that, The specific implementation steps of step 4 are as follows: Step 4.1: Extract the three-vehicle mode car-following pairs of the guide vehicle, autonomous vehicle and car-following vehicle, and randomly sample 80% of the extracted car-following pairs as the training set and 20% as the test set; Step 4.2: Set the Adam optimizer, learning rate to 3e-4, discount reward rate to 0.99, experience pool size to 10e6, and mini-batch size to 256; Step 4.3: Input the extracted car-following pairs into the network, train the network, and calculate the collision avoidance rate and average reward of the current autonomous vehicle after every 200 rounds. Step 4.4: Save the model after training is complete.

5. The vehicle following speed control method based on deep reinforcement learning according to claim 1, characterized in that, The specific implementation steps of step 5 are as follows: Step 5.1: Load the trained model and read the test dataset; Step 5.2: Calculate the collision avoidance rate and average reward of the autonomous vehicle on the test dataset; Step 5.3: Verify the Time-to-Collision (TTC) of the autonomous vehicle and compare it with that of a current human driver. The formula for calculating the TTC is as follows: in, Position for the guide vehicle To follow the position of the car, To keep up with the speed of the car, To guide the speed of the vehicle; Step 5.4: To demonstrate the driving efficiency of the model, the headway TH of the autonomous vehicle is tested and compared with that of a current human driver. The formula for calculating TH is as follows: 。

Images