Automatic driving vehicle scheduling method under continuous confluence and divergence bottleneck traffic scene

Abstract

The invention discloses a collaborative automatic driving vehicle scheduling method based on a deep reinforcement learning framework. The method comprises the following steps: firstly, designing a reward function formed by combining a system-level speed reward and a time headway punishment; then, a joint state space based on a reinforcement learning framework is designed, and the joint state space comprises information such as relative speed and relative distance between a front vehicle / rear vehicle of the same lane, a front vehicle / rear vehicle of a left adjacent lane and a front vehicle / rear vehicle of a right adjacent lane; thirdly, combining the acceleration output vector and the lane changing action of the automatic driving vehicle, and designing a joint action space for reinforcement learning; finally, the designed deep reinforcement learning framework is fully trained, stable vehicle speed, lane changing and other control information can be output, and cooperative intelligent control over the automatic driving vehicle is achieved. According to the method provided by the invention, the average passing speed of the vehicle under the condition of mixed driving of the manual driving vehicle and the automatic driving vehicle in the continuous confluence and divergence bottleneck traffic scene can be improved, and the collision danger frequency of the vehicle is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and specifically relates to an automatic driving vehicle scheduling method in a continuous merging and diverting bottleneck traffic scenario. Background Art

[0002] With the continuous development of urban transportation and the continuous innovation of intelligent technology, V2X communication technology, as a key means of intelligent transportation, has attracted widespread attention in the field of traffic management. In order to effectively improve the traffic efficiency of traffic nodes, reduce the incidence of traffic congestion, and thus improve the overall operation of urban transportation, it is particularly important to study connected and autonomous vehicles (CAV) based on vehicle networking technology and autonomous driving technology. Through real-time information exchange between vehicles and the optimization and scheduling of traffic flow with the help of intelligent algorithms, it is expected to effectively improve the traffic capacity of vehicles at traffic nodes and make positive contributions to the construction of an efficient, intelligent, and green urban transportation system.

[0003] The development of autonomous vehicles throughout the industry raises the question of how autonomous vehicles can help reduce traffic congestion. Since autonomous vehicles are controlled by predefined policies, they can be made to act selflessly and drive strategically, thereby influencing the behavior of human drivers and reducing congestion. For example, an autonomous vehicle may decelerate smoothly when reaching a ramp to allow another car to merge into the main road, causing the following vehicles to also decelerate smoothly, thereby preventing the propagation of stop-and-go congestion waves caused by sharp deceleration. Past studies have shown that a small number of autonomous vehicles executing controlled agent driving policies in human-driven traffic can alleviate congestion in simplified simulations and real-world scenarios. Nowadays, autonomous driving technology is becoming more and more mature, and there are more and more autonomous vehicles appearing on real roads. Therefore, mixed traffic flow is a typical representative of the current intelligent transportation system, which includes situations where autonomous vehicles and traditional non-intelligent vehicles co-drive on the same road section. With the development of autonomous driving technology, the popularization of autonomous vehicles will become a major trend in future intelligent transportation systems. Therefore, studying the driving strategy of scheduling autonomous vehicles in mixed traffic flow is of great significance to promote the development of intelligent transportation systems. In this new era of intelligent transportation systems, reinforcement learning (RL) technology has become a key driving force in the field of autonomous driving. Reinforcement learning-based autonomous driving technology provides autonomous vehicles with efficient decision-making and control strategies, enabling them to respond accurately and quickly in complex mixed traffic flows, thereby improving the operating efficiency of the overall transportation system.

[0004] Although many researchers have studied how to alleviate traffic congestion, most of them focus on the scenario of one main road and one entrance ramp, without considering the problem of multiple lanes. The present invention will model a scenario including multiple main roads, multiple entrance ramps and exit ramps, and then study the intelligent driving strategy of the reinforcement learning algorithm on the road model after merging and then diverging to complete the scheduling of autonomous driving vehicles, thereby alleviating traffic congestion problems at merging points and diverging points. Summary of the invention

[0005] In view of the above problems and the shortcomings of the prior art, the present invention proposes a method for dispatching autonomous driving vehicles in a continuous merging and diverging bottleneck traffic scenario. A continuous merging and diverging bottleneck traffic scenario is designed under mixed traffic, and a collaborative intelligent driving strategy based on a deep reinforcement learning framework is used to dispatch autonomous driving vehicles, thereby improving the traffic efficiency of vehicles at traffic nodes, and improving the overall road throughput and vehicle speed.

[0006] The present invention adopts the following technical solutions to solve the above technical problems:

[0007] The present invention comprises the following steps:

[0008] S1. Considering the multi-lane problem, the road network modeling is carried out for the continuous merge-diverge bottleneck traffic scenario, specifically:

[0009] A continuous merging and diverging bottleneck traffic scenario was built under the traffic reinforcement learning framework Flow and combined with the microscopic traffic simulation software SUMO, in which the traffic on the left four-lane main road and the traffic on the two entrance ramps will merge at the road intersection, and then the vehicles will diverge after driving a certain distance on the four-lane main road after the merging point. A continuous merging and diverging bottleneck traffic scenario with a four-lane main road, two entrance ramps and two exit ramps was constructed.

[0010] S2. Construction of autonomous driving vehicle control model based on reinforcement learning, specifically:

[0011] (1) State space design

[0012] For an autonomous driving vehicle, the state space s of each autonomous driving vehicle is represented by using the current local state information of the agent, including: the longitudinal speed of the current autonomous driving vehicle, the relative speeds between the vehicle in front of it and the vehicle in front of it in the same lane, the vehicle in the rear of it in the same lane, the vehicle in front of it in the left adjacent lane, the vehicle in front of it in the right adjacent lane, and the vehicle in rear of it in the right adjacent lane, and the relative distances between the vehicle in front of it and the vehicle in front of it in the same lane, the vehicle in front of it in the left adjacent lane, the vehicle in rear of it in the left adjacent lane, the vehicle in front of it in the right adjacent lane, and the vehicle in rear of it in the right adjacent lane.

[0013] (2) Action space design

[0014] Two actions are designed, namely the acceleration and lane-changing action of the autonomous driving vehicle.

[0015] (3) Reward function design

[0016] The reward function R at time step t is:

[0017] R=ω 1 r 1 +ω 2 r 2

[0018] where r 1 is the system-level speed partial reward function, r 2 is the reward function for the headway penalty part, ω 1 and ω 2 They are the weight coefficients of the system-level speed part reward function and the headway penalty part reward function respectively.

[0019] S3. Build a deep reinforcement learning model for autonomous vehicle control

[0020] The deep reinforcement learning model includes a current policy network, an old policy network, and a value network. Based on the obtained state space information, a deep reinforcement learning model for the autonomous driving vehicle scheduling method is established. The input of the model is the joint state space of the autonomous driving vehicle, and the output is the probability distribution of acceleration actions and lane change actions. The specific action is sampled and selected based on the probability distribution of the action. The deep reinforcement learning model for the autonomous driving vehicle scheduling method is trained by combining deep learning and reinforcement learning, and the overall cycle steps are repeated until the predetermined conditions are met, and finally a trained deep reinforcement learning model is obtained.

[0021] Furthermore, in step S3, the overall cycle steps include:

[0022] (1) Execute the first loop step to obtain each step of interaction information between the autonomous driving vehicle and the environment (S t , A t , R t ), wherein the first cycle step comprises:

[0023] (a) S2, obtains the state space formed by the interaction between the autonomous driving vehicle and the environment, and the local state spaces of all agents are combined to form the global state space S t ;

[0024] (b) The joint state space S t Input into the current deep reinforcement learning model to obtain the joint action space A of the autonomous driving vehicle t, calculate the reward value R t , forming the mutual information of the corresponding step (S t , A t , R t ) and the joint action space A t The information is sent to the corresponding agent, and the output information is the probability distribution of the action. The specific action is sampled and selected according to the probability distribution of the action.

[0025] (c) The autonomous vehicle is based on the joint action space A t Select and execute the corresponding action so that the environment jumps to the next joint state S t+1 ;

[0026] (2) After a round, the interaction information (S t , A t , R t ) are input into the value network to obtain the state value V = [V 1 , V 2 , ..., V T ]. Starting from the last step at the end of the round, we calculate the discounted reward G = [G 1 , G 2 , ..., G T ], whose formula is G t =R t +γG t+1 , where γ is the discount factor;

[0027] (3) Calculate the loss function L(ω) of the value network based on the mean square error between the state value and the discounted reward, and update the value network parameters using gradient descent and back propagation;

[0028] (4) Execute the second loop step to update the current policy network M times (M≤T), wherein the second loop step includes:

[0029] The state S of the interactive information t and action A t Input them into the policy network respectively, and use the importance sampling method to calculate the importance weight I t (θ); Calculate the advantage function based on the state value V and the discounted reward R Combined with the importance weight I t (θ) Find the objective function L of the current policy network CLIP (θ), use back propagation to update the parameters of the current policy network;

[0030] (6) Use the updated parameters of the current policy network to update the parameters of the old policy network.

[0031] Furthermore, the joint state space S of S2 t Specifically:

[0032] S t =(s 1 ,s 2 , ..., s n )

[0033]

[0034] Among them, s i,t represents the state space of the i-th autonomous driving vehicle, S t Represents the global state space formed by combining the local state spaces of all agents. i,t represents the longitudinal velocity of the autonomous vehicle i, Respectively represent the relative speeds between vehicle i and the preceding vehicle in the same lane, the following vehicle in the same lane, the preceding vehicle in the left adjacent lane, the following vehicle in the left adjacent lane, the preceding vehicle in the right adjacent lane, and the following vehicle in the right adjacent lane; They respectively represent the relative distances between vehicle i and the preceding vehicle in the same lane, the following vehicle in the same lane, the preceding vehicle in the left adjacent lane, the following vehicle in the left adjacent lane, the preceding vehicle in the right adjacent lane, and the following vehicle in the right adjacent lane.

[0035] Furthermore, the joint action space A of S2 t Specifically:

[0036] A t =(a 1 , a 2 , ..., a n )

[0037] a i,t =(acc i,t , l i，t )

[0038] where acc i,t ∈[-4.5,2.6]m / s 2 , l i,t ∈{-1, 0, 1}

[0039] Among them, a i，t represents the action space of the i-th autonomous driving vehicle, A t represents the set of action spaces of all agents. i,t represents the acceleration of the i-th vehicle; l i,t Represents the lane changing action of the i-th vehicle, where -1, 0, and 1 represent changing lanes to the left, going straight, and changing lanes to the right, respectively.

[0040] Lane change action i,tis composed of three consecutive values ​​x 1 , x 2 , x 3 It is converted by Softmax operation. First, calculate the exponential function e of the three values. x1 , e x2 , e x3 , and then apply the Softmax function to convert these values ​​into a probability distribution.

[0041] where i∈{1, 2, 3}

[0042] The formula for expanding each component is:

[0043]

[0044]

[0045]

[0046] where p 1 , p 2 , p 3 denote the probabilities of changing lanes to the left, not changing lanes, and changing lanes to the right. The agent chooses its lane-changing action l based on these probabilities. i，t .

[0047] Furthermore, the reward function R of S2 t Specifically:

[0048] R t =ω 1 r 1 +ω 2 r 2

[0049] 1) System-level speed bonus 1 : Calculates a system-level speed bonus that aims to encourage all vehicles to drive as close to the desired speed as possible.

[0050]

[0051]

[0052]

[0053] The maximum cost max_cost is the theoretical maximum deviation value, which is used as the benchmark value for normalization; the current cost cur_cost is the current deviation value of the system, which is used to measure the current system performance; ∈ is a very small value to avoid division by zero errors. The reward is normalized to the interval [0, 1], where 1 means that all vehicles fully reach the desired speed.

[0054] 2) Headway penalty r 2 :For vehicles less than the minimum headway t min This penalty mechanism encourages vehicles to maintain a reasonable safety distance, thereby improving the safety and stability of the overall traffic system.

[0055]

[0056]

[0057] Further, discount reward G t It is defined as the weighted sum of all rewards from time t onwards in a Markov decision process, and is calculated as follows:

[0058] The calculation formula for discount rewards is:

[0059] G t =R t +γG t+1

[0060] Therefore, according to the discounted reward G at the last T step T =R T Step forward and calculate the discounted reward G for each step t :

[0061]

[0062] Where γ is the discount factor, which reflects the relative importance of future rewards at the current moment. If it is close to 0, it indicates that only the rewards at the current moment are evaluated. When it is close to 1, it indicates that long-term rewards are also considered.

[0063] Furthermore, the loss function L(ω) of the value network is calculated as follows:

[0064] L(ω)=MSE(G t -V t )

[0065] Where MSE is the mean square error function, G t is the discounted reward at step t, V t is the state value of step t;

[0066] Based on L(ω), the gradient descent method is used to update the parameter ω of the value network:

[0067]

[0068] Among them, α is the learning rate of the gradient method, It is a gradient operator that uses gradient descent and back propagation to update the parameters of the value network, so that the state value output by the value network is closer to the actual state value.

[0069] Furthermore, the importance weight I t The calculation method of (θ) is as follows:

[0070]

[0071] where π(·|s t ;θ)、π(·|s t θ old ) means that the state s t The policy distribution obtained by inputting the current policy network and the old policy network, θ represents the parameters of the current policy network, θ old represents the parameters of the old policy network, π(a t |s t ;θ),π(a t |s t θ old ) indicates the actual action to be taken t Input into the corresponding strategy distribution to obtain the value of the action under the corresponding strategy distribution.

[0072] Furthermore, the advantage function is calculated based on the state value and the discounted reward Combined with the importance weight I t (θ) Find the objective function L of the current policy network CLIP (θ), including:

[0073] According to the state value V t and discount reward R t Calculate the advantage function for:

[0074]

[0075] According to the advantage function and importance weight I t (θ), find the objective function L of the current policy network CLIP (θ):

[0076]

[0077] Where ε is the restricted importance weight I t (θ) is a hyperparameter, clip(I t (θ), 1-ε, 1+ε) will be the importance weight I t The value of (θ) is limited to the range of (1-ε, 1+ε). t When (θ)<1-ε, let I t(θ)=1-ε, similarly when I t When (θ)>1+ε, let I t (θ) = 1 + ε;

[0078] Based on the objective function L of the current policy network CLIP (θ), and use back-propagation to update the parameters θ of the current policy network.

[0079] Further, the old policy network θ is updated using the updated current policy network parameters θ old , thus completing one iteration.

[0080] θ old ←-θ

[0081] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0082] The present invention first models the road network for the continuous merging and diverting bottleneck traffic scenario, and introduces a multi-agent deep reinforcement learning algorithm. Through intelligent control strategies, the multi-agent and non-autonomous driving vehicles are coordinated to reduce cross-interference. The local state space of each autonomous driving vehicle is combined to form a joint state space, and two actions are designed so that the joint state space is input into the deep reinforcement learning model, which can generate the acceleration output vector and lane change action of the controlled vehicle, and efficiently dispatch the autonomous driving vehicle. Thereby improving the traffic efficiency of the vehicle, improving the throughput rate and vehicle speed of the traffic network, and reducing the number of collision hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a schematic diagram of the continuous merging and diverting bottleneck traffic scenario model of the present invention.

[0084] Figure 2 It is a schematic diagram of the process of the present invention.

[0085] Figure 3 Schematic diagram of agent state information interaction.

[0086] Figure 4(a)-4(c) A bar chart comparing indicators for implementation cases. DETAILED DESCRIPTION

[0087] The specific implementation modes of the present invention are described in detail below with reference to the accompanying drawings.

[0088] Before describing the present invention in detail, the continuous merging and diverting bottleneck traffic scenario designed by the present invention is taken as a specific scenario example. Figure 1 As shown, it is a continuous merging and diverting bottleneck traffic scenario model diagram of the present invention, and some relevant settings of the experimental scenario are as follows:

[0089] (1) There are both autonomous vehicles and non-autonomous vehicles in the scene. All CAVs in the scene are equipped with an on-board unit (OBU) that can obtain their own position and speed information as well as the position and speed information of surrounding vehicles.

[0090] (2) The continuous merging and diverging bottleneck traffic scenario refers to the traffic flow on the left four-lane main road and the traffic flow on the two entrance ramps will merge at the road intersection, and then the vehicles will diverge after driving a certain distance on the four-lane main road after the merging point. The left side is a 400m four-lane entrance main road and two one-lane 200m entrance ramps; the middle section is a 400m four-lane main road; the right side is a 400m two-lane exit main road and two one-lane 200m exit ramps.

[0091] (3) At road intersections, roadside units (RSUs) are set up to assist connected autonomous vehicles (CAVs) in reinforcement learning training. RSUs collect data, upload the data to the cloud or data center, and then train and optimize the model in these centralized computing environments.

[0092] The experimental parameters of this embodiment are set as shown in the following table:

[0093]

[0094] This embodiment provides a method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario, comprising the following steps:

[0095] S1. Considering the multi-lane problem, road network modeling is performed for continuous merge-diverge bottleneck traffic scenarios

[0096] A continuous merging and diverging bottleneck traffic scenario was built under the traffic reinforcement learning framework Flow and combined with the microscopic traffic simulation software SUMO, in which the traffic on the left four-lane main road and the traffic on the two entrance ramps will merge at the road intersection, and then the vehicles will diverge after driving a certain distance on the four-lane main road after the merging point. A continuous merging and diverging bottleneck traffic scenario with a four-lane main road, two entrance ramps and two exit ramps was constructed.

[0097] S2. Construction of autonomous driving vehicle control model based on reinforcement learning

[0098] (1) State space design

[0099] For autonomous driving vehicles, the state space s of each autonomous driving vehicle is represented by using the current local state information of the agent, including: the longitudinal speed of the current autonomous driving vehicle, the relative speeds between the front vehicle in the same lane, the rear vehicle in the same lane, the front vehicle in the left adjacent lane, the rear vehicle in the left adjacent lane, the front vehicle in the right adjacent lane, and the rear vehicle in the right adjacent lane, and the relative distances between the front vehicle in the same lane, the rear vehicle in the same lane, the front vehicle in the left adjacent lane, the rear vehicle in the left adjacent lane, the front vehicle in the right adjacent lane, and the rear vehicle in the right adjacent lane.

[0100] Joint state space S t Specifically:

[0101] S t =(s 1 ,s 2 , ..., s n )

[0102]

[0103] Among them, s i，t represents the state space of the i-th autonomous driving vehicle, S t Represents the global state space formed by combining the local state spaces of all agents. i,t represents the longitudinal velocity of the autonomous vehicle i, Respectively represent the relative speeds between vehicle i and the preceding vehicle in the same lane, the following vehicle in the same lane, the preceding vehicle in the left adjacent lane, the following vehicle in the left adjacent lane, the preceding vehicle in the right adjacent lane, and the following vehicle in the right adjacent lane; They respectively represent the relative distances between vehicle i and the preceding vehicle in the same lane, the following vehicle in the same lane, the preceding vehicle in the left adjacent lane, the following vehicle in the left adjacent lane, the preceding vehicle in the right adjacent lane, and the following vehicle in the right adjacent lane.

[0104] (2) Action space design

[0105] Two actions are designed, namely the acceleration and lane-changing action of the autonomous driving vehicle.

[0106] Joint action space A t Specifically:

[0107] A t =(a 1 , a 2 , ..., a n )

[0108] a i,t =(acc i,t , l i,t )

[0109] where acc i,t ∈[-4.5,2.6]m / s 2 , l i，t ∈{-1, 0, 1}

[0110] Among them, a i,t represents the action space of the i-th autonomous driving vehicle, A t represents the set of action spaces of all agents. i,t represents the acceleration of the i-th vehicle; l i，t Represents the lane changing action of the i-th vehicle, where -1, 0, and 1 represent changing lanes to the left, going straight, and changing lanes to the right, respectively.

[0111] Lane change action i,t is composed of three consecutive values ​​x 1 , x 2 , x 3 It is converted by Softmax operation. First, calculate the exponential function e of the three values. x1 , e x2 , e x3 , and then apply the Softmax function to convert these values ​​into a probability distribution.

[0112] where i∈{1,2,3}

[0113] The formula for expanding each component is:

[0114]

[0115]

[0116]

[0117] where p 1 , p 2 , p 3 denote the probabilities of changing lanes to the left, not changing lanes, and changing lanes to the right. The agent chooses its lane-changing action l based on these probabilities. i,t .

[0118] (3) Reward function design

[0119] Reward function R t Specifically:

[0120] R t =ω 1 r 1 +ω 2 r 2

[0121] 1) System-level speed bonus 1: Calculates a system-level speed bonus that aims to encourage all vehicles to drive as close to the desired speed as possible.

[0122]

[0123]

[0124]

[0125] The maximum cost max_cost is the theoretical maximum deviation value, which is used as the benchmark value for normalization; the current cost cur_cost is the current deviation value of the system, which is used to measure the current system performance; ∈ is a very small value to avoid division by zero errors. The reward is normalized to the interval [0, 1], where 1 means that all vehicles fully reach the desired speed.

[0126] 2) Headway penalty r 2 :For vehicles less than the minimum headway t min This penalty mechanism encourages vehicles to maintain a reasonable safety distance, thereby improving the safety and stability of the overall traffic system.

[0127]

[0128]

[0129] S3. Build a deep reinforcement learning model for autonomous vehicle control

[0130] The deep reinforcement learning model includes a current policy network, an old policy network, and a value network. Based on the obtained state space information, a deep reinforcement learning model for the autonomous driving vehicle scheduling method is established. The input of the model is the joint state space of the autonomous driving vehicle, and the output is the probability distribution of acceleration actions and lane change actions. The specific action is sampled and selected based on the probability distribution of the action. The deep reinforcement learning model for the autonomous driving vehicle scheduling method is trained by combining deep learning and reinforcement learning, and the overall cycle steps are repeated until the predetermined conditions are met, and finally a trained deep reinforcement learning model is obtained.

[0131] In some specific embodiments, Figure 2 As shown, the intelligent vehicle dispatching method for the continuous merging and diverging area scenario of this embodiment includes the following steps:

[0132] Step 1: Initialize the road network environment and network model parameters. The road network environment is set as a 400m four-lane entrance main road and two one-lane 200m entrance ramps on the left; a 400m four-lane main road in the middle; a 400m two-lane exit main road and two one-lane 200m exit ramps on the right, such as Figure 1 As shown. The current penetration rate of autonomous driving vehicles is set to 25%, and they enter the road network from the left entrance main road and entrance ramp. The autonomous driving vehicle scheduling method is based on the PPO (Proximal Policy Optimization) deep reinforcement learning algorithm. Therefore, the deep reinforcement learning model includes the current policy network, the old policy network and the value network. old , ω are used to initialize the three network parameters. The policy network is used to select the appropriate action according to the current state, and the value network is used to score the current state.

[0133] Step 2: During the training process in step t, the autonomous vehicle obtains local state information through its own sensors and sends this information to the roadside unit RSU. Then, the RSU combines this local state information with the global state information it observes to form a joint state space S t , the design is as follows:

[0134] S t =(s 1 ,s 2 , ..., s n )

[0135]

[0136] Among them, s i，t represents the state space of the i-th autonomous driving vehicle, S t Represents the global state space formed by combining the local state spaces of all agents. i，t represents the longitudinal velocity of the autonomous vehicle i, Respectively represent the relative speeds between vehicle i and the preceding vehicle in the same lane, the following vehicle in the same lane, the preceding vehicle in the left adjacent lane, the following vehicle in the left adjacent lane, the preceding vehicle in the right adjacent lane, and the following vehicle in the right adjacent lane; They respectively represent the relative distances between vehicle i and the preceding vehicle in the same lane, the following vehicle in the same lane, the preceding vehicle in the left adjacent lane, the following vehicle in the left adjacent lane, the preceding vehicle in the right adjacent lane, and the following vehicle in the right adjacent lane.

[0137] Step 3: During the training process of step t, S t Input into the current deep reinforcement learning model to generate the joint action space A of the autonomous driving vehicle t The joint action space A t The design is as follows:

[0138] A t =(a 1 , a 2 , ..., an )

[0139] aa i,t =(acc i,t , l i,t )

[0140] where acc i,t ∈[-4.5,2.6]m / s 2 , l i，t ∈{-1, 0, 1}

[0141] Among them, a i,t represents the action space of the i-th autonomous driving vehicle, A t represents the set of action spaces of all agents. i,t represents the acceleration of the i-th vehicle; l i，t Represents the lane changing action of the i-th vehicle, where -1, 0, and 1 represent changing lanes to the left, going straight, and changing lanes to the right, respectively.

[0142] Lane change action i,t is composed of three consecutive values ​​x 1 , x 2 , x 3 It is converted by Softmax operation. First, calculate the exponential function e of the three values. x1 , e x2 , e x3 , and then apply the Softmax function to convert these values ​​into a probability distribution.

[0143] where i∈{1, 2, 3}

[0144] The formula for expanding each component is:

[0145]

[0146]

[0147]

[0148] where p 1 , p 2 , p 3 denote the probabilities of changing lanes to the left, not changing lanes, and changing lanes to the right. The agent chooses its lane-changing action l based on these probabilities. i，t .

[0149] Step 4: Then, the RSU inputs the sampled actions into the environment. After the autonomous vehicle performs these actions, the corresponding reward value is calculated, and then the environment transfers to the next state S t+1 . The reward function R t The design is as follows:

[0150] R t =ω 1 r 1 +ω 2 r 2

[0151] 1) System-level speed bonus 1 : Calculates a system-level speed bonus that aims to encourage all vehicles to drive as close to the desired speed as possible.

[0152]

[0153]

[0154]

[0155] The maximum cost max_cost is the theoretical maximum deviation value, which is used as the benchmark value for normalization; the current cost cur_cost is the current deviation value of the system, which is used to measure the current system performance; f is a very small value to avoid division by zero errors. The reward is normalized to the interval [0, 1], where 1 means that all vehicles fully reach the desired speed.

[0156] 2) Headway penalty r 2 :For vehicles less than the minimum headway t min This penalty mechanism encourages vehicles to maintain a reasonable safety distance, thereby improving the safety and stability of the overall traffic system.

[0157]

[0158]

[0159] Step 5: Store the interaction information obtained at this time step (S t , A t ,R t ).

[0160] Step 6: Execute the loop for T time steps and store the interaction information of each step in the local cache of RSU.

[0161] Step 7: After a round, the interaction information (S t , A t ,R t ) are input into the value network to obtain the state value V = [V 1 ,V 2 , ..., V T ]. Starting from the last step at the end of the round, we calculate the discounted reward G = [G1 , G 2 , ..., G T ], whose formula is G t =R t +γG t+1 , where γ is the discount factor; according to the mean square error between the state value and the discounted reward, the loss function L(ω) of the value network is calculated, and the value network parameters are updated using gradient descent and back propagation.

[0162] Among them, the discount reward G t It is defined as the weighted sum of all rewards from time t onwards in a Markov decision process, and is calculated as follows:

[0163] The calculation formula for discount rewards is:

[0164] G t =R t +γG t+1

[0165] Therefore, according to the discounted reward G at the last T step T =R T Step forward and calculate the discounted reward G for each step t :

[0166]

[0167] Where γ is the discount factor, which reflects the relative importance of future rewards at the current moment. If it is close to 0, it indicates that only the rewards at the current moment are evaluated. When it is close to 1, it indicates that long-term rewards are also considered.

[0168] The loss function L(ω) of the value network is calculated as follows:

[0169] L(ω)=MSE(G t -V t )

[0170] Where MSE is the mean square error function, G t is the discounted reward at step t, V t is the state value of step t;

[0171] Based on L(ω), the gradient descent method is used to update the parameter ω of the value network:

[0172]

[0173] Among them, α is the learning rate of the gradient method, It is a gradient operator that uses gradient descent and back propagation to update the parameters of the value network, so that the state value output by the value network is closer to the actual state value.

[0174] Step 8: Execute the second loop step to update the current policy network M times (M≤T), wherein the second loop step includes:

[0175] The state S of the interactive information t and action A t Input them into the policy network respectively, and use the importance sampling method to calculate the importance weight I t (θ); Calculate the advantage function based on the state value V and the discounted reward R Combined with the importance weight I t (θ) Find the objective function L of the current policy network CLIP (θ), use back propagation to update the parameters of the current policy network;

[0176] Among them, the importance weight I t The calculation method of (θ) is as follows:

[0177]

[0178] where π(·|s t ;θ)、π(·|s t θ old ) means that the state s t The policy distribution obtained by inputting the current policy network and the old policy network, θ represents the parameters of the current policy network, θ old represents the parameters of the old policy network, π(a t |s t ;θ),π(a t |s t θ old ) indicates the actual action to be taken t Input into the corresponding strategy distribution to obtain the value of the action under the corresponding strategy distribution.

[0179] Calculate advantage function based on state value and discounted reward Combined with the importance weight I t (θ) Find the objective function L of the current policy network CLIP (θ), including:

[0180] According to the state value V t and discount reward R t Calculate the advantage function for:

[0181]

[0182] According to the advantage function and importance weight I t (θ), find the objective function L of the current policy network CLIP (θ):

[0183]

[0184] Where ε is the restricted importance weight I t (θ) is a hyperparameter, clip(I t (θ), 1-ε, 1+ε) will be the importance weight I t The value of (θ) is limited to the range of (1-ε, 1+ε). t When (θ)<1-ε, let I t (θ)=1-ε, similarly when I t When (θ)>1+ε, let I t (θ) = 1 + ε;

[0185] Based on the objective function L of the current policy network CLIP (θ), and use back-propagation to update the parameters θ of the current policy network.

[0186] Step 9: Update the old policy network θ using the updated current policy network parameters θ old , thus completing one iteration.

[0187] θ old ←θ

[0188] Step 10: Repeat steps 1 to 9 until N iterations are reached, and finally the entire training is completed to obtain the trained deep reinforcement learning model.

[0189] Finally, this implementation case conducted a comparative analysis from the three performance indicators of average vehicle speed, network throughput, and number of collision risks, which showed that the method of the present invention can improve the average vehicle speed and the throughput of the traffic network in the continuous merging and diversion bottleneck traffic scenario, and reduce the number of collision risks.

[0190] The above specific implementations are only used to illustrate the technical solution of the present invention, but not to limit it. Although the present invention is described in detail with reference to the above specific implementations, those skilled in the art should understand that the technical solution of the present invention can be modified without departing from the scope of the technical solution of the present invention, and should be covered within the scope of the claims of the present invention.

Claims

1. A method for dispatching an autonomous driving vehicle in a continuous merging and diverting bottleneck traffic scenario, characterized by: The following steps are involved: S1. Considering the multi-lane problem, the road network modeling is carried out for the continuous merge-diverge bottleneck traffic scenario, specifically: A continuous merging and diverging bottleneck traffic scenario was built under the traffic reinforcement learning framework Flow and combined with the microscopic traffic simulation software SUMO, in which the traffic on the left four-lane main road and the traffic on the two entrance ramps will merge at the road intersection, and then the vehicles will diverge after driving a certain distance on the four-lane main road after the merging point. A continuous merging and diverging bottleneck traffic scenario with a four-lane main road, two entrance ramps and two exit ramps was constructed. S2. Construction of autonomous driving vehicle control model based on reinforcement learning, specifically: (1) State space design For an autonomous driving vehicle, the state space s of each autonomous driving vehicle is represented by using the current local state information of the agent, including: the longitudinal speed of the current autonomous driving vehicle, the relative speeds between the vehicle in front of it and the vehicle in front of it in the same lane, the vehicle in the rear of it in the same lane, the vehicle in front of it in the left adjacent lane, the vehicle in front of it in the right adjacent lane, and the vehicle in rear of it in the right adjacent lane, and the relative distances between the vehicle in front of it and the vehicle in front of it in the same lane, the vehicle in front of it in the left adjacent lane, the vehicle in rear of it in the left adjacent lane, the vehicle in front of it in the right adjacent lane, and the vehicle in rear of it in the right adjacent lane. (2) Action space design Two actions are designed, namely the acceleration and lane-changing action of the autonomous driving vehicle. (3) Reward function design The reward function R at time step t t for: R t =ω1r1+ω2r2 Where r1 is the system-level speed reward function, r2 is the headway penalty reward function, ω1 and ω2 are the weight coefficients of the system-level speed reward function and the headway penalty reward function, respectively. S3. Build a deep reinforcement learning model for autonomous vehicle control The deep reinforcement learning model includes a current policy network, an old policy network, and a value network. Based on the obtained state space information, a deep reinforcement learning model for the autonomous driving vehicle scheduling method is established. The input of the model is the joint state space of the autonomous driving vehicle, and the output is the probability distribution of acceleration actions and lane change actions. The specific action is sampled and selected based on the probability distribution of the action. The deep reinforcement learning model for the autonomous driving vehicle scheduling method is trained by combining deep learning and reinforcement learning, and the overall cycle steps are repeated until the predetermined conditions are met, and finally a trained deep reinforcement learning model is obtained.

2. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 1, characterized in that: The overall cycle steps include: (1) Execute the first loop step to obtain each step of interaction information between the autonomous driving vehicle and the environment (S t , A t , R t ), wherein the first cycle step comprises: (a) according to S2 in claim 1, the state space formed by the interaction between the autonomous driving vehicle and the environment is obtained, and the local state spaces of all intelligent agents are combined to form a global state space S t ; (b) The joint state space S t Input into the current deep reinforcement learning model to obtain the joint action space A of the autonomous driving vehicle t , calculate the reward value R t , forming the mutual information of the corresponding step (S t , A t , R t ) and the joint action space A t The information is sent to the corresponding agent, and the output information is the probability distribution of the action. The specific action is sampled and selected according to the probability distribution of the action. (c) The autonomous vehicle is based on the joint action space A t Select and execute the corresponding action so that the environment jumps to the next joint state S t+1 ; (2) After a round, the interaction information (S t , A t , R t ) are input into the value network to obtain the state value V = [V1, V2, ..., V T ]. Starting from the last step at the end of the round, we calculate the discounted reward G = [G1, G2, ..., G T ], whose formula is G t =R t +γG t+1 , where γ is the discount factor; (3) Calculate the loss function L(ω) of the value network based on the mean square error between the state value and the discounted reward, and update the value network parameters using gradient descent and back propagation; (4) Execute the second loop step to update the current policy network M times (M≤T), wherein the second loop step includes: The state S of the interactive information t and action A t Input them into the policy network respectively, and use the importance sampling method to calculate the importance weight I t (θ); Calculate the advantage function based on the state value V and the discounted reward R Combined with the importance weight I t (θ) Find the objective function L of the current policy network CLIP (θ), use back propagation to update the parameters of the current policy network; (5) Use the updated parameters of the current policy network to update the parameters of the old policy network.

3. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 1, characterized in that: The joint state space S of S2 t Specifically: S t =(s1,s2,...,s n ) Among them, s i,t represents the state space of the i-th autonomous driving vehicle, S t Represents the global state space formed by combining the local state spaces of all agents. i,t represents the longitudinal velocity of the autonomous vehicle i, Respectively represent the relative speeds between vehicle i and the preceding vehicle in the same lane, the following vehicle in the same lane, the preceding vehicle in the left adjacent lane, the following vehicle in the left adjacent lane, the preceding vehicle in the right adjacent lane, and the following vehicle in the right adjacent lane; They respectively represent the relative distances between vehicle i and the preceding vehicle in the same lane, the following vehicle in the same lane, the preceding vehicle in the left adjacent lane, the following vehicle in the left adjacent lane, the preceding vehicle in the right adjacent lane, and the following vehicle in the right adjacent lane.

4. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 1, characterized in that: The joint action space A of S2 t Specifically: <h2 style=";text-align:left;direction:ltr">A<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (a1, a2,..., a)<h2 style=";text-align:left;direction:ltr"> n <h2 style=";text-align:left;direction:ltr"> ) a i,t =(acc i,t ,l i,t ) where acc i,t ∈[-4.5,2.6]m / s 2 , l i,t ∈{-1, 0, 1} Among them, a i,t represents the action space of the i-th autonomous driving vehicle, A t represents the set of action spaces of all agents. i,t represents the acceleration of the i-th vehicle; l i，t Represents the lane changing action of the i-th vehicle, where -1, 0, and 1 represent changing lanes to the left, going straight, and changing lanes to the right, respectively. Lane change action i，t is obtained by converting three continuous values ​​x1, x2, and x3 through the Softmax operation. First, the exponential function e of the three values ​​is calculated. x1 , e x2 , e x3 , and then apply the Softmax function to convert these values ​​into a probability distribution. where i∈{1, 2, 3} The formula for expanding each component is: Where p1, p2, and p3 represent the probabilities of changing lanes to the left, not changing lanes, and changing lanes to the right, respectively. The agent chooses its lane-changing action l based on these probabilities. i，t .

5. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 1, characterized in that: The reward function R of S2 t Specifically: R t =ω1r1+ω2r2 1) System-level speed reward r1: Calculates the system-level speed reward, which aims to encourage all vehicles to have a speed as close to the desired speed as possible. The maximum cost max_cost is the theoretical maximum deviation value, which is used as the benchmark value for normalization; the current cost cur_cost is the current deviation value of the system, which is used to measure the current system performance; ∈ is a very small value to avoid division by zero errors. The reward is normalized to the interval [0, 1], where 1 means that all vehicles fully reach the desired speed. 2) Headway penalty r2: for vehicles with a headway less than the minimum headway t min This penalty mechanism encourages vehicles to maintain a reasonable safety distance, thereby improving the safety and stability of the overall traffic system.

6. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 2, characterized in that: Discount RewardsG t It is defined as the weighted sum of all rewards from time t onwards in a Markov decision process, and is calculated as follows: The calculation formula for discount rewards is: G t =R t +γG t+1 Therefore, according to the discounted reward G at the last T steps T =R T Step forward and calculate the discounted reward G for each step t : Where γ is the discount factor, which reflects the relative importance of future rewards at the current moment. If it is close to 0, it indicates that only the rewards at the current moment are evaluated. When it is close to 1, it indicates that long-term rewards are also considered.

7. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 2, characterized in that: The loss function L(ω) of the value network is calculated as follows: L(ω)=MSE(G t -V t ) Where MSE is the mean square error function, G t is the discounted reward at step t, V t is the state value of step t; Based on L(ω), the gradient descent method is used to update the parameter ω of the value network: Among them, α is the learning rate of the gradient method, It is a gradient operator that uses gradient descent and back propagation to update the parameters of the value network, so that the state value output by the value network is closer to the actual state value.

8. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 2, characterized in that: Importance Weight I t The calculation method of (θ) is as follows: where π(·|s t ;θ)、π(·|s t θ old ) means that the state s t The policy distribution obtained by inputting the current policy network and the old policy network, θ represents the parameters of the current policy network, θ old represents the parameters of the old policy network, π(a t |s t ;θ),π(a t |s t θ old ) indicates the actual action to be taken t Input into the corresponding strategy distribution to obtain the value of the action under the corresponding strategy distribution.

9. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 2, characterized in that: Calculate advantage function based on state value and discounted reward Combined with the importance weight I t (θ) Find the objective function L of the current policy network CLIP (θ), including: According to the state value V t and discount reward R t Calculate the advantage function for: According to the advantage function and importance weight I t (θ), find the objective function L of the current policy network CLIP (θ): Where ε is the restricted importance weight I t The hyperparameters of (θ), clip(I t (θ), 1-ε, 1+ε) will be the importance weight I t The value of (θ) is limited to the range of (1-ε, 1+ε). t When (θ)<1-ε, let I t (θ)=1-ε, similarly when I t When (θ)>1+ε, let I t (θ) = 1 + ε; Based on the objective function L of the current policy network CLIP (θ), and use back-propagation to update the parameters θ of the current policy network.

10. The method for dispatching an autonomous driving vehicle in a continuous merging and diverging bottleneck traffic scenario according to claim 2, characterized in that: Update the old policy network θ using the updated current policy network parameters θ old , thus completing one iteration. i old ←θ。

Images