Bottleneck area lane changing control method, device and medium based on deep reinforcement learning
By employing deep reinforcement learning to observe overall vehicle density and select coordinated lane-changing actions, the method addresses reduced traffic capacity in bottleneck areas, stabilizing flow and improving efficiency.
Patent Information
- Application Number
- CN202310808834.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-03
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2043-07-03
AI Technical Summary
Existing lane-changing control schemes based on deep reinforcement learning primarily focus on individual vehicle behavior without considering the macroscopic impact on overall traffic flow, making it difficult to effectively address reduced traffic capacity in bottleneck areas.
A method that divides the road network into lane-changing and bottleneck areas, using deep reinforcement learning to observe overall vehicle density and select lane-changing actions through an ε-greedy strategy, integrating experiences to train an agent for optimal lane-changing control, and employing a reward function to stabilize traffic flow.
This approach coordinates lane-changing behaviors to reduce concentrated lane changes, stabilizes traffic flow, and enhances traffic efficiency by minimizing capacity reduction in bottleneck areas.
Smart Images
Figure CN117079450B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of lane changing control, and in particular to a bottleneck area lane changing control method, device and medium based on deep reinforcement learning. Background Art
[0002] Traffic congestion often occurs in bottleneck areas where the number of available lanes upstream and downstream is different due to traffic accidents, road construction and maintenance, and route design. The bottleneck area is the section with the lowest traffic capacity, that is, the section with the least number of vehicles that can pass per unit time. In order to smoothly merge into the downstream of the bottleneck, the vehicles upstream are forced to change lanes, causing the traffic speed of the rear and adjacent lanes to decrease, the vehicles to "queue", and the density upstream to increase. When the density upstream of the bottleneck is greater than the critical density, the bottleneck "capacity drop" phenomenon will occur, that is, the traffic volume in the bottleneck area will suddenly decrease with the increase of the upstream density. The decrease in traffic capacity causes the traffic flow at the bottleneck to be extremely unstable, further aggravating the degree of traffic congestion, increasing travel time, and reducing road traffic efficiency and safety. In addition, the flow in the bottleneck area determines the traffic capacity of the entire road. Therefore, it is very important to solve the problem of decreased traffic capacity in the bottleneck area.
[0003] In order to solve the problem of reduced traffic capacity in the bottleneck area caused by the forced lane change behavior of vehicles near the bottleneck, the number of vehicles that change lanes near the bottleneck area can be reduced by making the vehicles upstream of the bottleneck area change lanes in advance while ensuring smooth traffic, so as to minimize the interference of vehicles entering the bottleneck area and reduce the impact of reduced traffic capacity. In addition, it is also necessary to limit the frequent lane change behavior of vehicles in pursuit of higher driving speeds, which will aggravate the chaos of traffic flow. However, how to achieve lane change control for specific vehicles? The emergence of vehicle networking technology makes this control strategy possible. In this network, the traffic environment information and vehicle driving information required for control decisions can be obtained based on on-board sensors, and various communication technologies can be used to realize information exchange between people and vehicles, vehicles and vehicles, vehicles and road test units, etc., so that the control unit can make better control decisions. Compared with the traditional method of obtaining detection data by establishing fixed facilities, the data collected by this technology is more accurate and comprehensive. Based on these data, it is helpful to make more appropriate decisions. In addition, the emergence of unmanned vehicle technology better demonstrates the performance of control technology. Compared with human-driven vehicles, it is not restricted or interfered by the driver's driving skills and can execute control instructions more accurately and timely.
[0004] The traditional lane-changing control strategy is to establish a lane-changing decision rule base to determine whether the vehicle meets certain specific lane-changing rules, and then decide whether the vehicle needs to change lanes. This method is highly interpretable and simple in structure, and is more commonly used. However, this solution is only feasible for fixed bottleneck areas caused by route design, and is not feasible for dealing with occasional bottleneck areas caused by traffic accidents. This is because the rule-based decision algorithm can only deal with driving scenarios in the rule base, and cannot deal with changes and uncertainties in traffic scenarios.
[0005] As a milestone breakthrough in the field of reinforcement learning, deep reinforcement learning provides a solution to this drawback of traditional lane-changing control. The deep reinforcement learning algorithm is a model-free algorithm that combines the perception ability of deep learning and the decision-making ability of reinforcement learning. It can cope with traffic environments with a large amount of dynamic and uncertain information and make appropriate decisions in complex environments. Nowadays, using this algorithm to solve intelligent traffic problems is a major research hotspot. It is widely used in strategies such as traffic signal control and ramp control, and significantly optimizes the performance of these control technologies. However, there are currently few control schemes based on this algorithm to solve forced lane-changing behavior, and most of them only consider the lane-changing control of a single vehicle at the microscopic level, without considering the impact of lane-changing control on the entire traffic flow from a macroscopic perspective. The effectiveness of the lane-changing control scheme based on deep reinforcement learning has not been effectively exerted, and it is difficult to effectively solve the problem of reduced traffic capacity in bottleneck areas. Summary of the invention
[0006] In response to the problems raised by the background technology, the purpose of the present invention is to provide a lane changing control method, device and medium in the bottleneck area based on deep reinforcement learning, which solves the problem that the lane changing control scheme based on deep reinforcement learning only considers the lane changing control of a single vehicle at the micro level, but does not consider the impact of lane changing control on the entire traffic flow from a macro perspective, making it difficult to effectively solve the problem of reduced traffic capacity in the bottleneck area.
[0007] The present invention is achieved through the following technical solutions:
[0008] The first aspect of the present invention provides a bottleneck area lane change control method based on deep reinforcement learning, comprising:
[0009] S1. Divide the road network into lane-changing areas and bottleneck areas in the direction of vehicle travel, set the lane-changing areas as the environment, observe the environment, and obtain the overall state s of the environment at time t t ;
[0010] S2. Construct an agent that determines the overall state s through an ε-greedy strategy. t The lane-changing action a at time t t ;
[0011] S3, the agent executes the lane changing action at , and obtain the lane change action a t The reward r at time t is determined based on the vehicle density of each lane. t , and obtain the overall state s of the environment at time t+1 t+1 ;
[0012] S4, the overall state s t , Lane changing action a t , Reward t , overall status t+1 Integrate into experience and store in an experience replay pool, and use the experience in the experience replay pool to train the agent;
[0013] S5. Repeat the environment and agent interactive training of steps S1 to S4 until the maximum number of iterations is reached. At this time, the agent obtains a Q value table of the optimal lane change control action.
[0014] In the above technical scheme, the road network is divided into a lane changing area and a bottleneck area along the vehicle driving direction, and the lane changing area is set as the environment. By observing the overall state of the two areas upstream of the bottleneck area, a corresponding lane changing action is selected through the overall state combined with the ε-greedy strategy. Compared with the prior art, the present invention considers the impact of the overall state of the lane changing area on the entire traffic flow from a macro perspective by observing the overall vehicle density in the non-control area and the control area, thereby coordinating the lane changing behavior of all vehicles in the lane changing area.
[0015] After the agent determines the lane-changing action, it acts on the environment. The environment then receives a reward and the overall state of the environment at the next moment after executing the lane-changing action. The overall state s t , Lane changing action a t and the reward r at time t obtained from the environment t and the overall state s at time t+1 t+1 The experience in the experience replay pool is integrated into the experience replay pool, and the experience in the experience replay pool is used to train the agent. By repeating the environment and agent interactive training from step S1 to step S4 until the maximum number of iterations is reached, through multiple iterations, the agent can learn a Q value table that can guide it to select the optimal lane change control action under various conditions, that is, the optimal lane change control strategy. The optimal lane change strategy can coordinate and control the lane change behavior of vehicles in the bottleneck area, and avoid traffic congestion caused by concentrated lane changes of vehicles near the bottleneck area by changing lanes in advance upstream of the bottleneck area, effectively solving the problem of reduced traffic capacity in the bottleneck area.
[0016] In a possible embodiment, the lane change area includes four lanes, and the four lanes are numbered 0, 1, 2, and 3 from right to left.
[0017] In a possible embodiment, the environment is observed and the overall state s of the environment at time t is obtained. t Specifically include:
[0018] Get the vehicle density of each lane in the lane change area at time t respectively, and use vector S i,j (t) represents the vehicle density of the jth lane in the i-th zone at time t;
[0019] Wherein, the lane-changing area includes a non-control area and a control area, i=0 represents a non-control area, and i=1 represents a control area;
[0020] At this time, the overall state of the environment at time t s t Represented as s t ={S 0,0 (t),S 0,1 (t),…,S 1,2 (t),S 1,3 (t)}.
[0021] In one possible embodiment, the agent determines the overall state s by an ε-greedy strategy t The lane-changing action a at time t t Specifically include:
[0022] Constructing a lane change ratio value, wherein the lane change ratio values include 0, 0.1, 0.2, and 0.3;
[0023] Using vector M k (t) represents the lane-changing action of the lane participating in the lane-changing lane at the kth lane-changing ratio value at time t;
[0024] At this time, the overall state s t The lane-changing action a at time t t Represented as a t ={M0(t),M1(t),M2(t),M3(t)}.
[0025] In a possible embodiment, the lanes participating in the lane change include lane 0, lane 1, and lane 2.
[0026] In a possible embodiment, the agent performs the lane change action a t , and obtain the lane change action a t The vehicle density of each lane includes:
[0027] Deploy a roadside unit, and collect the traffic volume and average speed of each lane at time t through the roadside unit;
[0028] The vehicle flow and the average vehicle speed are calculated to obtain the vehicle density of each lane at time t.
[0029] In a possible embodiment, the reward r at time t is determined according to the vehicle density of each lane. t Previously included:
[0030] Construct a reward function at time t, where the reward function at time t is constructed as follows:
[0031] r n (t)=|ΔD(t-1)|-|ΔD(t)|
[0032] Among them, r n (t) is the reward function at time t, ΔD(t) is the density difference at time t, and ΔD(t-1) is the density difference at time t-1.
[0033] In a possible embodiment, the density difference at time t is expressed as follows:
[0034]
[0035] Among them, d j represents the vehicle density of the jth lane, represents the average lane density.
[0036] A second aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a bottleneck area lane changing control method based on deep reinforcement learning is implemented.
[0037] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a bottleneck area lane changing control method based on deep reinforcement learning.
[0038] Compared with the prior art, this application has the following advantages and beneficial effects:
[0039] The present invention uses a deep reinforcement learning algorithm to coordinate and control the lane-changing behavior of vehicles in the bottleneck area. By selecting a certain proportion of vehicles to change lanes in advance, it avoids the concentrated lane-changing of vehicles near the bottleneck area, which leads to a decrease in traffic capacity. The design of the reward function achieves the average density of vehicles in each lane, so as to avoid concentrated lane-changing and frequent lane-changing behaviors in pursuit of speed to the greatest extent, and ensure the stability of traffic flow in the bottleneck area. In addition, through the interactive update training of the DQN network and the environment, compared with the traditional traffic flow control scheme, the average travel time of vehicles is reduced and the traffic efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:
[0041] Figure 1 A schematic flow chart of a bottleneck lane changing control method based on deep reinforcement learning provided in Example 1 of the present invention;
[0042] Figure 2 A schematic diagram of the architecture of a lane-changing control method in a bottleneck area based on deep reinforcement learning provided in Embodiment 1 of the present invention;
[0043] Figure 3 A schematic diagram of the structure of an electronic device provided in Example 2 of the present invention. DETAILED DESCRIPTION
[0044] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.
[0045] Example 1
[0046] Figure 1 A flowchart of a lane changing control method in a bottleneck area based on deep reinforcement learning provided in Example 1 of the present invention is shown in FIG. Figure 1 As shown, the bottleneck area lane changing control method based on deep reinforcement learning includes:
[0047] Step S1: divide the road network into a lane-changing area and a bottleneck area in sequence along the vehicle travel direction, set the lane-changing area as an environment, observe the environment, and obtain the overall state s of the environment at time t t .
[0048] It should be noted that the bottleneck area is the point with the lowest road capacity and is a frequent area of traffic congestion. When congestion occurs, the traffic density increases, resulting in insufficient driving space. However, due to the particularity of the bottleneck area structure, vehicles in the narrowed lanes must change lanes to pass through the bottleneck area and merge downstream. This forced lane change behavior causes the speed of the rear vehicles and vehicles in the adjacent lanes to decrease. Vehicles in the bottleneck area queue up and continue to spread upstream. Therefore, by making the vehicles upstream of the bottleneck area change lanes in advance, the number of vehicles that change lanes near the bottleneck area can be reduced, and the interference of vehicles merging into the bottleneck area can be minimized to solve the problem of reduced capacity.
[0049] Therefore, the present invention divides the road network into a lane changing area and a bottleneck area along the vehicle driving direction, and sets the lane changing area as the environment. By observing and analyzing the environment as a whole, the vehicle is controlled to change lanes in advance in the lane changing area upstream of the bottleneck area, so as to solve the problem of reduced traffic capacity in the bottleneck area caused by forced lane changing of vehicles near the bottleneck.
[0050] In a possible embodiment, the lane change area includes four lanes, and the four lanes are numbered 0, 1, 2, and 3 from right to left.
[0051] like Figure 2 As shown, in the embodiment of the present invention, the bottleneck area where four lanes become three lanes is taken as an example, and the four lanes are numbered 0, 1, 2, and 3 from right to left. The lane numbered 0 is the rightmost narrowing lane among the three lanes on the four sides.
[0052] Specifically, observe the environment and obtain the overall state s of the environment at time t t Specifically include:
[0053] Get the vehicle density of each lane in the lane change area at time t respectively, and use vector S i,j (t) represents the vehicle density of the jth lane in the i-th zone at time t.
[0054] The lane-changing area includes a non-controlled area and a controlled area, i=0 represents the non-controlled area, and i=1 represents the controlled area.
[0055] At this time, the overall state of the environment at time t s t Represented as s t ={S 0,0 (t),S 0,1 (t),…,S 1,2 (t),S 1,3 (t)}.
[0056] It should be noted that the present invention divides the lane change area into a non-controlled area and a controlled area, wherein the non-controlled area and the lane change control area are both upstream of the bottleneck area, and the lane change control area is located in the section near the bottleneck area. i,j (t) represents the vehicle density of the four lanes in the non-controlled area and the controlled area respectively.
[0057] It should be noted that the prior art usually adopts local observation of the state of each vehicle, and only considers the lane changing control of a single vehicle at the micro level, without considering the impact of lane changing control on the entire traffic flow from a macro perspective. The present invention observes the overall state of the two areas upstream of the bottleneck area, and selects a corresponding lane changing action based on the overall state combined with the action selection strategy. Compared with the prior art, the present invention observes the overall vehicle density in the non-control area and the control area, and considers the impact of the overall state of the lane changing area on the entire traffic flow from a macro perspective, thereby coordinating the lane changing behavior of all vehicles in the lane changing area.
[0058] Step S2: construct an agent that determines the overall state s through an ε-greedy strategy t The lane-changing action a at time t t .
[0059] The agent is the role of the learner and decision maker in the DQN network. In the present invention, the agent is used to interact with the environment based on lane changing actions, vehicle density in each lane, and rewards. Figure 2 As shown, the agent in the present invention observes the state of the environment and selects the execution action through the ε-greedy strategy. After executing the action, the agent can get a reward, and the environment changes, and the agent will also get a new state. This is repeated until the termination condition is reached (for example, the maximum number of iterations is reached or the target state is reached). Finally, the agent learns a Q value table that can guide it to choose the best action in various states, that is, the optimal strategy.
[0060] The ε-greedy strategy is the most commonly used strategy for balancing development and exploration in reinforcement learning. It means that when the agent makes a decision, there is a small positive number ε (ε<1) probability of randomly selecting an unknown action, and the remaining probability of selecting the action with the largest action value among the existing actions. And it is guaranteed that each action value space is visited at least once. Therefore, in the embodiment of the present invention, the ε-greedy strategy is used to determine the lane change action, that is, to determine a certain proportion of vehicles to change lanes in the upstream area of the bottleneck area to ensure that each action will be tried.
[0061] Specifically, the agent determines the overall state s through the ε-greedy strategy t The lane-changing action a at time t t Specifically include:
[0062] A lane change ratio value is constructed, where the lane change ratio values include: 0, 0.1, 0.2, and 0.3.
[0063] Using vector M k (t) represents the lane changing action at the kth lane changing ratio value at time t.
[0064] At this time, the overall state s tThe lane-changing action a at time t t Represented as a t ={M0(t),M1(t),M2(t),M3(t)}.
[0065] The lane change ratio value indicates that the vehicles randomly selected with the lane change ratio value change lanes to the left, and the other vehicles keep driving in the original lanes. For example, 0.2 indicates that 20% of the vehicles randomly selected change lanes to the left, and the other vehicles keep driving in the original lanes.
[0066] It should be noted that the embodiment of the present invention uses the ε-greedy strategy to determine the action, that is, to determine and select a certain proportion of vehicles to change lanes in advance. Therefore, in the present invention, by constructing n lane change ratio values, the lane change action of each road under n lane change ratio values is determined. According to the characteristics of driving into the bottleneck area, it can be known that the narrowing lane needs to change lanes to the left in order to smoothly blend into the downstream, and the density of its adjacent lanes becomes larger, so it will change lanes to the left again to reduce the density and avoid congestion. Therefore, only the control behaviors of changing lanes to the left and maintaining the current lane need to be considered, and the leftmost lane does not participate in the lane change control. Therefore, in the present invention, only three lanes, namely Lane 0, Lane 1 and Lane 2, need to make lane change decisions, and these three lanes have four situations of changing lanes and not changing lanes with four lane change ratio values. Therefore, there are a total of 4 in the embodiment of the present invention. 3 = 64 actions. The lane-changing action at the second lane-changing ratio at time t is recorded as vector M2(t), which represents the overall state s t At time t, 20% of the vehicles are randomly selected to change lanes to the left, while the remaining vehicles keep driving in their original lanes.
[0067] Step S3: The agent performs the lane changing action a t , and obtain the lane change action a t The reward r at time t is determined based on the vehicle density of each lane. t , and obtain the overall state s of the environment at time t+1 t+1 .
[0068] The agent performs the lane changing action a t It means selecting a certain proportion of vehicles to change lanes to the left. At this time, the four lanes have completed a lane change. At this time, the vehicle density of each lane after the lane change action is obtained to calculate the reward at time t.
[0069] Specifically, the agent executes the lane changing action at, and obtains the vehicle density of each lane after executing the lane changing action at specifically includes:
[0070] Roadside units are deployed to collect the traffic volume and average speed of each lane at time t.
[0071] The vehicle flow and the average vehicle speed are calculated to obtain the vehicle density of each lane at time t.
[0072] In a possible embodiment, in the present invention, vehicle density=vehicle flow / average vehicle speed.
[0073] It should be noted that the goal of the present invention is to solve the problem of reduced traffic capacity caused by the concentration of vehicles near the bottleneck area and frequent lane changes. Generally, lane changes are performed in the following situations:
[0074] (1) The driver drives in pursuit of higher speed;
[0075] (2) Routing problem, that is, the vehicle must change lanes to enter the next road in its route.
[0076] Since the reward function needs to be designed based on the optimization goal, and the optimization goal of the present invention is to reduce the density difference between lanes, reducing the density difference between lanes can reduce the number of lane changes and avoid concentrated lane changes, avoid concentrated lane changes that are prone to congestion and frequent lane changes in pursuit of speed, thereby increasing the traffic flow in the bottleneck area and solving the problem of reduced traffic capacity.
[0077] Therefore, in the embodiment of the present invention, the vehicle density of the lane is determined by the traffic volume and average speed of each lane, and the vehicle density is used as an indicator for constructing the reward function.
[0078] Specifically, the reward r at time t is determined according to the vehicle density of each lane. t Previously included:
[0079] Construct a reward function at time t, where the reward function at time t is constructed as follows:
[0080] r n (t)=|ΔD(t-1)|-|ΔD(t)|
[0081] Among them, r n (t) is the reward function at time t, ΔD(t) is the density difference at time t, and ΔD(t-1) is the density difference at time t-1.
[0082] Furthermore, the density difference at time t is expressed as follows:
[0083]
[0084] Among them, d j represents the vehicle density of the jth lane, represents the average lane density.
[0085] It should be noted that the problem of decreased capacity in the bottleneck area, that is, when the bottleneck density is greater than the critical density, the output flow of the bottleneck area suddenly decreases as the bottleneck density increases. Therefore, the solution to the problem of decreased capacity in the bottleneck is to reduce the vehicle density of the lane and increase its output flow. When the vehicle density of each lane is more average, the demand for lane changes will be less, the concentrated and frequent lane changes will be reduced, the traffic flow will be more stable, and the output flow of the bottleneck will be larger, thus avoiding the problem of decreased capacity. In the present invention, ΔD(t) represents the vehicle density of each lane at time t and its average lane density The smaller |ΔD(t)| is, the more even the vehicle density in each lane is, the more stable the traffic flow is, and therefore the greater the output traffic volume in the bottleneck area is.
[0086] In the present invention, the reward value at time t is determined by the above-mentioned reward function: if the vehicle density difference is small, a larger reward value is given to indicate that this is a better lane change control strategy; if the vehicle density difference is large, a smaller reward value is given to indicate that this is a worse lane change control strategy.
[0087] Step S4: The overall state s t , Lane changing action a t , Reward t , overall status t+1 The experience is integrated into the experience replay pool, and the experience in the experience replay pool is used to train the agent.
[0088] in, Figure 2 Schematic diagram of the architecture of the bottleneck lane changing control method based on deep reinforcement learning provided in Example 1 of the present invention. After the agent determines the lane changing action, it acts on the environment. After executing the lane changing action, the environment obtains a reward and the overall state of the environment at the next moment. The overall state s t , Lane changing action a t and the reward r at time t obtained from the environment t and the overall state s at time t+1 t+1 They are integrated into the experience replay pool, and the experience in the experience replay pool is used to train the agent.
[0089] Step S5, repeating the environment and agent interactive training from step S1 to step S4 until a maximum number of iterations is reached, at which point the agent obtains a Q value table of the optimal lane-changing control action.
[0090] Among them, by repeating the environment and agent interactive training from step S1 to step S4 until the maximum number of iterations is reached, through multiple iterations, the agent can learn a Q value table that can guide it to select the optimal lane changing control action under various conditions, that is, the desired optimal lane changing control strategy.
[0091] It should be noted that the maximum number of iterations can be set by those skilled in the art according to actual needs and is not further limited in the embodiments of the present invention.
[0092] It should be emphasized that the traditional lane-changing control strategy is to establish a lane-changing decision rule base to determine whether the vehicle meets certain specific lane-changing rules, and then decide whether the vehicle needs to change lanes. However, due to the rule-based decision algorithm, it can only deal with the driving scenarios in the rule base, and cannot deal with the changes and uncertainties of traffic scenarios. Therefore, it is not feasible to deal with occasional bottleneck areas caused by traffic accidents. The lane-changing control strategy in the present invention is to observe and calculate the overall state and overall vehicle density upstream of the bottleneck area, select a certain proportion of vehicles to change lanes in advance, and design a reward function to ensure that the vehicle density of each lane is in an average state, so as to avoid concentrated lane changes and frequent lane changes in pursuit of speed to the greatest extent, and then obtain an optimal lane-changing control strategy through interactive update training between the DQN network and the environment. Compared with the traditional traffic flow control scheme, the lane-changing control strategy of the present invention is more flexible and can not only deal with fixed traffic scenes, but also deal with constantly changing and uncertain traffic scenes.
[0093] In summary, the present invention has the following advantages and beneficial effects compared with the prior art:
[0094] The present invention uses a deep reinforcement learning algorithm to coordinate and control the lane-changing behavior of vehicles in the bottleneck area. By selecting a certain proportion of vehicles to change lanes in advance, it avoids the concentrated lane-changing of vehicles near the bottleneck area, which leads to a decrease in traffic capacity. The design of the reward function achieves the average density of vehicles in each lane, so as to avoid concentrated lane-changing and frequent lane-changing behaviors in pursuit of speed to the greatest extent, and ensure the stability of traffic flow in the bottleneck area. In addition, through the interactive update training of the DQN network and the environment, compared with the traditional traffic flow control scheme, the average travel time of vehicles is reduced and the traffic efficiency is improved.
[0095] Example 2
[0096] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 2 of the present invention, such as Figure 3 As shown, the electronic device includes a processor 21, a memory 22, an input device 23 and an output device 24; the number of processors 21 in the computer device can be one or more. Figure 3 A processor 21 is taken as an example; the processor 21, the memory 22, the input device 23 and the output device 24 in the electronic device can be connected by a bus or other means. Figure 3 The example of connecting through bus is taken in the following.
[0097] The memory 22 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules. The processor 21 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 22, that is, the bottleneck area lane change control method based on deep reinforcement learning of Example 1 is implemented.
[0098] The memory 22 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0099] The input device 23 can be used to receive the ID and password input by the user, etc. The output device 24 is used to output the network configuration page.
[0100] Example 3
[0101] Embodiment 3 of the present invention further provides a computer-readable storage medium, wherein the computer-executable instructions, when executed by a computer processor, are used to implement the bottleneck area lane changing control method based on deep reinforcement learning as provided in Embodiment 1.
[0102] An embodiment of the present invention provides a storage medium containing computer executable instructions, and its computer executable instructions are not limited to the method operations provided in Example 1, and can also execute related operations in the bottleneck area lane changing control method based on deep reinforcement learning provided in any embodiment of the present invention.
[0103] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A bottleneck lane-changing control method based on deep reinforcement learning, characterized in that: include: S1. Divide the road network into lane-changing areas and bottleneck areas in the direction of vehicle travel, set the lane-changing areas as the environment, observe the environment, and obtain the overall state s of the environment at time t t ; The lane change area includes 4 lanes, which are numbered 0, 1, 2, and 3 from right to left; Observe the environment and obtain the overall state s of the environment at time t t Specifically include: Get the vehicle density of each lane in the lane change area at time t respectively, and use vector S i,j (t) represents the vehicle density of the jth lane in the i-th zone at time t; Wherein, the lane-changing area includes a non-control area and a control area, i=0 represents a non-control area, and i=1 represents a control area; At this time, the overall state of the environment at time t s t Represented as s t ={S 0,0 (t),S 0,1 (t),…,S 1,2 (t),S 1,3 (t)}; S2. Construct an agent that determines the overall state s through an ε-greedy strategy. t The lane-changing action a at time t t ; S3, the agent executes the lane changing action a t , and obtain the lane change action a t The reward r at time t is determined based on the vehicle density of each lane. t , and obtain the overall state s of the environment at time t+1 t+1 ; Among them, the reward r at time t is determined according to the vehicle density of each lane t Previously included: Construct a reward function at time t, where the reward function at time t is constructed as follows: r n (t)=|ΔD(t-1)|-|ΔD(t)| Among them, r n (t) is the reward function at time t, ΔD(t) is the density difference at time t, and ΔD(t-1) is the density difference at time t-1; The density difference at time t is expressed as follows: Among them, d j represents the vehicle density of the jth lane, represents the average lane density; S4, the overall state s t , Lane change action a t , Reward t , overall status t+1 Integrate into experience and store in an experience replay pool, and use the experience in the experience replay pool to train the agent; S5. Repeat the environment and agent interactive training of steps S1 to S4 until the maximum number of iterations is reached. At this time, the agent obtains a Q value table of the optimal lane change control action.
2. The bottleneck lane changing control method based on deep reinforcement learning according to claim 1, characterized in that: The agent determines the overall state s through the ε-greedy strategy t The lane-changing action a at time t t Specifically include: Constructing a lane change ratio value, wherein the lane change ratio values include 0, 0.1, 0.2, and 0.3; Using vector M k (t) represents the lane-changing action of the lane participating in the lane-changing lane at the kth lane-changing ratio value at time t; At this time, the overall state s t The lane-changing action a at time t t Represented as a t ={M0(t),M1(t),M2(t),M3(t)}.
3. The bottleneck lane changing control method based on deep reinforcement learning according to claim 2 is characterized in that: The lanes involved in the lane change include lane 0, lane 1 and lane 2.
4. The bottleneck lane changing control method based on deep reinforcement learning according to claim 1, characterized in that: The agent performs the lane change action a t , and obtain the lane change action a t The vehicle density of each lane includes: Deploy a roadside unit, and collect the traffic volume and average speed of each lane at time t through the roadside unit; The vehicle flow and the average vehicle speed are calculated to obtain the vehicle density of each lane at time t.
5. An electronic device, characterized in that It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the bottleneck area lane changing control method based on deep reinforcement learning as described in any one of claims 1 to 4 is implemented.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the bottleneck area lane changing control method based on deep reinforcement learning as described in any one of claims 1 to 4 is implemented.