Bottleneck area traffic flow optimization control method and system based on deep reinforcement learning

By building a deep reinforcement learning environment and agent on urban expressway sections, identifying and optimizing speed limit areas and lane configurations, traffic congestion problems of fixed bottlenecks and mobile bottlenecks are solved, and a comprehensive solution to traffic bottlenecks and road traffic efficiency is achieved. promote.

CN119005567APending Publication Date: 2024-11-22CHONGQING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410933708.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The prior art cannot effectively solve the traffic congestion problems caused by fixed bottlenecks and mobile bottlenecks at the same time, and the control solutions based on deep reinforcement learning are mostly targeted at one type of bottleneck.

Method used

By building environments and agents on urban expressway sections, a variable speed limit control strategy is determined in iterative training using deep reinforcement learning algorithms, and the speed limit area and lane configuration under fixed bottlenecks and mobile bottlenecks are identified and optimized.

Benefits of technology

A comprehensive solution to traffic bottlenecks has been achieved, and through refined speed limit control of sub-regional and lane, the comprehensive effect of road traffic efficiency and traffic management has been improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005567A_ABST
    Figure CN119005567A_ABST
Patent Text Reader

Abstract

The invention discloses a bottleneck area traffic flow optimization control method and system based on deep reinforcement learning, and the method achieves the collection of vehicle information on a road through the Internet of Vehicles, carries out the decision making through the deep reinforcement learning, finally obtains a variable speed limiting control strategy, and achieves the refined regional and lane-divided speed limiting control. The variable speed limit control mode is a universal method for solving the traffic bottleneck problem, the fixed bottleneck can be solved, the movable bottleneck can also be solved, and a comprehensive and effective solution is provided for traffic management.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of traffic flow optimization control, and in particular to a bottleneck area traffic flow optimization control method and system based on deep reinforcement learning. Background Art

[0002] Traffic congestion can lead to reduced road capacity, increased emissions, travel delays and safety risks. Traffic congestion can be divided into two main types: one is congestion caused by fixed bottlenecks, and the other is congestion caused by mobile bottlenecks.

[0003] Fixed bottleneck congestion often occurs in bottleneck areas where the number of available lanes upstream and downstream is different due to traffic accidents, road construction and maintenance, and route design. The bottleneck area is the section with the lowest traffic capacity, that is, the section with the least number of vehicles that can pass per unit time. In order to smoothly merge into the downstream of the bottleneck, the upstream vehicles are forced to change lanes, causing the traffic speed of the rear and adjacent lanes to decrease, the vehicles to "queue", and the upstream density to increase. When the density upstream of the bottleneck is greater than the critical density, the bottleneck "capacity drop (C city drop)" phenomenon will occur, that is, the traffic volume in the bottleneck area will suddenly decrease with the increase of the upstream density.

[0004] The mobile bottleneck on the expressway is a common traffic bottleneck. On the expressway, when a slow vehicle appears in the traffic flow, the vehicles behind it either need to find an opportunity to change lanes or need to slow down and follow the slow vehicle. The existence of these vehicles is equivalent to the existence of some mobile bottlenecks in the road network. When the traffic volume on the road is large, the deceleration and lane changing behaviors near the mobile bottleneck will intensify, and queuing behavior will occur upstream of the bottleneck, resulting in a decrease in the traffic efficiency of the road.

[0005] There are two mechanisms for applying VSL. The first is that VSL below the critical speed will cause the capacity of the basic graph to be lower than normal conditions. Therefore, applying VSL upstream of the bottleneck will permanently reduce the mainstream arrival flow, thereby avoiding bottleneck activation. This mechanism is mostly used for congestion caused by fixed bottlenecks.

[0006] The second is when a lower VSL is applied on a highway section below the critical density, a transition occurs to a new traffic state that provides the same flow at a lower speed and higher density. During this transition period, the increase in density causes a temporary reduction in mainstream flow. If the inflow of the congestion is lower than the outflow, the traffic congestion will gradually resolve. This mechanism is mostly used in mobile bottleneck scenarios.

[0007] The current method of using VSL to solve congestion only solves fixed bottlenecks or mobile bottlenecks alone, and there is no universal method that can solve both fixed bottlenecks and mobile bottlenecks.

[0008] As a milestone breakthrough in the field of reinforcement learning, deep reinforcement learning provides a solution to variable speed limit control. The deep reinforcement learning algorithm combines the perception ability of deep learning with the decision-making ability of reinforcement learning. It can cope with traffic environments with a large amount of dynamic and uncertain information and make appropriate decisions in complex environments. Nowadays, using this algorithm to solve intelligent transportation problems is a major research hotspot. It is widely used in strategies such as variable speed limit control and ramp control, and significantly optimizes the performance of these control technologies. However, there are currently few control schemes based on this algorithm to solve fixed bottlenecks and mobile bottlenecks, and most of the schemes only consider solving variable speed limit control for fixed bottlenecks or mobile bottlenecks, and there is no universal method that can solve both fixed bottlenecks and mobile bottlenecks. Summary of the invention

[0009] In response to the problems raised by the above-mentioned background technology, the purpose of the present invention is to provide a bottleneck area traffic flow optimization control method and system based on deep reinforcement learning. By constructing an environment and an intelligent agent on an urban expressway section, and utilizing interactive iterative training between the intelligent agent and the environment to respectively find variable speed limit control strategies under mobile bottlenecks and fixed bottlenecks, the problem that the prior art only considers one of the fixed bottlenecks or the mobile bottleneck, and its algorithm cannot be applied to both fixed bottlenecks and mobile bottlenecks at the same time is solved.

[0010] The present invention is achieved through the following technical solutions:

[0011] The first aspect of the present invention provides a bottleneck area traffic flow optimization control method based on deep reinforcement learning, comprising the following steps:

[0012] Step S1: Divide the road into N driving areas along the vehicle driving direction, determine the control area in the bottleneck scenario based on the N driving areas, set the control area as the environment, and obtain the traffic state s of the environment at time slot t t ;

[0013] Step S2: construct an intelligent agent, which determines the traffic state s through the ε-greedy strategy. t The next action a t , the action a t Used to change the location and length of the speed limit area;

[0014] Step S3: Obtain the environment in which the agent performs action a t Traffic status after t+1 , for the action at and the traffic status s t+1 Calculate and get reward r t ;

[0015] Step S4: Traffic status s t 、Action a t , Reward t , Traffic status t+1 The experience is integrated into the experience and stored in the experience replay pool, and the experience in the experience replay pool is used to iteratively train the agent.

[0016] In the above technical solution, the road is divided into N areas of equal length along the vehicle driving direction, and the areas are numbered from the 1st to the Nth driving areas from upstream to downstream. The control area is determined in the N driving areas and set as the environment in the deep reinforcement learning network of the present invention. A roadside unit (RSU) is deployed near the bottleneck area to obtain the vehicle and lane information in the lane change control area, and the traffic state s collected by the RSU at time slot t is converted into the traffic state s of the lane change control area. t Transmitted to the central controller for algorithm calculation.

[0017] Construct an agent that determines the traffic state s at time slot t through the ε-greedy strategy t The agent takes the action and applies it to the environment to interact with it. The environment moves from the current traffic state to the next traffic state s according to the action performed by the agent. t+1 ∈S, after the environment is transformed, the agent receives the reward r according to the new state and action t To evaluate the quality of the action, t , a t , r t ,s t+1 , done} is saved in the experience replay pool, and the experience in the experience replay pool is used to train the agent. The agent can continuously learn and optimize its behavior strategy by observing the traffic status in the environment, selecting actions and obtaining rewards, so as to obtain the appropriate speed limit value and determine the speed limit area and lane accordingly to cope with the congestion caused by fixed bottlenecks and mobile bottlenecks.

[0018] In an optional embodiment, determining the control area in the bottleneck scenario based on the N driving areas includes:

[0019] In the fixed bottleneck scenario, the Nth driving area is set as the bottleneck area, the N-1th driving area is set as the acceleration area, and the 1st driving area to the N-2th driving area are set as the control area;

[0020] In the mobile bottleneck scenario, the first driving area to the downstream of the congestion area are set as control areas;

[0021] The control area is provided with M speed limit areas. For any speed limit area z m There is a speed limit v m .

[0022] In an optional embodiment, the speed limit value v m satisfy:

[0023] v max ≥v m+1 ≥v m ≥v m-1 ≥v min ;

[0024] N ≥ z m-1 ≥z m ≥z m+1 ≥1.

[0025] In an optional embodiment, the traffic state s of the environment at time slot t is obtained. t include:

[0026] Construct a 2N+M+1 element vector, and use the 2N+M+1 element vector to record the vehicle speed p in the N driving areas n (t) and vehicle density d n (t), the current position z of the M speed limit areas m (t) and the traffic flow f(t) of the Nth driving area;

[0027] The traffic state s is represented by the 2N+M+1 element vector t , the traffic state s t It is expressed as follows:

[0028] s t ={p1(t), d1(t), ...p N (t), d N (t), z1(t), …z M (t), f(t)}.

[0029] In an optional embodiment, the agent determines the traffic state s by an ε-greedy strategy. t The next action a t , the action a t Methods for changing the location and length of speed limit zones include:

[0030] The agent determines M speed limit areas in the traffic state s through the ε-greedy strategy. t The action set {a1,a2,…,a M ,h},; where h is used to represent the speed classification, a M∈[-1,0,1];

[0031] The positions and lengths of the M speed limit areas are changed using the action set.

[0032] In an optional embodiment, using the action set to change the position and length of the M speed limit areas includes: m =z m +a m .

[0033] In an optional embodiment, for the action a t and the traffic status s t+1 Calculate and get reward r t include:

[0034] In a fixed bottleneck scenario, the vehicle density in the acceleration zone is obtained, and the vehicle density in the acceleration zone is compared with the traffic state s t+1 Perform joint calculations to obtain rewards in fixed bottleneck scenarios;

[0035] In the mobile bottleneck scenario, observe the number and length of congested areas under N driving areas, and use the number and length of congested areas to determine the reward in the mobile bottleneck scenario.

[0036] In an optional embodiment, in a fixed bottleneck scenario, the vehicle density in the acceleration zone is obtained, and the vehicle density in the acceleration zone is compared with the traffic state s t+1 Perform joint calculations to obtain rewards for fixed bottleneck scenarios, including:

[0037]

[0038] In the above formula, d acc (t) represents the vehicle density in the acceleration zone at time t, d max represents the maximum vehicle density, d ex represents the expected vehicle density, f(t) represents the traffic volume, and f min represents the minimum traffic flow, f max Indicates the maximum traffic volume.

[0039] In an optional embodiment, in a mobile bottleneck scenario, observing the number of congested areas in N driving areas, and using the number of congested areas to determine the reward in the mobile bottleneck scenario includes:

[0040]

[0041] In the above formula, l jam Represents the length of the congested area, and F is a constant.

[0042] The second aspect of the present invention provides a bottleneck area traffic flow optimization control system based on deep reinforcement learning, comprising:

[0043] An environment setting module, the environment setting module is used to divide the road into N driving areas along the vehicle driving direction, determine the control area in the bottleneck scenario based on the N driving areas, set the control area as the environment, and obtain the traffic state s of the environment at time slot t t ;

[0044] An action simulation module, the action simulation module is used to construct an intelligent agent, the intelligent agent determines the traffic state s through an ε-greedy strategy t The next action a t , the action a t Used to change the location and length of the speed limit area;

[0045] A reward calculation module is used to obtain the value of the environment when the agent performs action a. t Traffic status after t+1 , for the action a t and the traffic status s t+1 Calculate and get reward r t ;

[0046] Iterative training module, the iterative training module is used to convert the traffic state s t 、Action a t , Reward t , Traffic status t+1 The experience is integrated into the experience and stored in the experience replay pool, and the experience in the experience replay pool is used to iteratively train the agent.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] The present invention collects vehicle information on the road through the Internet of Vehicles, uses deep reinforcement learning to make decisions, and finally obtains a variable speed limit control strategy, realizing refined regional and lane speed limit control. This variable speed limit control method is a general method for solving traffic bottleneck problems. It can solve both fixed bottlenecks and mobile bottlenecks, providing a comprehensive and effective solution for traffic management. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0050] Figure 1 A schematic diagram of a flow chart of a bottleneck area traffic flow optimization control method based on deep reinforcement learning provided in Example 1 of the present invention;

[0051] Figure 2 A schematic diagram of the architecture of a bottleneck area traffic flow optimization control method based on deep reinforcement learning provided in Example 1 of the present invention;

[0052] Figure 3 A schematic diagram of the structure of a fixed bottleneck area provided in Example 1 of the present invention;

[0053] Figure 4 A schematic diagram of the structure of a mobile bottleneck area provided in Embodiment 1 of the present invention;

[0054] Figure 5 A schematic diagram of the structure of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0056] Example 1

[0057] Figure 1 A flow chart of a bottleneck area traffic flow optimization control method based on deep reinforcement learning provided in Example 1 of the present invention is shown in FIG. Figure 1 As shown in FIG, the bottleneck area traffic flow optimization control method based on deep reinforcement learning includes the following steps:

[0058] Step S1: Divide the road into N driving areas along the vehicle driving direction, determine the control area in the bottleneck scenario based on the N driving areas, set the control area as the environment, and obtain the traffic state s of the environment at time slot t t ;

[0059] Step S2: construct an intelligent agent, which determines the traffic state s through the ε-greedy strategy. t The next action a t , the action a t Used to change the location and length of the speed limit area;

[0060] Step S3: Obtain the environment in which the agent performs action a t Traffic status after t+1 , for the action a t and the traffic status s t+1 Calculate and get reward r t ;

[0061] Step S4: Traffic status s t 、Action a t , Reward t , Traffic status t+1 The experience is integrated into the experience and stored in the experience replay pool, and the experience in the experience replay pool is used to iteratively train the agent.

[0062] It should be noted that fixed bottlenecks usually result from limitations in road design and network configuration, while dynamic bottlenecks are caused by random events such as accidents or slow traffic. Due to the different characteristics of different traffic bottlenecks, the current variable speed limit control method can only solve fixed bottlenecks or mobile bottlenecks alone. Currently, it is impossible to use a general algorithm to solve both fixed bottlenecks and mobile bottlenecks at the same time.

[0063] Based on the above-mentioned defects, the present invention proposes a bottleneck area traffic flow optimization control method based on deep reinforcement learning. The core of the method is to use deep reinforcement learning to systematically learn the lanes and areas of the bottleneck, obtain appropriate speed limit values, and determine the speed limit areas and lanes accordingly to cope with the congestion caused by fixed bottlenecks and mobile bottlenecks.

[0064] Specifically, the road is divided into N areas of equal length along the vehicle driving direction, and the areas are numbered from the 1st to the Nth driving areas from upstream to downstream. The control area is determined in the N driving areas and set as the environment in the deep reinforcement learning network of the present invention. A roadside unit (RSU) is deployed near the bottleneck area to obtain the vehicle and lane information in the lane change control area, and the traffic state s collected by the RSU at time slot t is converted into the traffic state s of the lane change control area. t Transmitted to the central controller for algorithm calculation.

[0065] like Figure 2 As shown, an intelligent agent is constructed, which determines the traffic state s at time slot t through the ε-greedy strategy t The agent takes the action and applies it to the environment to interact with it. The environment moves from the current traffic state to the next traffic state s according to the action performed by the agent. t+1 ∈S, after the environment is transformed, the agent receives the reward r according to the new state and action t To evaluate the quality of the action, t , a t , r t ,s t+1 , done} is saved in the experience replay pool, and the experience in the experience replay pool is used to train the agent. The agent can continuously learn and optimize its behavior strategy by observing the traffic status in the environment, selecting actions and obtaining rewards, so as to obtain the appropriate speed limit value and determine the speed limit area and lane accordingly to cope with the congestion caused by fixed bottlenecks and mobile bottlenecks.

[0066] In an optional embodiment, determining the control area in the bottleneck scenario based on the N driving areas includes:

[0067] In the fixed bottleneck scenario, the Nth driving area is set as the bottleneck area, the N-1th driving area is set as the acceleration area, and the 1st driving area to the N-2th driving area are set as the control area;

[0068] In the mobile bottleneck scenario, the first driving area to the downstream of the congestion area are set as control areas;

[0069] The control area is provided with M speed limit areas. For any speed limit area z m There is a speed limit v m .

[0070] It should be noted that the structure of the fixed bottleneck scenario is different from that of the mobile bottleneck scenario, and the subsequent reward value calculation will also be different. Therefore, the division of the driving area in the fixed bottleneck scenario and the mobile bottleneck scenario needs to be clear. Specifically:

[0071] Figure 3 The structural diagram of the fixed bottleneck area provided in Example 1 of the present invention is as follows: Figure 3 As shown, in a fixed bottleneck scenario, the Nth driving area is the bottleneck area, the N-1th driving area is the acceleration area, and the 1st driving area to the N-2nd driving area are the control area. Roadside units (RSUs) are deployed in the bottleneck area, and the 1st driving area to the N-2nd driving area are observed for the environment constructed by the embodiment of the present invention and the vehicle density in the acceleration area for subsequent reward calculation.

[0072] Figure 4 The schematic diagram of the structure of the mobile bottleneck area provided in Embodiment 1 of the present invention is as follows: Figure 4 As shown in the figure, a moving bottleneck refers to a dynamic congestion area formed when some vehicles are slow in the traffic flow, causing the following vehicles to be forced to slow down (equivalent to a congestion formed by a group of slow-moving vehicles, so it is called a moving bottleneck). The bottleneck area is uncertain. Therefore, in the embodiment of the present invention, the first driving area to the downstream of the congestion area are set as control areas, and the length and number of congestion areas are used as conditions for calculating the reward value.

[0073] There are M speed levels along the vehicle's driving direction, corresponding to M speed limits, and corresponding to M speed limit areas. The vehicle's driving direction is from right to left, and the L lanes are numbered from lane 1 to lane L. At this time, the road is divided into M*L areas, and the smallest unit in the N driving areas is one of the M speed limit areas.

[0074] In an optional embodiment, for any speed limit area z m There is a speed limit v m , the speed limit value v m satisfy:

[0075] v max ≥v m+1 ≥v m ≥v m-1 ≥v min ;

[0076] N ≥ z m-1 ≥z m ≥z m+1 ≥1.

[0077] It should be noted that the starting point of the present invention is to find suitable speed limit values, speed limit areas and lanes by lane and area. Therefore, in the process of dividing the driving area, it is necessary to associate and define the speed limit area, speed limit value and driving area. In the present invention, the road is divided into N driving areas along the driving direction of the vehicle, but not all of these N driving areas are speed limit areas. Figure 3 and Figure 4 As shown, in Figure 3 In , the location of the congestion area is located in the N-1th driving area, while the Nth driving area is the free flow speed area, and the 1st to N-2 driving areas are the speed limit areas; Figure 4 In the control area, the first driving area is the downstream of the congestion area. M speed limit areas with different speed limits are set in the speed limit area. In each speed limit area, there is a speed limit value and meets certain conditions.

[0078] Specifically, when m>1, in [z m , z m-1 ] area, the speed limit is v m When m=1, in the [z1,1] region, the speed limit is v m .

[0079] In an optional embodiment, the traffic state s of the environment at time slot t is obtained. t include:

[0080] Construct a 2N+M+1 element vector, and use the 2N+M+1 element vector to record the vehicle speed p in the N driving areas n(t) and vehicle density d n (t), the current position z of the M speed limit areas m (t) and the traffic flow f(t) of the Nth driving area;

[0081] The traffic state s is represented by the 2N+M+1 element vector t , the traffic state s t It is expressed as follows:

[0082] s t ={p1(t), d1(t), ...p N (t), d N (t), z1(t), …z M (t), f(t)}.

[0083] It should be noted that each driving area has a driving speed and a vehicle density, and N driving areas have N driving speeds and N vehicle densities; each speed limit area has a current position, and M speed limit areas have M current positions; and the Nth driving area, that is, the traffic volume downstream of the road. The above data are integrated to form the traffic status on the road in the current environment, which includes 2N+M+1 data in total, so a 2N+M+1 element vector is constructed for storage.

[0084] In an optional embodiment, the agent determines the traffic state s by an ε-greedy strategy. t The next action a t , the action a t Methods for changing the location and length of speed limit zones include:

[0085] The agent determines M speed limit areas in the traffic state s through the ε-greedy strategy. t The action set {a1,a2,…,a M ,h},; where h is used to represent the speed classification, a M ∈[-1,0,1];

[0086] The positions and lengths of the M speed limit areas are changed using the action set.

[0087] In an optional embodiment, using the action set to change the position and length of the M speed limit areas includes: m =z m +a m .

[0088] It should be noted that z m =z m +a m The essence of this is to change the location of the speed limit area, which can be achieved by changing zm The position and length of the speed limit zone can be changed by adjusting the position of the speed limit zone.

[0089] In the present invention, the purpose of speed classification h is to classify speed control for lanes. Different speed classifications are determined according to different values. We pre-designed speed classifications, including all reasonable speed classifications. The actual rule is that the speed of vehicles on the left side of the road is greater than that of vehicles on the right side of the road, and the speed difference between two adjacent lanes is guaranteed not to exceed 10km / h. There are a total of [0,0], [0,10], [10,10], [10,20], which represent speed limits higher than the speed limit of the rightmost lane. Different values ​​of h represent different speed classifications. It should be noted that the speed control classification for lanes and sections are different control angles.

[0090] In an optional embodiment, for the action a t and the traffic status s t+1 Calculate and get reward r t include:

[0091] In a fixed bottleneck scenario, the vehicle density in the acceleration zone is obtained, and the vehicle density in the acceleration zone is compared with the traffic state s t+1 Perform joint calculations to obtain rewards in fixed bottleneck scenarios;

[0092] In the mobile bottleneck scenario, observe the number and length of congested areas under N driving areas, and use the number and length of congested areas to determine the reward in the mobile bottleneck scenario.

[0093] In an optional embodiment, in a fixed bottleneck scenario, the vehicle density in the acceleration zone is obtained, and the vehicle density in the acceleration zone is compared with the traffic state s t+1 Perform joint calculations to obtain rewards for fixed bottleneck scenarios, including:

[0094]

[0095] In the above formula, d acc (t) represents the vehicle density in the acceleration zone at time t, d max represents the maximum vehicle density, d ex represents the expected vehicle density, f(t) represents the traffic volume, and f min represents the minimum traffic flow, f max Indicates the maximum traffic volume.

[0096] In an optional embodiment, in a mobile bottleneck scenario, observing the number of congested areas in N driving areas, and using the number of congested areas to determine the reward in the mobile bottleneck scenario includes:

[0097]

[0098] In the above formula, l jam Represents the length of the congested area, and F is a constant.

[0099] In this embodiment, the expected density ranges from 20-30 veh / km / lane, at which time the traffic efficiency is the highest. F represents that the congestion has been resolved, which can be set to +100.

[0100] The present invention collects vehicle information on the road through the Internet of Vehicles, uses deep reinforcement learning to make decisions, and finally obtains a variable speed limit control strategy, realizing refined regional and lane speed limit control. This variable speed limit control method is a general method for solving traffic bottleneck problems. It can solve both fixed bottlenecks and mobile bottlenecks, providing a comprehensive and effective solution for traffic management.

[0101] Example 2

[0102] Embodiment 2 of the present invention provides a bottleneck area traffic flow optimization control system based on deep reinforcement learning, including:

[0103] An environment setting module, the environment setting module is used to divide the road into N driving areas along the vehicle driving direction, determine the control area in the bottleneck scenario based on the N driving areas, set the control area as the environment, and obtain the traffic state s of the environment at time slot t t ;

[0104] An action simulation module, the action simulation module is used to construct an intelligent agent, the intelligent agent determines the traffic state s through an ε-greedy strategy t The next action a t , the action a t Used to change the location and length of the speed limit area;

[0105] A reward calculation module is used to obtain the value of the environment when the agent performs action a. t Traffic status after t+1 , for the action a t and the traffic status s t+1 Calculate and get reward r t ;

[0106] Iterative training module, the iterative training module is used to convert the traffic state s t 、Action a t , Reward t , Traffic status t+1 The experience is integrated into the experience and stored in the experience replay pool, and the experience in the experience replay pool is used to iteratively train the agent.

[0107] Example 3

[0108] Figure 5 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention is shown in FIG. Figure 5 As shown, the electronic device includes a processor 21, a memory 22, an input device 23 and an output device 24; the number of processors 21 in the computer device can be one or more. Figure 5 A processor 21 is taken as an example; the processor 21, the memory 22, the input device 23 and the output device 24 in the electronic device can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0109] The memory 22 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules. The processor 21 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 22, that is, the bottleneck area traffic flow optimization control method based on deep reinforcement learning of Example 1 is implemented.

[0110] The memory 22 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] The input device 23 can be used to receive the ID and password input by the user, etc. The output device 24 is used to output the network configuration page.

[0112] Example 4

[0113] Embodiment 4 of the present invention further provides a computer-readable storage medium, wherein the computer-executable instructions, when executed by a computer processor, are used to implement the bottleneck area traffic flow optimization control method based on deep reinforcement learning as provided in Embodiment 1.

[0114] An embodiment of the present invention provides a storage medium containing computer executable instructions, and its computer executable instructions are not limited to the method operations provided in Example 1, and can also execute related operations in the bottleneck area traffic flow optimization control method based on deep reinforcement learning provided in any embodiment of the present invention.

[0115] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. Optimal control method of traffic flow in bottleneck area based on deep reinforcement learning, It is characterized in that The steps include: Step S1: Divide the road into N driving areas along the vehicle driving direction, determine the control area in the bottleneck scenario based on the N driving areas, set the control area as the environment, and obtain the traffic state s of the environment at time slot t t ; Step S2: construct an intelligent agent, which determines the traffic state s through the ε-greedy strategy. t The next action a t , the action a t Used to change the location and length of the speed limit area; Step S3: Obtain the environment in which the agent performs action a t Traffic status after t+1 , for the action a t and the traffic status s t+1 Calculate and get reward r t ; Step S4: Traffic status s t 、Action a t , Reward t , Traffic status t+1 The experience is integrated into the experience and stored in the experience replay pool, and the experience in the experience replay pool is used to iteratively train the agent.

2. According to the bottleneck area traffic flow optimization control method based on deep reinforcement learning according to claim 1, It is characterized in that Determining the control area in the bottleneck scenario based on the N driving areas includes: In the fixed bottleneck scenario, the Nth driving area is set as the bottleneck area, the N-1th driving area is set as the acceleration area, and the 1st driving area to the N-2th driving area are set as the control area; In the mobile bottleneck scenario, the first driving area to the downstream of the congestion area are set as control areas; The control area is provided with M speed limit areas. For any speed limit area z m There is a speed limit v m .

3. According to claim 2, the bottleneck area traffic flow optimization control method based on deep reinforcement learning, It is characterized in that The speed limit value v m satisfy: in max ≥in m+1 ≥in m ≥in m-1 ≥v min ; N≥z m-1 ≥z m ≥z m+1 ≥1。 4. According to claim 2, the bottleneck area traffic flow optimization control method based on deep reinforcement learning, It is characterized in that Get the traffic status s of the environment at time slot t t include: Construct a 2N+M+1 element vector, and use the 2N+M+1 element vector to record the vehicle speed p in the N driving areas n (t) and vehicle density d n (t), the current position z of the M speed limit areas m (t) and the traffic flow f(t) of the Nth driving area; The traffic state s is represented by the 2N+M+1 element vector t , the traffic state s t It is expressed as follows: s t ={p 1 (t),d 1 (t),…p N (t),d N (t),z 1 (t),…z M (t),f(t)}。 5. According to claim 1, the bottleneck area traffic flow optimization control method based on deep reinforcement learning, It is characterized in that The agent determines the traffic state s through the ε-greedy strategy t The next action a t , the action a t Methods for changing the location and length of speed limit zones include: The agent determines M speed limit areas in the traffic state s through the ε-greedy strategy. t The action set under {a 1 , a 2 , ..., a M ,h},; where h is used to represent the speed classification, a M ∈[-1, 0, 1]; The positions and lengths of the M speed limit areas are changed using the action set.

6. According to claim 5, the bottleneck area traffic flow optimization control method based on deep reinforcement learning, It is characterized in that Using the action set to change the position and length of M speed limit areas includes: m =z m +a m .

7. The bottleneck area traffic flow optimization control method based on deep reinforcement learning according to claim 1, It is characterized in that For the action a t and the traffic status s t+1 Calculate and get reward r t include: In a fixed bottleneck scenario, the vehicle density in the acceleration zone is obtained, and the vehicle density in the acceleration zone is compared with the traffic state s t+1 Perform joint calculations to obtain rewards in fixed bottleneck scenarios; In the mobile bottleneck scenario, observe the number and length of congested areas under N driving areas, and use the number and length of congested areas to determine the reward in the mobile bottleneck scenario.

8. The bottleneck area traffic flow optimization control method based on deep reinforcement learning according to claim 7, It is characterized in that In a fixed bottleneck scenario, the vehicle density in the acceleration zone is obtained, and the vehicle density in the acceleration zone is compared with the traffic state s t+1 Perform joint calculations to obtain rewards for fixed bottleneck scenarios, including: In the above formula, d acc (t) represents the vehicle density in the acceleration zone at time t, d max represents the maximum vehicle density, d ex represents the expected vehicle density, f(t) represents the traffic volume, and f min represents the minimum traffic flow, f max Indicates the maximum traffic volume.

9. The bottleneck area traffic flow optimization control method based on deep reinforcement learning according to claim 7, It is characterized in that In the mobile bottleneck scenario, observe the number of congested areas under N driving areas, and use the number of congested areas to determine the rewards in the mobile bottleneck scenario, including: In the above formula, l jam Represents the length of the congested area, and F is a constant.

10. Bottleneck area traffic flow optimization control system based on deep reinforcement learning, It is characterized in that include: An environment setting module, the environment setting module is used to divide the road into N driving areas along the vehicle driving direction, determine the control area in the bottleneck scenario based on the N driving areas, set the control area as the environment, and obtain the traffic state s of the environment at time slot t t ; An action simulation module, the action simulation module is used to construct an intelligent agent, the intelligent agent determines the traffic state s through an ε-greedy strategy t The next action a t , the action a t Used to change the location and length of the speed limit area; A reward calculation module is used to obtain the value of the environment when the agent performs action a. t Traffic status after t+1 , for the action a t and the traffic status s t+1 Calculate and get reward r t ; Iterative training module, the iterative training module is used to convert the traffic state s t 、Action a t , Reward t , Traffic status t+1 The experience is integrated into the experience and stored in the experience replay pool, and the experience in the experience replay pool is used to iteratively train the agent.