A route-linked traffic signal control method and system based on reinforcement learning

Abstract

The present invention discloses a route-linked traffic signal control method and system based on reinforcement learning, belonging to the field of traffic control. The method comprises obtaining a route map of an implementation area and marking traffic lights on the route map of the implementation area; obtaining a traffic network topology map of the implementation area based on the route map of the implementation area, with traffic lights as nodes; collecting historical traffic flow information during congested periods in the implementation area; maximizing node throughput efficiency and minimizing node congestion rates during congested periods as optimization goals; and utilizing a deep reinforcement learning network model based on the optimization goals, the traffic network topology map, and the historical traffic flow information to obtain a traffic signal control strategy. The present invention solves the problem that existing traffic control methods dynamically adjust the duration of traffic signals while ignoring the unified regulation of the entire duration of congested periods.

Description

Technical Field

[0001] The present invention belongs to the field of traffic control, and in particular relates to a route-linked traffic signal control method and system based on reinforcement learning. Background Art

[0002] Traditional traffic control methods, such as timed control and sensor control, have alleviated traffic pressure to a certain extent. Timed control switches traffic lights according to a preset schedule and is suitable for scenarios with relatively stable traffic flows. However, in real-world traffic conditions, traffic flows often exhibit dynamic characteristics, with significant variations in traffic flow at different times and along different road sections. For example, during rush hour on weekdays, traffic volume on urban arterial roads increases dramatically. However, timed control cannot respond to these changes in real time, easily resulting in excessively long green light periods in some directions, inefficient traffic flow, and a large backlog of vehicles in other directions.

[0003] While sensor-based control can dynamically adjust traffic light durations based on real-time traffic information collected by vehicle detectors, such as vehicle arrival times and queue lengths, it primarily relies on local traffic information and lacks a comprehensive view of the entire transportation network. When traffic flows at multiple intersections are interconnected, sensor-based control struggles to coordinate traffic lights across them, resulting in discontinuous and unbalanced traffic flow across intersections, impacting the smoothness of the entire transportation network.

[0004] Furthermore, existing traffic control methods are inadequate for responding to emergencies. Traffic flow can fluctuate dramatically when unexpected events such as traffic accidents and road construction occur, and traditional control methods are unable to adapt quickly, easily leading to traffic paralysis.

[0005] In order to overcome the shortcomings of the above-mentioned traditional traffic control methods and improve the operating efficiency and reliability of the traffic system, a more intelligent and efficient traffic control method is needed that can perceive the traffic status in real time and make dynamic decisions based on global information to adapt to the complex and changing traffic environment. Summary of the Invention

[0006] In response to the above-mentioned deficiencies in the prior art, the present invention provides a route-linked traffic signal control method and system based on reinforcement learning, which solves the problem that the existing traffic control method dynamically adjusts the duration of the signal while ignoring the problem of unified regulation of the entire duration of the congestion period.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a route-linked traffic signal control method based on reinforcement learning, comprising:

[0008] Obtain a road map of the implementation area and mark the traffic lights on the road map of the implementation area;

[0009] Based on the implementation area roadmap, the traffic network topology of the implementation area is obtained with traffic lights as nodes;

[0010] Collect historical traffic flow information during congestion periods in the implementation area;

[0011] The optimization goals are to maximize the node passing efficiency during congestion periods and minimize the node congestion rate;

[0012] Based on the optimization objectives, traffic network topology and historical traffic flow information, the traffic signal control strategy is obtained using the deep reinforcement learning network model.

[0013] Furthermore, the expression of the optimization objective is:

[0014]

[0015] in, To optimize the goal; To measure the Node No. A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node No. The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes through the previous node and the The time required for each node; is the total number of vehicles; For the The car passes through The status of a node.

[0016] Furthermore, based on the optimization objective, the traffic network topology and historical traffic flow information, a traffic signal control strategy is obtained by using a deep reinforcement learning network model, specifically:

[0017] S1. Initialize the Q network, target network, and experience pool of the deep reinforcement learning network model, and initialize the state of the deep reinforcement learning network model based on the traffic network topology graph;

[0018] S2. According to the current status , using greedy strategy, select action ;

[0019] S3, based on the optimization goal and historical traffic information according to the action Feedback Rewards and the next state ;

[0020] S4. According to the current status ,action ,award and the next state , calculate the state transition sequence , and the state transition sequence Store in experience pool :

[0021]

[0022] S5. Judgment Experience Pool Whether the storage capacity reaches the preset value, if so, from the experience pool Extract N state transition sequences from the Q network and the target network to train the deep reinforcement learning network model; otherwise, the current state Update to next state , and return to step S2;

[0023] S6. Obtain traffic flow information during congestion periods in the area to be controlled, and use the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

[0024] Furthermore, the state is the control state of each node:

[0025]

[0026]

[0027]

[0028]

[0029] in, For the The control status of each node; For the The length of time it takes to pass through each node unit; For the The duration of the ban on each node unit; For the The node connected Vehicles waiting to pass through the road section; For the The physical state of each node; For the The node connected road sections; For road sections The other end node; For the The unit controllable duration of each node; The shortest travel time for pedestrians.

[0030] Furthermore, the action The expression is:

[0031]

[0032] in, To increase the The length of time it takes to pass through each node unit; To reduce the The length of time it takes to pass through each node unit; For the The travel time per node unit remains unchanged.

[0033] Furthermore, the reward The expression is:

[0034]

[0035] in, The node passing efficiency during the congestion period in the current state; is the node congestion rate in the current state; Current status Take action The node passing efficiency during the congestion period when reaching the next state; Current status Take action The node congestion rate when it reaches the next state; To measure the Node No. A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node No. The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes from the previous node to the The time required for each node; is the total number of vehicles; For the The car passes The status of a node.

[0036] Furthermore, the experience pool N state transition sequences are extracted from the network to train the Q network and the target network, completing the training of the deep reinforcement learning network model. Specifically:

[0037] A1. From the experience pool Extract N state transition sequences and obtain the estimated value of each extracted state transition sequence and the value of the next state ;

[0038] A2. The value of the next state according to each state transition sequence and rewards , calculate the target value of each state transition sequence :

[0039]

[0040] in, for The coefficient of fluctuation; is the maximum value of the next state;

[0041] A3. Estimated values based on each state transition sequence and target value , calculate the loss function :

[0042]

[0043] in, is the number of state transition sequences extracted each time; is the sequence number of the state transition sequence;

[0044] A4. Adjust the parameters of the Q network through the back propagation mechanism of the loss , and using the optimizer Minimize loss function ;

[0045] A5. Determine the distance from the last update of the target network parameters Is the number of steps equal to the set value? If so, update the parameters of the target network , go to step A7; otherwise, go to step A6;

[0046] A6. From the experience pool Re-extract N state transition sequences and return to step A1;

[0047] A7. Determine whether the loss function converges. If so, the training of the Q network and the target network is completed, and the trained deep reinforcement learning network model is obtained; otherwise, the training is completed from the experience pool. Re-extract N state transition sequences and return to step A1.

[0048] The beneficial effects of the present invention are as follows: starting from the communication efficiency and congestion rate of each road section connected to each traffic light, the present invention solves the optimal control strategy of the traffic light based on historical traffic information, ensures the global traffic efficiency, and can maximize the smoothness between nodes while avoiding sacrificing the travel time of pedestrians.

[0049] The present invention provides a route-linked traffic signal control system based on reinforcement learning, comprising an environment module, a deep reinforcement learning network module, and a strategy output module;

[0050] The environment module is used to abstract the traffic lights in the area into nodes and the road sections between the nodes into edges connecting the nodes to obtain a regional traffic network topology map; and simulate regional traffic conditions on the regional traffic network topology map based on traffic flow information;

[0051] The deep reinforcement learning network module is used to train the deep reinforcement learning network model according to the historical regional traffic conditions simulated in the environment module to obtain a trained deep reinforcement learning network model;

[0052] The strategy output module is used to optimize the traffic control area to be optimized using the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

[0053] Furthermore, the environment module includes a traffic network topology generation unit, a traffic flow information integration unit, a traffic constraint unit and an environment simulation unit;

[0054] The traffic network topology generation unit is used to abstract the traffic lights in the area into nodes and the road sections between the nodes into edges connecting the nodes, thereby obtaining a traffic network topology map of the area;

[0055] The traffic flow information integration unit is used to collect traffic flow information;

[0056] The traffic constraint unit is used to generate node constraints and vehicle constraints according to the actual traffic control strategy of the road section;

[0057] The environment simulation unit is used to simulate regional traffic conditions based on traffic flow information and the regional traffic network topology diagram, while satisfying node constraints and vehicle constraints.

[0058] Furthermore, the node constraints include the pedestrian travel time reserved for each node and the shortest travel time connecting each road section with a single node; the vehicle constraints include the speed limit of each road section.

[0059] The beneficial effects of the present invention are as follows: the present invention incorporates the limitations of each road section itself as constraints into the environmental module, which can ensure that when training the deep reinforcement learning network model, the state and action of each node are in line with the actual traffic conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Flow chart of the method of the present invention.

[0061] Figure 2 This is a diagram of the deep reinforcement learning network model architecture of the present invention.

[0062] Figure 3 This is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0063] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0064] Example 1

[0065] like Figure 1 As shown, in one embodiment of the present invention, a route-linked traffic signal control method based on reinforcement learning includes:

[0066] Obtain a road map of the implementation area and mark the traffic lights on the road map of the implementation area;

[0067] Based on the implementation area roadmap, the traffic network topology of the implementation area is obtained with traffic lights as nodes;

[0068] Collect historical traffic flow information during congestion periods in the implementation area;

[0069] The optimization goals are to maximize the node passing efficiency during congestion periods and minimize the node congestion rate;

[0070] Based on the optimization objectives, traffic network topology and historical traffic flow information, the traffic signal control strategy is obtained using the deep reinforcement learning network model.

[0071] The expression of the optimization objective is:

[0072]

[0073] in, To optimize the goal; To measure the Node No. A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node No. The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes through the previous node and the The time required for each node; is the total number of vehicles; For the The car passes The status of a node.

[0074] In this embodiment, node transit efficiency and node congestion rate appear to be relative concepts, with one increasing while the other decreasing. If the same measurement method is used, the two will squeeze each other, and subsequent training will inevitably result in extreme cases where one is large and the other is small, which does not conform to actual traffic conditions. However, this invention uses two measurement methods to strive for a balance between smooth flow and congestion, minimizing congestion while maintaining transit efficiency. The time required for a vehicle to pass through a node is counted as zero.

[0075] The node passing efficiency is measured based on the original total congestion duration, ensuring that the node passing time accounts for as high a proportion of the total congestion duration as possible; and this does not only consider the passing efficiency of a single node, but links all nodes, thus taking into account the overall nature of traffic.

[0076] The node congestion rate is measured based on the time required for a vehicle to pass through the current node from the previous node, and the total sum is based on traffic flow information; it is implemented by minimizing the time it takes for all vehicles to pass through all road sections.

[0077] like Figure 2 As shown in the figure, based on the optimization objective, traffic network topology and historical traffic flow information, a traffic signal control strategy is obtained by using a deep reinforcement learning network model, specifically:

[0078] S1. Initialize the Q network, target network, and experience pool of the deep reinforcement learning network model, and initialize the state of the deep reinforcement learning network model based on the traffic network topology graph;

[0079] S2. According to the current status , using greedy strategy, select action ;

[0080] S3, based on the optimization goal and historical traffic information according to the action Feedback Rewards and the next state ;

[0081] S4. According to the current status ,action ,award and the next state , calculate the state transition sequence , and the state transition sequence Store in experience pool :

[0082]

[0083] S5. Judgment Experience Pool Whether the storage capacity reaches the preset value, if so, from the experience pool Extract N state transition sequences from the Q network and the target network to train the deep reinforcement learning network model; otherwise, the current state Update to next state , and return to step S2;

[0084] S6. Obtain traffic flow information during congestion periods in the area to be controlled, and use the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

[0085] In this embodiment, the DQN deep reinforcement learning network is used to solve the strategy.

[0086] The state is the control state of each node:

[0087]

[0088]

[0089]

[0090]

[0091] in, For the The control status of each node; For the The length of time it takes to pass through each node unit; For the The duration of the ban on each node unit; For the The node connected Vehicles waiting to pass through the road section; For the The physical state of each node; For the The node connected road sections; For road sections The other end node; For the The unit controllable duration of each node; The shortest travel time for pedestrians.

[0092] In this example, the ultimate solution is the duration of traffic lights at each node, expressed in the formula as the pass duration and the no-go duration. Since the duration of each set of traffic lights at each node must be controlled, a constraint is imposed on the sum of the pass duration and the no-go duration. Furthermore, a constraint is imposed on the no-go duration to take into account the actual pedestrian traffic conditions.

[0093] The action The expression is:

[0094]

[0095] in, To increase the The length of time it takes to pass through each node unit; To reduce the The length of time it takes to pass through each node unit; For the The travel time per node unit remains unchanged.

[0096] The reward The expression is:

[0097]

[0098] in, The node passing efficiency during the congestion period in the current state; is the node congestion rate in the current state; Current status Take action The node passing efficiency during the congestion period when reaching the next state; Current status Take action The node congestion rate when it reaches the next state; To measure the Node No. A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node No. The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes from the previous node to the The time required for each node; is the total number of vehicles; For the The car passes The status of a node.

[0099] In this embodiment, the greater the distance between the two optimization targets, the better the action execution effect; when the next state execution effect is better than the current state execution effect, the reward is a positive number, otherwise, it is a negative number.

[0100] The experience pool N state transition sequences are extracted from the network to train the Q network and the target network, completing the training of the deep reinforcement learning network model. Specifically:

[0101] A1. From the experience pool Extract N state transition sequences and obtain the estimated value of each extracted state transition sequence and the value of the next state ;

[0102] A2. The value of the next state according to each state transition sequence and rewards , calculate the target value of each state transition sequence :

[0103]

[0104] in, for The coefficient of fluctuation; is the maximum value of the next state;

[0105] A3. Estimated values based on each state transition sequence and target value , calculate the loss function :

[0106]

[0107] in, is the number of state transition sequences extracted each time; is the sequence number of the state transition sequence;

[0108] A4. Adjust the parameters of the Q network through the back propagation mechanism of the loss , and using the optimizer Minimize loss function ;

[0109] A5. Determine the distance from the last update of the target network parameters Is the number of steps equal to the set value? If so, update the parameters of the target network , go to step A7; otherwise, go to step A6;

[0110] A6. From the experience pool Re-extract N state transition sequences and return to step A1;

[0111] A7. Determine whether the loss function converges. If so, the training of the Q network and the target network is completed, and the trained deep reinforcement learning network model is obtained; otherwise, the training is completed from the experience pool. Re-extract N state transition sequences and return to step A1.

[0112] Example 2

[0113] like Figure 3 As shown, a route-linked traffic signal control system based on reinforcement learning includes an environment module, a deep reinforcement learning network module and a strategy output module;

[0114] The environment module is used to abstract the traffic lights in the area into nodes and the road sections between the nodes into edges connecting the nodes to obtain a regional traffic network topology map; and simulate regional traffic conditions on the regional traffic network topology map based on traffic flow information;

[0115] The deep reinforcement learning network module is used to train the deep reinforcement learning network model according to the historical regional traffic conditions simulated in the environment module to obtain a trained deep reinforcement learning network model;

[0116] The strategy output module is used to optimize the traffic control area to be optimized using the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

[0117] The environment module includes a traffic network topology generation unit, a traffic flow information integration unit, a traffic constraint unit and an environment simulation unit;

[0118] The traffic network topology generation unit is used to abstract the traffic lights in the area into nodes and the road sections between the nodes into edges connecting the nodes, thereby obtaining a traffic network topology map of the area;

[0119] The traffic flow information integration unit is used to collect traffic flow information;

[0120] The traffic constraint unit is used to generate node constraints and vehicle constraints according to the actual traffic control strategy of the road section;

[0121] The environment simulation unit is used to simulate regional traffic conditions based on traffic flow information and the regional traffic network topology diagram, while satisfying node constraints and vehicle constraints.

[0122] In this embodiment, the environment module participates in the training of the deep reinforcement learning network model. Based on traffic network topology and traffic flow information, it can simulate real-world conditions. During the training of the deep reinforcement learning network model, the environment module can provide feedback on rewards and next states based on the current state and actions. The real-world constraints inherent in the environment module make the solution strategy more accurate and responsive to actual conditions.

[0123] The node constraints include the pedestrian travel time reserved for each node and the shortest travel time connecting each road section with a single node; the vehicle constraints include the speed limit of each road section.

[0124] In this embodiment, there are various constraints in the actual traffic conditions. For example, the time required for pedestrians to pass needs to be reserved, which also constrains the length of time that pedestrians are prohibited from passing. Speed limits will also affect the time it takes for vehicles to pass through the node. Each road section connected to each node needs to be allocated a travel time, which will also affect the number of vehicles waiting to pass through each road section. The number of vehicles waiting to pass through each road section is closely related to the time required for vehicles to pass through the node in the optimization objective.

Claims

1. A route-linked traffic signal control method based on reinforcement learning, characterized in that: include: Obtain a road map of the implementation area and mark the traffic lights on the road map of the implementation area; Based on the implementation area roadmap, the traffic network topology of the implementation area is obtained with traffic lights as nodes; Collect historical traffic flow information during congestion periods in the implementation area; The optimization goals are to maximize the node passing efficiency during congestion periods and minimize the node congestion rate; Based on the optimization objectives, traffic network topology and historical traffic flow information, the traffic signal control strategy is obtained using the deep reinforcement learning network model; reward The expression is: in, The node passing efficiency during the congestion period in the current state; is the node congestion rate in the current state; Current status Take action The node passing efficiency during the congestion period when reaching the next state; Current status Take action The node congestion rate when it reaches the next state; To measure the Node No. A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node No. The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes through the previous node and passes through the The time required for each node; is the total number of vehicles; For the The car passes through The status of a node.

2. The route-linked traffic signal control method based on reinforcement learning according to claim 1 is characterized in that: The expression of the optimization objective is: in, To optimize the goal; To measure the Node No. A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node No. The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes through the previous node and the The time required for each node; is the total number of vehicles; For the The car passes The status of a node.

3. The route-linked traffic signal control method based on reinforcement learning according to claim 1 is characterized in that: Based on the optimization objective, the traffic network topology and historical traffic flow information, the traffic signal control strategy is obtained by using the deep reinforcement learning network model, specifically: S1. Initialize the Q network, target network, and experience pool of the deep reinforcement learning network model, and initialize the state of the deep reinforcement learning network model based on the traffic network topology graph; S2. According to the current status , using greedy strategy, select action ; S3, based on the optimization goal and historical traffic information according to the action Feedback Rewards and the next state ; S4. According to the current status ,action ,award and the next state , calculate the state transition sequence , and the state transition sequence Store in experience pool : S5. Judgment Experience Pool Whether the storage capacity reaches the preset value, if so, from the experience pool Extract N state transition sequences from the Q network and the target network to train the deep reinforcement learning network model; otherwise, the current state Update to next state , and return to step S2; S6. Obtain traffic flow information during congestion periods in the area to be controlled, and use the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

4. The route-linked traffic signal control method based on reinforcement learning according to claim 3 is characterized in that: The state is the control state of each node: in, For the The control status of each node; For the The length of time it takes to pass through each node unit; For the The duration of the ban on each node unit; For the The node connected Vehicles waiting to pass through the road section; For the The physical state of each node; For the The node connected road sections; For road sections The other end node; For the The unit controllable duration of each node; The shortest travel time for pedestrians.

5. The route-linked traffic signal control method based on reinforcement learning according to claim 3 is characterized in that: The action The expression is: in, To increase the The length of time it takes to pass through each node unit; To reduce the The length of time it takes to pass through each node unit; For the The travel time per node unit remains unchanged.

6. The route-linked traffic signal control method based on reinforcement learning according to claim 3 is characterized in that: The experience pool N state transition sequences are extracted from the network to train the Q network and the target network, completing the training of the deep reinforcement learning network model. Specifically: A1. From the experience pool Extract N state transition sequences and obtain the estimated value of each extracted state transition sequence and the value of the next state ; A2. The value of the next state according to each state transition sequence and rewards , calculate the target value of each state transition sequence : in, for The coefficient of fluctuation; is the maximum value of the next state; A3. Estimated values based on each state transition sequence and target value , calculate the loss function : in, is the number of state transition sequences extracted each time; is the sequence number of the state transition sequence; A4. Adjust the parameters of the Q network through the back propagation mechanism of the loss , and using the optimizer Minimize loss function ; A5. Determine the distance from the last update of the target network parameters Is the number of steps equal to the set value? If so, update the parameters of the target network , go to step A7; otherwise, go to step A6; A6. From the experience pool Re-extract N state transition sequences and return to step A1; A7. Determine whether the loss function converges. If so, the training of the Q network and the target network is completed, and the trained deep reinforcement learning network model is obtained; otherwise, the training is completed from the experience pool. Re-extract N state transition sequences and return to step A1.

7. A route-linked traffic signal control system based on reinforcement learning, adopting the route-linked traffic signal control method based on reinforcement learning according to any one of claims 1 to 6, characterized in that: Includes environment module, deep reinforcement learning network module and strategy output module; The environment module is used to abstract the traffic lights in the area into nodes and the road sections between nodes into edges connecting the nodes, thereby obtaining a traffic network topology diagram of the area; Based on the traffic flow information, the regional traffic situation is simulated on the regional traffic network topology map; The deep reinforcement learning network module is used to train the deep reinforcement learning network model according to the historical regional traffic conditions simulated in the environment module to obtain a trained deep reinforcement learning network model; The strategy output module is used to optimize the traffic control area to be optimized using the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

8. The route-linked traffic signal control system based on reinforcement learning according to claim 7 is characterized in that: The environment module includes a traffic network topology generation unit, a traffic flow information integration unit, a traffic constraint unit and an environment simulation unit; The traffic network topology generation unit is used to abstract the traffic lights in the area into nodes and the road sections between the nodes into edges connecting the nodes, thereby obtaining a traffic network topology map of the area; The traffic flow information integration unit is used to collect traffic flow information; The traffic constraint unit is used to generate node constraints and vehicle constraints according to the actual traffic control strategy of the road section; The environment simulation unit is used to simulate regional traffic conditions based on traffic flow information and the regional traffic network topology diagram, while satisfying node constraints and vehicle constraints.

9. The route-linked traffic signal control system based on reinforcement learning according to claim 8, characterized in that: The node constraints include the pedestrian travel time reserved for each node and the shortest travel time connecting each road section with a single node; the vehicle constraints include the speed limit of each road section.

Images