Route linkage type traffic signal control method and system based on reinforcement learning

Abstract

The invention discloses a route linkage type traffic signal control method and system based on reinforcement learning, and belongs to the field of traffic control, and the method comprises the steps: obtaining an implementation region route map, and marking traffic lights on the implementation region route map; obtaining a traffic network topological graph of the implementation region by taking traffic lights as nodes based on the implementation region route map; historical traffic flow information in the congestion time period of the implementation area is collected; the node passing efficiency maximization in the congestion period and the node congestion rate minimization are taken as optimization targets; and based on the optimization target, the traffic network topological graph and the historical traffic flow information, utilizing a deep reinforcement learning network model to obtain a traffic signal control strategy. The method solves the problem that the time length of a dynamic regulation and control signal of an existing traffic control method ignores the unified regulation and control of the full time length of a congestion period.

Description

Technical Field

[0001] The present invention belongs to the field of traffic control, and in particular relates to a route-linked traffic signal control method and system based on reinforcement learning. Background Art

[0002] Traditional traffic control methods, such as timing control and induction control, have alleviated traffic pressure to a certain extent. Timing control switches traffic lights according to a preset time plan, which is suitable for scenes with relatively stable traffic flow. However, in actual traffic conditions, traffic flow often has the characteristics of dynamic change, and traffic flow varies greatly at different times and on different sections of roads. For example, during rush hours in the morning and evening on weekdays, the traffic volume on the main roads of the city will increase sharply, and timing control cannot respond to this change in real time, which can easily cause the green light time in some directions to be too long, resulting in low vehicle traffic efficiency, while there will be a large backlog of vehicles in other directions.

[0003] Although inductive control can dynamically adjust the duration of traffic lights based on real-time traffic information collected by vehicle detectors, such as vehicle arrival time and queue length, it mainly makes decisions based on local traffic information and lacks overall consideration of the entire traffic network. When the traffic flows between multiple intersections are interrelated, inductive control is difficult to coordinate the traffic lights at each intersection, resulting in discontinuity and imbalance in traffic flow between intersections, affecting the smoothness of the entire traffic network.

[0004] In addition, existing traffic control methods are also inadequate in dealing with emergencies. Once an accident, road construction or other unexpected situation occurs, traffic flow will change suddenly, and traditional control methods cannot make quick adaptive adjustments, which can easily lead to traffic paralysis.

[0005] In order to overcome the shortcomings of the above-mentioned traditional traffic control methods and improve the operating efficiency and reliability of the traffic system, a more intelligent and efficient traffic control method is needed that can perceive the traffic status in real time and make dynamic decisions based on global information to adapt to the complex and changing traffic environment. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a route-linked traffic signal control method and system based on reinforcement learning, which solves the problem that the existing traffic control method dynamically adjusts the duration of the signal but ignores the problem of unified regulation of the entire duration during the congestion period.

[0007] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a route-linked traffic signal control method based on reinforcement learning, comprising: Obtain a road map of the implementation area and mark the traffic lights on the road map of the implementation area; Based on the implementation area roadmap, the traffic network topology of the implementation area is obtained with traffic lights as nodes; Collect historical traffic flow information during congestion periods in the implementation area; The optimization goal is to maximize the node passing efficiency during congestion and minimize the node congestion rate; Based on the optimization objectives, traffic network topology and historical traffic flow information, the traffic signal control strategy is obtained using the deep reinforcement learning network model.

[0008] Furthermore, the expression of the optimization objective is:

[0009] in, To optimize the goal; To measure the Node A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes through the previous node and the The time required for each node; is the total number of vehicles; For the The car passes through The status of a node.

[0010] Furthermore, based on the optimization objective, the traffic network topology and the historical traffic flow information, the traffic signal control strategy is obtained by using the deep reinforcement learning network model, which is specifically: S1. Initialize the Q network, target network and experience pool of the deep reinforcement learning network model, and initialize the state of the deep reinforcement learning network model based on the traffic network topology map; S2. According to the current status , using the greedy strategy, select the action ; S3, based on the optimization goal and historical traffic information according to the action Feedback Rewards and the next state ; S4. According to the current status ,action ,award and the next state , calculate the state transition sequence , and the state transition sequence Store to experience pool :

[0011] S5. Judgment experience pool Whether the storage capacity reaches the preset value, if so, from the experience pool Extract N state transition sequences from the Q network and the target network to train the deep reinforcement learning network model; otherwise, the current state Update to next state , and return to step S2; S6. Obtain traffic flow information during congestion periods in the area to be controlled, and use the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

[0012] Furthermore, the state is the control state of each node:

[0013]

[0014]

[0015]

[0016] in, For the The control status of each node; For the The travel time of each node unit; For the The length of time a node unit is prohibited from driving; For the The node connected Vehicles waiting to pass through the road section; For the The physical state of each node; For the The node connected road sections; For road section The other end node of For the The unit controllable duration of each node; The shortest travel time for pedestrians.

[0017] Furthermore, the action The expression is:

[0018] in, To increase the The travel time of each node unit; To reduce the The travel time of each node unit; For the The travel time per node remains unchanged.

[0019] Furthermore, the reward The expression is:

[0020] in, The node passing efficiency during the congestion period in the current state; is the node congestion rate in the current state; For the current state Take Action The node passing efficiency during the congestion period when it reaches the next state; For the current state Take Action The node congestion rate when it reaches the next state; To measure the Node A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes from the previous node to the The time required for each node; is the total number of vehicles; For the The car passes through The status of a node.

[0021] Furthermore, the experience pool N state transition sequences are extracted from the training to train the Q network and the target network, and the training of the deep reinforcement learning network model is completed. Specifically: A1. From the experience pool Extract N state transition sequences from the , and obtain the estimated value of each extracted state transition sequence and the value of the next state ; A2. The value of the next state according to each state transition sequence and rewards , calculate the target value of each state transition sequence :

[0022] in, for The coefficient of volatility; is the maximum value of the next state; A3. According to the estimated value of each state transition sequence and target value , calculate the loss function :

[0023] in, is the number of state transition sequences extracted each time; is the sequence number of the state transition sequence; A4. Adjust the parameters of the Q network through the back propagation mechanism of the loss , and use the optimizer Reduce the loss function ; A5. Determine the distance from the last update of the target network parameters Is the number of steps equal to the set value? If so, update the parameters of the target network. , go to step A7; otherwise, go to step A6; A6. From the experience pool Re-extract N state transition sequences and return to step A1; A7. Determine whether the loss function converges. If so, the training of the Q network and the target network is completed, and a trained deep reinforcement learning network model is obtained; otherwise, the loss function is obtained from the experience pool. Re-extract N state transition sequences and return to step A1.

[0024] The beneficial effects of the present invention are as follows: the present invention starts from the communication efficiency and congestion rate of each road section connected to each traffic light, and based on historical traffic information, solves the optimal control strategy of the traffic light to ensure the global traffic efficiency, and can maximize the smoothness between nodes while avoiding the situation of sacrificing the travel time of pedestrians.

[0025] The present invention provides a route-linked traffic signal control system based on reinforcement learning, comprising an environment module, a deep reinforcement learning network module and a strategy output module; The environment module is used to abstract the traffic lights of the area into nodes, and the road sections between the nodes into edges connecting the nodes, so as to obtain a traffic network topology map of the area; and based on the traffic flow information, simulate the regional traffic conditions on the regional traffic network topology map; The deep reinforcement learning network module is used to train the deep reinforcement learning network model according to the historical regional traffic conditions simulated in the environment module to obtain a trained deep reinforcement learning network model; The strategy output module is used to optimize the traffic control area to be optimized by using the trained deep reinforcement learning network model to obtain the traffic signal control strategy.

[0026] Furthermore, the environment module includes a traffic network topology generation unit, a traffic flow information integration unit, a traffic constraint unit and an environment simulation unit; The traffic network topology generation unit is used to abstract the traffic lights in the area into nodes, and the road sections between the nodes into edges connecting the nodes, so as to obtain a traffic network topology map of the area; The vehicle flow information integration unit is used to collect vehicle flow information; The traffic constraint unit is used to generate node constraints and vehicle constraints according to the actual traffic control strategy of the road section; The environment simulation unit is used to simulate regional traffic conditions based on vehicle flow information and a regional traffic network topology diagram, under the premise of satisfying node constraints and vehicle constraints.

[0027] Furthermore, the node constraints include the pedestrian travel time reserved for each node and the shortest travel time connecting each road section with a single node; the vehicle constraints include the speed limit of each road section.

[0028] The beneficial effect of the present invention is that the present invention integrates the limitations existing in each road section itself as constraints into the environmental module, which can ensure that when training the deep reinforcement learning network model, the state and action of each node are in line with the actual traffic conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a flow chart of the method of the present invention.

[0030] Figure 2 This is a diagram of the deep reinforcement learning network model architecture of the present invention.

[0031] Figure 3 It is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0032] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0033] Example 1 like Figure 1 As shown, in one embodiment of the present invention, a route-linked traffic signal control method based on reinforcement learning includes: Obtain a road map of the implementation area and mark the traffic lights on the road map of the implementation area; Based on the implementation area roadmap, the traffic network topology of the implementation area is obtained with traffic lights as nodes; Collect historical traffic flow information during congestion periods in the implementation area; The optimization goal is to maximize the node passing efficiency during congestion and minimize the node congestion rate; Based on the optimization objectives, traffic network topology and historical traffic flow information, the traffic signal control strategy is obtained using the deep reinforcement learning network model.

[0034] The expression of the optimization objective is:

[0035] in, To optimize the goal; To measure the Node A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes through the previous node and the The time required for each node; is the total number of vehicles; For the The car passes through The status of a node.

[0036] In this embodiment, the node passing efficiency and node congestion rate seem to be relative, a set of concepts that increase and decrease. If the same measurement method is used, the two are squeezing each other, and subsequent training will inevitably result in extreme situations where one is very large and the other is very small, which does not conform to actual traffic conditions. However, the present invention uses two measurement methods to seek a balance between smooth flow and congestion, and minimize congestion while satisfying passing efficiency. The time required for a node that a vehicle does not need to pass through is counted as 0.

[0037] The node passing efficiency is measured based on the original total congestion duration, ensuring that the node passing time accounts for as high a proportion of the total congestion duration as possible; and this does not only consider the passing efficiency of a single node, but links all nodes, thus taking into account the overall nature of traffic.

[0038] The node congestion rate is measured based on the time required for a vehicle to pass through the current node from the previous node, and the total sum is based on traffic information; it is implemented by minimizing the time it takes for all vehicles to pass through all road sections.

[0039] like Figure 2 As shown, based on the optimization objective, the traffic network topology and the historical traffic flow information, the traffic signal control strategy is obtained by using the deep reinforcement learning network model, which is specifically: S1. Initialize the Q network, target network and experience pool of the deep reinforcement learning network model, and initialize the state of the deep reinforcement learning network model based on the traffic network topology map; S2. According to the current status , using the greedy strategy, select the action ; S3, based on the optimization goal and historical traffic information according to the action Feedback Rewards and the next state ; S4. According to the current status ,action ,award and the next state , calculate the state transition sequence , and the state transition sequence Store to experience pool :

[0040] S5. Judgment experience pool Whether the storage capacity reaches the preset value, if so, from the experience pool Extract N state transition sequences from the Q network and the target network to train the deep reinforcement learning network model; otherwise, the current state Update to next state , and return to step S2; S6. Obtain traffic flow information during congestion periods in the area to be controlled, and use the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

[0041] In this embodiment, the DQN deep reinforcement learning network is used to solve the strategy.

[0042] The state is the control state of each node:

[0043]

[0044]

[0045]

[0046] in, For the The control status of each node; For the The travel time of each node unit; For the The length of time a node unit is prohibited from driving; For the The node connected Vehicles waiting to pass through the road section; For the The physical state of each node; For the The node connected road sections; For road section The other end node of For the The unit controllable duration of each node; The shortest travel time for pedestrians.

[0047] In this embodiment, the final solution is the duration of the traffic lights at each node, which is expressed in the formula as the pass duration and the no-go duration. The duration of a single group of traffic lights at each node should be controlled, so the sum of the pass duration and the no-go duration is constrained; at the same time, considering the actual situation of pedestrian traffic, the no-go duration is constrained.

[0048] The action The expression is:

[0049] in, To increase the The travel time of each node unit; To reduce the The travel time of each node unit; For the The travel time per node remains unchanged.

[0050] The Reward The expression is:

[0051] in, The node passing efficiency during the congestion period in the current state; is the node congestion rate in the current state; For the current state Take Action The node passing efficiency during the congestion period when it reaches the next state; For the current state Take Action The node congestion rate when it reaches the next state; To measure the Node A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes from the previous node to the The time required for each node; is the total number of vehicles; For the The car passes through The status of a node.

[0052] In this embodiment, the farther the distance between the two optimization targets is, the better the action execution effect is; when the execution effect of the next state is better than that of the current state, the reward is a positive number, otherwise, it is a negative number.

[0053] From the experience pool N state transition sequences are extracted from the training to train the Q network and the target network, and the training of the deep reinforcement learning network model is completed. Specifically: A1. From the experience pool Extract N state transition sequences from the , and obtain the estimated value of each extracted state transition sequence and the value of the next state ; A2. The value of the next state according to each state transition sequence and rewards , calculate the target value of each state transition sequence :

[0054] in, for The coefficient of volatility; is the maximum value of the next state; A3. According to the estimated value of each state transition sequence and target value , calculate the loss function :

[0055] in, is the number of state transition sequences extracted each time; is the sequence number of the state transition sequence; A4. Adjust the parameters of the Q network through the back propagation mechanism of the loss , and use the optimizer Reduce the loss function ; A5. Determine the distance from the last update of the target network parameters Is the number of steps equal to the set value? If so, update the parameters of the target network. , go to step A7; otherwise, go to step A6; A6. From the experience pool Re-extract N state transition sequences and return to step A1; A7. Determine whether the loss function converges. If so, the training of the Q network and the target network is completed, and a trained deep reinforcement learning network model is obtained; otherwise, the loss function is obtained from the experience pool. Re-extract N state transition sequences and return to step A1.

[0056] Example 2 like Figure 3 As shown, a route-linked traffic signal control system based on reinforcement learning includes an environment module, a deep reinforcement learning network module and a strategy output module; The environment module is used to abstract the traffic lights of the area into nodes, and the road sections between the nodes into edges connecting the nodes, so as to obtain a traffic network topology map of the area; and based on the traffic flow information, simulate the regional traffic conditions on the regional traffic network topology map; The deep reinforcement learning network module is used to train the deep reinforcement learning network model according to the historical regional traffic conditions simulated in the environment module to obtain a trained deep reinforcement learning network model; The strategy output module is used to optimize the traffic control area to be optimized by using the trained deep reinforcement learning network model to obtain the traffic signal control strategy.

[0057] The environment module includes a traffic network topology generation unit, a traffic flow information integration unit, a traffic constraint unit and an environment simulation unit; The traffic network topology generation unit is used to abstract the traffic lights in the area into nodes, and the road sections between the nodes into edges connecting the nodes, so as to obtain a traffic network topology map of the area; The vehicle flow information integration unit is used to collect vehicle flow information; The traffic constraint unit is used to generate node constraints and vehicle constraints according to the actual traffic control strategy of the road section; The environment simulation unit is used to simulate regional traffic conditions based on vehicle flow information and a regional traffic network topology diagram, under the premise of satisfying node constraints and vehicle constraints.

[0058] In this embodiment, the environment module participates in the training of the deep reinforcement learning network model. Based on the traffic network topology and traffic flow information, it can simulate real working conditions. When training the deep reinforcement learning network model, the environment module can feedback rewards and next states based on the current state and action. The real constraints in the environment module itself can make the solution strategy more accurate and in line with the actual situation.

[0059] The node constraints include the pedestrian travel time reserved for each node and the shortest travel time connecting each road section with a single node; the vehicle constraints include the speed limit of each road section.

[0060] In this embodiment, there are various constraints in the actual traffic conditions. For example, the time for pedestrians to pass needs to be reserved, which also constrains the no-entry time; the speed limit will also affect the time it takes for vehicles to pass through the node; each road section connected to each node needs to be allocated a travel time, which will also affect the vehicles waiting to pass through each road section; and the vehicles waiting to pass through each road section are closely related to the time required for vehicles to pass through the node in the optimization target.

Claims

1. A route-linked traffic signal control method based on reinforcement learning, characterized in that: include: Obtain a road map of the implementation area and mark the traffic lights on the road map of the implementation area; Based on the implementation area roadmap, the traffic network topology of the implementation area is obtained with traffic lights as nodes; Collect historical traffic flow information during congestion periods in the implementation area; The optimization goal is to maximize the node passing efficiency during congestion and minimize the node congestion rate; Based on the optimization objectives, traffic network topology and historical traffic flow information, the traffic signal control strategy is obtained using the deep reinforcement learning network model.

2. The route-linked traffic signal control method based on reinforcement learning according to claim 1 is characterized in that: The expression of the optimization objective is: in, To optimize the goal; To measure the Node A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes through the previous node and then passes through the The time required for each node; is the total number of vehicles; For the The car passes through The status of a node.

3. The route-linked traffic signal control method based on reinforcement learning according to claim 1 is characterized in that: Based on the optimization objective, the traffic network topology and the historical traffic flow information, the traffic signal control strategy is obtained by using the deep reinforcement learning network model, which is specifically: S1. Initialize the Q network, target network and experience pool of the deep reinforcement learning network model, and initialize the state of the deep reinforcement learning network model based on the traffic network topology map; S2. According to the current status , using the greedy strategy, select the action ; S3, based on the optimization goal and historical traffic information according to the action Feedback Rewards and the next state ; S4. According to the current status ,action ,award and the next state , calculate the state transition sequence , and the state transition sequence Store to experience pool : S5. Judgment experience pool Whether the storage capacity reaches the preset value, if so, from the experience pool Extract N state transition sequences from the Q network and the target network to train the deep reinforcement learning network model; otherwise, the current state Update to next state , and return to step S2; S6. Obtain traffic flow information during congestion periods in the area to be controlled, and use the trained deep reinforcement learning network model to obtain a traffic signal control strategy.

4. The route-linked traffic signal control method based on reinforcement learning according to claim 3 is characterized in that: The state is the control state of each node: in, For the The control status of each node; For the The travel time of each node unit; For the The length of time a node unit is prohibited from driving; For the The node connected Vehicles waiting to pass through the road section; For the The physical state of each node; For the The node connected road sections; For road section The other end node of For the The unit controllable duration of each node; The shortest travel time for pedestrians.

5. The route-linked traffic signal control method based on reinforcement learning according to claim 3 is characterized in that: The action The expression is: in, To increase the The travel time of each node unit; To reduce the The travel time of each node unit; For the The travel time per node remains unchanged.

6. The route-linked traffic signal control method based on reinforcement learning according to claim 3 is characterized in that: The Reward The expression is: in, The node passing efficiency during the congestion period in the current state; is the node congestion rate in the current state; For the current state Take Action The node passing efficiency during the congestion period when it reaches the next state; For the current state Take Action The node congestion rate when it reaches the next state; To measure the Node A binary parameter indicating whether a vehicle passes at the moment, if yes, it is 1, otherwise, it is 0; For the Node The state of the moment; is the total number of nodes; is the congestion duration; For the The vehicle passes from the previous node to the The time required for each node; is the total number of vehicles; For the The car passes through The status of a node.

7. The route-linked traffic signal control method based on reinforcement learning according to claim 3 is characterized in that: From the experience pool N state transition sequences are extracted from the training to train the Q network and the target network, and the training of the deep reinforcement learning network model is completed. Specifically: A1. From the experience pool Extract N state transition sequences from the , and obtain the estimated value of each extracted state transition sequence and the value of the next state ; A2. The value of the next state according to each state transition sequence and rewards , calculate the target value of each state transition sequence : in, for The coefficient of volatility; is the maximum value of the next state; A3. According to the estimated value of each state transition sequence and target value , calculate the loss function : in, is the number of state transition sequences extracted each time; is the sequence number of the state transition sequence; A4. Adjust the parameters of the Q network through the back propagation mechanism of the loss , and use the optimizer Reduce the loss function ; A5. Determine the distance from the last update of the target network parameters Is the number of steps equal to the set value? If so, update the parameters of the target network. , go to step A7; otherwise, go to step A6; A6. From the experience pool Re-extract N state transition sequences and return to step A1; A7. Determine whether the loss function converges. If so, the training of the Q network and the target network is completed, and a trained deep reinforcement learning network model is obtained; otherwise, the loss function is obtained from the experience pool. Re-extract N state transition sequences and return to step A1.

8. A route-linked traffic signal control system based on reinforcement learning, using the route-linked traffic signal control method based on reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: Includes environment module, deep reinforcement learning network module and strategy output module; The environment module is used to abstract the traffic lights in the area into nodes, and the road sections between the nodes into edges connecting the nodes, so as to obtain a traffic network topology diagram of the area; Based on the traffic flow information, the regional traffic situation is simulated on the regional traffic network topology map; The deep reinforcement learning network module is used to train the deep reinforcement learning network model according to the historical regional traffic conditions simulated in the environment module to obtain a trained deep reinforcement learning network model; The strategy output module is used to optimize the traffic control area to be optimized by using the trained deep reinforcement learning network model to obtain the traffic signal control strategy.

9. The route-linked traffic signal control system based on reinforcement learning according to claim 8, characterized in that: The environment module includes a traffic network topology generation unit, a traffic flow information integration unit, a traffic constraint unit and an environment simulation unit; The traffic network topology generation unit is used to abstract the traffic lights in the area into nodes, and the road sections between the nodes into edges connecting the nodes, so as to obtain a traffic network topology map of the area; The vehicle flow information integration unit is used to collect vehicle flow information; The traffic constraint unit is used to generate node constraints and vehicle constraints according to the actual traffic control strategy of the road section; The environment simulation unit is used to simulate regional traffic conditions based on vehicle flow information and a regional traffic network topology diagram, under the premise of satisfying node constraints and vehicle constraints.

10. The route-linked traffic signal control system based on reinforcement learning according to claim 9, characterized in that: The node constraints include the pedestrian travel time reserved for each node and the shortest travel time connecting each road section with a single node; the vehicle constraints include the speed limit of each road section.

Images