Importance sampling based policy gradient for two-intersection traffic signal control

Abstract

A traffic signal control method for dual-intersection intersections based on importance sampling and policy gradient includes the following steps: S1, collecting given physical feature information, traffic flow information, and traffic light phase information of the dual-intersection, and building a simulation model of the traffic simulation platform using microscopic simulation software; S2, based on the built simulation model, building a deep neural network model, and training the distribution π of the traffic light phase control strategy based on the policy gradient principle of importance sampling. θ S3. Based on the trained deep neural network, the current intersection status information and traffic light phase information s can be updated. k ,a k The phase 'a' of the traffic lights that need to be executed at the next moment at the intersection is obtained. k+1 This invention enables faster training of signal control systems to achieve superior performance, and its application in road network control can more quickly alleviate traffic congestion.

Description

Technical Field

[0001] This invention belongs to the field of traffic control and relates to a traffic signal control method for dual intersections based on importance sampling and strategy gradient. Background Technology

[0002] With the rapid increase in the total number of cars owned by Chinese residents, cities are currently facing traffic congestion problems. At the same time, compared with improving road network traffic facilities, optimizing traffic timing schemes at intersections is more effective and has better economic benefits.

[0003] Traditional traffic light control methods all employ fixed configuration schemes, which cannot cope with the real-time nature and rapid changes in traffic flow characteristic of urban road networks. Currently, real-time traffic timing schemes are a popular research area. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a policy gradient traffic signal control method for dual intersections based on importance sampling. By collecting information from the intersection, a microscopic simulation model is built to train a deep neural network. The neural network controller trained by the policy gradient traffic signal control method for dual intersections based on importance sampling can effectively alleviate traffic congestion.

[0005] The technical solution adopted by this invention to solve its technical problem is:

[0006] A policy gradient-based traffic signal control method for dual intersections based on importance sampling includes the following steps:

[0007] S1. Collect the given physical characteristics of the double intersection, traffic flow information, and traffic light phase information, and use microscopic simulation software to build a simulation model of the traffic simulation platform.

[0008] S2. Based on the established simulation model, build a deep neural network model, and train the distribution of the traffic light phase control strategy based on the policy gradient principle of importance sampling. And update the parameters of the neural network. ;

[0009] S3. Based on the trained deep neural network, the current intersection status information and traffic light phase information can be used to determine the current intersection status information and traffic light phase information. This yields the traffic light phases that need to be executed at the next moment at the intersection. for:

[0010] .

[0011] Furthermore, in step S2, the parameters of the deep neural network are updated. The steps are as follows:

[0012] S21, Set the maximum number of iterations Initialize deep neural network parameters The neural network structure adaptively adjusts according to the size of the two intersections, and hyperparameters are set accordingly. Discount factor ;

[0013] S22, Settings According to the initial neural network The simulation software samples and obtains the traffic flow status and traffic light action set for the first intersection. and calculate ,in

[0014] ;

[0015] in This refers to state-action pairs The corresponding reward function, It is a baseline constant. , Indicates the number of state-action pairs contained in the trajectory;

[0016] S23, Settings The trajectory of the vehicle's state and actions is obtained by sampling through a microscopic simulation model. and calculate Importance sampling weight Calculated using the following formula:

[0017] ;

[0018] S24, Calculation renew ;

[0019] S25, Repeat steps S23 and S24 until the maximum number of iterations is reached. until;

[0020] S26, Output the latest neural network parameters .

[0021] Furthermore, in step S1, the steps for acquiring road network vehicle traffic information data are as follows:

[0022] S11, based on the data collection time Obtain current traffic flow and traffic light information. ;

[0023] S12, For a given double-intersection road network, the traffic flow distribution data at the entrance is obtained by statistical analysis based on the acquired information;

[0024] S13. Based on the traffic flow data at the entrance and the physical characteristics of the two intersections, including the location of traffic lights, lane distribution and lane length, a micro-simulation model is built in the micro-simulation software to simulate the daily driving conditions of the intersection.

[0025] The main benefits of this invention are that it enables signal control to be trained faster and achieve better performance, and when applied to road network control, it can alleviate traffic congestion more quickly. Attached Figure Description

[0026] Figure 1 This is a flowchart of the importance sampling neural network training of the present invention.

[0027] Figure 2 This is a schematic diagram of the road network for a double intersection according to the present invention. Detailed Implementation

[0028] The present invention will now be further described with reference to the accompanying drawings.

[0029] Reference Figure 1 and Figure 2 A strategy gradient-based traffic signal control method for dual intersections based on importance sampling includes the following steps:

[0030] S1. Collect the given physical characteristics of the double intersection, traffic flow information, and traffic light phase information, and use microscopic simulation software to build a simulation model of the traffic simulation platform.

[0031] In step S1, the steps for obtaining road network vehicle traffic information data are as follows:

[0032] S11, based on the data collection time Obtain current traffic flow and traffic light information. ;

[0033] S12, For a given double-intersection road network, the traffic flow distribution data at the entrance is obtained by statistical analysis based on the acquired information;

[0034] S13. Based on the traffic flow data at the entrance and the physical characteristics of the two intersections, including the location of traffic lights, lane distribution and lane length, a micro-simulation model is built in the micro-simulation software to simulate the daily driving conditions of the intersection.

[0035] S2. Based on the established simulation model, build a deep neural network model, and train the distribution of the traffic light phase control strategy based on the policy gradient principle of importance sampling. And update the parameters of the neural network. ;

[0036] In step S2, the parameters of the deep neural network are updated. The steps are as follows:

[0037] S21, Set the maximum number of iterations Initialize deep neural network parameters The neural network structure adaptively adjusts according to the size of the two intersections, and hyperparameters are set accordingly. Discount factor ;

[0038] S22, Settings According to the initial neural network The simulation software samples and obtains the traffic flow status and traffic light action set for the first intersection. and calculate ,in

[0039] ;

[0040] in This refers to state-action pairs The corresponding reward function, It is a baseline constant. , Indicates the number of state-action pairs contained in the trajectory;

[0041] S23, Settings The trajectory of the vehicle's state and actions is obtained by sampling through a microscopic simulation model. and calculate Importance sampling weight Calculated using the following formula:

[0042] ;

[0043] S24, Calculation renew ;

[0044] S25, Repeat steps S23 and S24 until the maximum number of iterations is reached. until;

[0045] S26, Output the latest neural network parameters .

[0046] S3. Based on the trained deep neural network, the current intersection status information and traffic light phase information can be used to determine the current intersection status information and traffic light phase information. This yields the traffic light phases that need to be executed at the next moment at the intersection. for:

[0047] .

[0048] This invention first collects information from two intersections, then builds a corresponding microscopic simulation model, and finally trains a neural network based on the gradient optimization principle of an importance sampling strategy. The resulting real-time traffic light controller can adapt to the sudden changes and uncertainties at urban traffic intersections.

[0049] See Figure 2 This embodiment uses a two-way four-lane dual-intersection road network built on the microscopic simulation software Simulation of Urban Mobility (SUMO) as an example. Using the above method, an intersection signal controller based on importance sampling was obtained. Simulation results show that the training time of the neural network is shorter and the performance of the signal controller is better.

[0050] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A traffic signal control method for dual intersections based on importance sampling and strategy gradient, characterized in that, The method includes the following steps: S1. Collect the given physical characteristics of the double intersection, traffic flow information, and traffic light phase information, and use microscopic simulation software to build a simulation model of the traffic simulation platform. S2. Based on the established simulation model, build a deep neural network model, and train the distribution of the traffic light phase control strategy based on the policy gradient principle of importance sampling. And update the parameters of the neural network. ; S3. Based on the trained deep neural network, and according to the current intersection status information and traffic light phase information... This yields the traffic light phases that need to be executed at the next moment at the intersection. for: ； In step S2, the parameters of the deep neural network are updated. The steps are as follows: S21, Set the maximum number of iterations Initialize deep neural network parameters The neural network structure adaptively adjusts according to the size of the two intersections, and hyperparameters are set accordingly. Discount factor ; S22, Settings According to the initial neural network The simulation software samples and obtains the traffic flow status and traffic light action set for the first intersection. and calculate ,in ； in This refers to state-action pairs The corresponding reward function, It is a baseline constant. , Indicates the number of state-action pairs contained in the trajectory; S23, Settings The trajectory of the vehicle's state and actions is obtained by sampling through a microscopic simulation model. and calculate Importance sampling weight Calculated using the following formula: ； S24, Calculation renew ; S25, Repeat steps S23 and S24 until the maximum number of iterations is reached. until; S26, Output the latest neural network parameters .

2. The traffic signal control method for dual intersections based on importance sampling as described in claim 1, characterized in that, In step S1, the steps for obtaining road network vehicle traffic information data are as follows: S11, based on the data collection time Obtain current traffic flow and traffic light information. ; S12, For a given double-intersection road network, the traffic flow distribution data at the entrance is obtained by statistical analysis based on the acquired information; S13. Based on the traffic flow data at the entrance and the physical characteristics of the two intersections, including the location of traffic lights, lane distribution and lane length, a micro-simulation model is built in the micro-simulation software to simulate the daily driving conditions of the intersection.

Images