A traffic signal control method for urban road networks based on multi-agent Actor-Critic and GRU
By applying the traffic signal control method of multi-agent Actor-Critic and GRU in the urban road network, the problem of traditional methods being unable to effectively deal with urban traffic randomness is solved, and more efficient traffic signal control is achieved, reducing vehicle waiting time during peak traffic.
Patent Information
- Application Number
- CN202310797286.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-29
- Publication Date
- 2025-03-25
- Estimated Expiration
- 2043-06-29
AI Technical Summary
Traditional traffic light timing methods cannot effectively deal with the randomness of urban traffic, resulting in large-scale congestion during peak traffic.
The traffic signal control method of the urban road network based on multi-agent Actor-Critic and GRU is adopted. Through the passing traffic information provided by the checkpoint, the mutual communication between multiple agents and the "memory" function of the deep recurrent neural network is used to build an agent training simulation platform to realize real-time optimization and control of signal lights.
The traffic efficiency of urban transportation and road network has been improved, especially during peak traffic periods, the average waiting time of vehicles has been reduced by 23%.
Smart Images
Figure CN116895158B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of traffic control and relates to a traffic signal control method for a large urban road network based on multi-agent Actor-Critic and GRU. Background Art
[0002] With the rapid growth of population and the continuous advancement of urbanization, traffic congestion is becoming increasingly serious. In the context of slow urban infrastructure construction, it is urgent to find an optimization solution for traffic light timing. Traditional timing schemes are almost all based on timed timing methods based on historical information, which cannot handle the strong randomness of urban traffic and are prone to large-scale congestion during peak traffic hours. One of the most promising methods is to develop an adaptive traffic management system that uses real-time continuous traffic data from sensors to dynamically control the duration of traffic lights at intersections according to congestion conditions, thereby optimizing the traffic conditions of urban road networks. However, the design of signal control methods is a difficult point. Summary of the invention
[0003] In view of the problems existing in the above background technology introduction, the purpose of the present invention is to provide a traffic signal control method for a large urban road network based on multi-agent Actor-Critic and GRU, through the traffic information provided by the checkpoint, and based on the mutual communication between multi-agents and the "memory" function of the deep recurrent neural network, using the passing vehicle data and the full OD estimation method, to build an agent training simulation platform, so as to achieve a 1:1 true restoration of the actual road network physical characteristics, and more than 90% of the passing vehicle data is approximate. And on this basis, the neural network is trained to obtain a real-time optimization control scheme for the signal lights of the urban road network, which can improve the traffic efficiency of the urban traffic road network.
[0004] The technical solution adopted by the present invention is:
[0005] A traffic signal control method for urban road networks based on multi-agent Actor-Critic and GRU, the specific steps are as follows:
[0006] S1. For large urban traffic networks, each intersection is considered as an agent in a multi-agent Actor-Critic network. Each agent has an independent Actor-Critic network. In a continuous period of time, at fixed sampling intervals, the number of vehicles entering each lane of the intersection is obtained through the checkpoints on the road. i , where i is the lane number at a single intersection, and then the number of vehicles at the same intersection on the road network is combined into the total number of vehicles at the corresponding intersection L kj, k represents the kth moment from the start of training, j = 1, 2, ..., N, N is the total number of intersections in the road network; at the same time, the average speed T of all vehicles at the intersection at time k is recorded kj and signal light status information A kj ;
[0007] S2, data preprocessing, deleting illogical abnormal checkpoint data;
[0008] S3, build a road network simulation model for the neural network training platform;
[0009] S4: The total number of vehicles L at each intersection obtained in step S1 kj and the average speed of all vehicles T kj Combined into status information S kj and action information, i.e., signal light status information A kj , input Actor-Critic network for training;
[0010] S5. The training model finally obtained can determine the next state of the light according to the traffic information and signal light status information of each intersection through the following formula: (k+1)j =argmax(Q(S kj ,A kj ))
[0011] Where Q(S kj ,A kj ) is the Q value of a single Actor-Critic network in a multi-agent network, and the argmax function is Q(S kj ,A kj )The value of the variable when it reaches its maximum value.
[0012] Furthermore, the abnormal data in step S2 includes the number of vehicles being too large or being an outlier in the time series, and the signal light state being a null value.
[0013] Furthermore, in step S3, the road network simulation model is simulated using a microscopic road network simulation model. The advantage of a microscopic road network model over a mesoscopic simulation model and a macroscopic simulation model is that the restoration degree is higher and the real road network data can be more truly reflected.
[0014] Furthermore, the data of the micro road network model includes: physical property data of all traffic elements of the road network, and spatiotemporal distribution data of traffic flow in the road network.
[0015] Furthermore, the physical characteristic data of all traffic elements of the road network are collected by collecting the original map data of the vector map, eliminating the redundant data irrelevant to traffic, and then importing all the data of traffic elements into the microscopic road network simulation model, and then constructing the relevant simulation road network map. It can achieve accurate restoration of the actual 1:1 road network.
[0016] Furthermore, the spatiotemporal distribution data of the road network traffic flow is allocated through OD estimation simulation, and the specific steps include:
[0017] The first step is to use the traffic flow data in the license plate recognition data of the checkpoint to obtain the total traffic flow X of the checkpoint;
[0018] The second step is to use the image data of the checkpoint to analyze the proportion of trips through the checkpoint in the community q i,j ,q i,j It means that the community enters the road network from the i-th lane of the j-th intersection in the road network. This data can also be obtained through field surveys, where j = 1, 2, ..., N, N is the total number of intersections in the road network, and i is the lane number on a single intersection;
[0019] The third step is to solve the equation
[0020]
[0021] Get the traffic flow p of the community distribution ij ;
[0022] The fourth step is to solve the traffic flow distribution of all the checkpoints involved in the community, so as to obtain the OD pairs of all sections of the large-scale urban traffic network, and assign the distribution weights of the community traffic flow distribution in the road network simulation model, so as to build a traffic simulation platform that conforms to the actual road network rules for neural network training. OD estimation can effectively restore the changes in vehicle flow, and when the data integrity reaches 90%, the estimation error can be basically ignored.
[0023] Further, in step 4, the state information S is used kj , initialize the Actor-Critic neural network weight parameters θ, ω, and the expected objective function of maximizing the cycle reward of the Actor is Among them, θ is the strategy, π θ (S kj ) is the policy function describing the probability of action selection, r kj (S kj ) is the reward for a single agent in one simulation step:
[0024]
[0025] where d m Indicates the shortest distance between the mth intersection and the current jth intersection (the distance is the number of roads between the two intersections).
[0026] Further, the specific steps of training in step S4 are as follows:
[0027] S41, set up the Actor-Critic network structure: the Actor network consists of 3 layers of fully connected layers with decreasing neurons and a GRU network. The state S kj First, the fully connected layer is input, and then the output of the fully connected layer is used as the input of the GRU network. The GRU network outputs action information A kj ; The neural network structure of Critic also consists of three layers of fully connected layers with decreasing number of neurons and a GRU network. kj First, input the fully connected layer, and then use the output of the fully connected layer as the input of the GRU network. The GRU network outputs the time difference evaluation δ of the state;
[0028] S42, based on the traffic information set of the current intersection {S kj ,A kj}, reward value r (k+1)j And the traffic information set of the next training moment {S (k+1)j ,A (k+1)j}, calculate the time difference evaluation δ of the Actor strategy, and update the Critic's network parameter ω, β is the learning rate of the Critic network, and the ω update formula is as follows:
[0029] δ←r (k+1)j +γQ(S (k+1)j ,A (k+1)j )-Q(S kj ,A kj )
[0030] ω=ω+βδ
[0031] S43, based on the traffic information set of the current intersection {S kj ,A kj}, and the signal light status information A kj Update Actor network parameters θ:
[0032]
[0033] S44, continuously update the neural network on the simulation platform built in step S3 until the number of iterations reaches the preset maximum value K and then calculate the total reward of a training cycle (episode) Then enter the next training cycle;
[0034] S45, after completing all training cycles, select the neural network of the training cycle corresponding to the largest total reward as the final model and save it.
[0035] Compared with the prior art, the present invention has the following significant advantages: by real-time detection of real-time traffic information at each intersection in a large-scale traffic network, a traffic signal control method for a large urban road network designed based on multi-agent Actor-Critic and GRU has better traffic operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 It is a flow chart of the urban road network traffic signal control method based on multi-agent Actor-Critic and GRU of the present invention.
[0037] Figure 2 It is a real road network schematic diagram of a certain district in Hangzhou City of the present invention.
[0038] Figure 3 It is a schematic diagram of a simulated road network corresponding to the present invention. DETAILED DESCRIPTION
[0039] The present invention is further described below in conjunction with specific embodiments, but the present invention is not limited to these specific embodiments. Those skilled in the art should recognize that the present invention covers all possible alternatives, improvements and equivalents within the scope of the claims.
[0040] See also Figure 1 , Figure 2 This embodiment provides a method for controlling traffic signals in a large urban road network based on multi-agent Actor-Critic and GRU. The specific steps are as follows:
[0041] S1. For large urban traffic networks, each intersection is considered as an agent in a multi-agent Actor-Critic network. Each agent has an independent Actor-Critic network. In a continuous period of time, at fixed sampling intervals, the number of vehicles entering each lane of the intersection is obtained through the checkpoints on the road. i , where i is the lane number at a single intersection, and then the number of vehicles at the same intersection on the road network is combined into the total number of vehicles at the corresponding intersection L kj , k represents the kth moment from the start of training, j = 1, 2, ..., N, N is the total number of intersections in the road network; at the same time, the average speed T of all vehicles at the intersection at time k is recorded kj and signal light status information A kj According to traffic flow theory, the three basic quantities of traffic flow are flow rate, density and speed. If any two of them are known, the third one can be obtained. Therefore, the density vector and speed vector used in the present invention can be considered to contain complete information about the traffic flow.
[0042] S2. Data preprocessing, deleting illogical abnormal checkpoint data; abnormal data includes excessive number of vehicles or outliers in the time series, traffic light status is null, etc.
[0043] S3. Build a road network simulation model to serve as a training platform for neural networks; in order to restore the real road network environment as much as possible, all road network models are simulated using a microscopic road network model. The advantage of a microscopic road network model over a mesoscopic simulation model and a macroscopic simulation model is that it has a higher degree of restoration and can more truly reflect the real road network data. The data of the microscopic road network model include: 1. Physical characteristic data of all traffic elements of the road network: The physical characteristic data of all traffic elements of the road network include the longitude and latitude of each lane and intersection, and the specific location information of traffic lights at intersections. The physical characteristic data of all traffic elements of the road network are obtained by collecting the original map data of the vector map, eliminating redundant data irrelevant to traffic, importing all the data of the traffic elements into the microscopic simulation model, and then constructing the relevant simulated road network map. It can achieve an accurate restoration of the actual 1:1 road network, see Figure 2 , Figure 3 . 2. The spatiotemporal distribution data of road network traffic flow is simulated and allocated through the vehicle's origin-destination (OD) estimation. For large urban road networks, the present invention adopts OD estimation in four steps. Taking a single checkpoint as an example, the OD estimation step includes:
[0044] The first step is to use the traffic flow data from the license plate recognition data of the checkpoint to obtain the total traffic flow X of the checkpoint; the second step is to use the image data analysis of the checkpoint to obtain the travel ratio q of the community passing through the checkpoint i,j ,q i,j It means that the community enters the road network from the i-th lane of the j-th intersection in the road network. This data can also be obtained through field research, where j = 1, 2, ..., N, N is the total number of intersections in the road network, and i is the lane number on a single intersection; the third step is to solve the equation
[0045]
[0046] Get the traffic flow p of the community distribution ij ; The fourth step is to solve the traffic flow distribution of all checkpoints involved in the community, so as to obtain the OD pairs of all sections of the large-scale urban traffic network, and assign the distribution weights of the community traffic flow distribution in the road network simulation model, so as to build a traffic simulation platform that conforms to the actual road network rules for neural network training. OD estimation can effectively restore the changes in vehicle flow, and when the data completeness reaches 90%, the estimation error can be basically ignored.
[0047] S4: Combine the real-time traffic information of the road network obtained in step S1 into the state information S in Actor-Critic trainingkj and the signal light action information, i.e., the signal light status information A kj , conduct Actor-Critic training;
[0048] Using the state information S that can fully describe the vehicle congestion information at the intersection kj , initialize the weight parameters θ, ω of the Actor-Critic neural network. The expected objective function of maximizing the periodic reward of the Actor is Among them, θ is the strategy, π θ (S kj ) is the policy function describing the probability of action selection, r kj (S kj ) is the reward for a single agent in one simulation step:
[0049]
[0050] where d m Indicates the shortest distance between the mth intersection and the current jth intersection (the distance is the number of roads between the two intersections).
[0051] The specific steps of training are as follows:
[0052] S41, setting the training parameters of the Actor-Critic network. In this embodiment, the discount factor γ is set to 0.98, and the learning rates α and β of the Actor and Critic are both 0.001; according to different traffic information inputs, the Actor network is composed of a fully connected layer with three layers of neurons showing a decreasing trend and a GRU network. In this embodiment, the number of neurons corresponding to the three fully connected layers of the Actor is set to 300-200-100, and the number of neurons of the GRU is 64; the neural network structure of the Critic is also composed of a fully connected layer with three layers of neurons showing a decreasing trend and a GRU network. In this embodiment, the number of neurons in the fully connected layer is set to 100-50-25, and the number of neurons in the GRU network is 32;
[0053] S42, based on the traffic information set of the current intersection {S kj ,A kj}, reward value r (k+1)j And the traffic information set of the next training moment {S (k+1)j ,A (k+1)j}, calculate the time difference (TD) δ of the evaluation Actor strategy, and update the Critic's network parameter ω, β is the learning rate of the Critic network, and the ω update formula is as follows:
[0054] δ←r (k+1)j +γQ(S (k+1)j ,A (k+1)j )-Q(Skj ,A kj )
[0055] ω=ω+βδ
[0056] S43, based on the traffic information set of the current intersection {S kj ,A kj}, and the signal light status information A kj Update the Actor network parameters θ:
[0057]
[0058] S44, continuously update the neural network on the simulation platform built in step S3 until the number of iterations reaches the preset maximum value K = 50000 and then calculate the total reward of a training cycle (episode) Then enter the next training cycle.
[0059] S45, after all training cycles are completed, the neural network of the training cycle corresponding to the largest total reward is selected as the final model and saved. In this embodiment, the number of training cycles is set to 1000 times.
[0060] S5. The training model finally obtained can determine the light state of the next state according to the traffic information and signal light status information of each intersection through the following formula:
[0061] A (k+1)j =argmax(Q(S kj ,A kj ))
[0062] Where Q(S kj ,A kj ) is the Q value of a single Actor-Critic network in a multi-agent network, and the argmax function is Q(S kj ,A kj )The value of the variable when it reaches its maximum value.
[0063] The present invention first collects the number of vehicles, average speed and status of traffic lights in large urban road networks and preprocesses them. Then, a high-fidelity simulation platform is built with the collected road network information and vehicle passing information in a 1:1 ratio. Then, based on the preprocessed data set and considering the communication between adjacent intersections, each intersection is trained based on Actor-Critic and GRU. According to the final training convergence results, the optimal timing scheme for traffic lights under different intersection states is obtained, which improves the traffic efficiency of urban roads and can adapt to the random changes in traffic flow.
[0064] This embodiment uses a large road network with 46 intersections generated based on a vector map in the same proportion in the simulation software SUMO as an example for simulation. The above method is used to obtain a traffic signal control method for a large urban road network based on multi-agent Actor-Critic and GRU. The results show that the average waiting time of vehicles is reduced by 23% compared with before, especially during peak traffic hours.
[0065] The present invention detects the real-time traffic information of each intersection in the large-scale traffic network in real time, and designs a traffic signal control method for a large urban road network based on multi-agent Actor-Critic and GRU, which has better traffic operation efficiency.
[0066] The above description is about the excellent effect shown by an embodiment of the present invention. Obviously, the present invention is not only suitable for the above embodiment, but also can be implemented with various changes without departing from the basic spirit of the present invention and without exceeding the content involved in the essential content of the present invention.
Claims
1. A traffic signal control method for urban road networks based on multi-agent Actor-Critic and GRU, the specific steps are as follows: S1. For large urban traffic networks, each intersection is considered as an agent in a multi-agent Actor-Critic network. Each agent has an independent Actor-Critic network. In a continuous period of time, at fixed sampling intervals, the number of vehicles entering each lane of the intersection is obtained through the checkpoints on the road. i , where i is the lane number at a single intersection, and then the number of vehicles at the same intersection on the road network is combined into the total number of vehicles at the corresponding intersection L kj , k represents the kth moment from the start of training, j = 1, 2, ..., N, N is the total number of intersections in the road network; at the same time, the average speed T of all vehicles at the intersection at time k is recorded kj and signal light status information A kj ; S2, data preprocessing, deleting illogical abnormal checkpoint data; S3, build a road network simulation model for the neural network training platform; S4: The total number of vehicles L at each intersection obtained in step S1 kj and the average speed of all vehicles T kj Combined into status information S kj and action information, i.e., signal light status information A kj , input Actor-Critic network for training; The state information S is used kj , initialize the Actor-Critic neural network weight parameters θ, ω, and the expected objective function of maximizing the cycle reward of the Actor is in, θ is the strategy, π θ (S kj ) is the policy function describing the probability of action selection, r kj (S kj ) is the reward for a single agent in one simulation step: Where dm represents the shortest distance between the mth intersection and the current jth intersection; S5. The training model finally obtained is based on the traffic information and signal light status information of each intersection. The next state of the light can be determined by the following formula: A (k+1)j =argmax(Q(S kj ,A kj )) Where Q(S kj ,A kj ) is the Q value of a single Actor-Critic network in a multi-agent network, and the argmax function is Q(S kj ,A kj )The value of the variable when it reaches its maximum value.
2. The method for controlling traffic signals in a large urban road network based on multi-agent Actor-Critic and GRU according to claim 1, characterized in that: The abnormal data in step S2 includes the number of vehicles being too large or being an outlier in the time series, and the signal light status being a null value.
3. The method for controlling traffic signals in a large urban road network based on multi-agent Actor-Critic and GRU according to claim 1 is characterized by: In step S3, the road network simulation model is simulated using a microscopic road network simulation model.
4. The method for controlling traffic signals in a large urban road network based on multi-agent Actor-Critic and GRU according to claim 3 is characterized by: The data of the microscopic road network model include: the physical property data of all traffic elements of the road network and the spatiotemporal distribution data of the road network traffic flow.
5. The method for controlling traffic signals in a large urban road network based on multi-agent Actor-Critic and GRU according to claim 4 is characterized by: The physical characteristic data of all traffic elements of the road network are obtained by collecting the original map data of the vector map, eliminating the redundant data irrelevant to the traffic, importing all the data of the traffic elements into the microscopic road network simulation model, and then constructing the relevant simulation road network map.
6. The method for controlling traffic signals in a large urban road network based on multi-agent Actor-Critic and GRU according to claim 4 is characterized by: The spatiotemporal distribution data of the road network traffic flow is allocated through OD estimation simulation, and the specific steps include: The first step is to use the traffic flow data in the license plate recognition data of the checkpoint to obtain the total traffic flow X of the checkpoint; The second step is to use the image data of the checkpoint to analyze the proportion of trips through the checkpoint in the community q i,j ,q i,j It means that the community enters the road network from the i-th lane of the j-th intersection in the road network, where j = 1, 2, ..., N, N is the total number of intersections in the road network, and i is the lane number on a single intersection; The third step is to solve the equation Get the traffic flow p of the community distribution ij ; The fourth step is to solve the traffic flow distribution in the communities involved in all checkpoints, so as to obtain the OD pairs of all sections of the community in the large-scale urban traffic network, and assign the distribution weights of the traffic flow distribution in the community in the road network simulation model, so as to build a traffic simulation platform that conforms to the actual road network rules for the training of neural networks.
7. The method for controlling traffic signals in a large urban road network based on multi-agent Actor-Critic and GRU according to claim 1 is characterized by: The specific steps of training in step S4 are as follows: S41, set up the Actor-Critic network structure: the Actor network consists of 3 layers of fully connected layers with decreasing neurons and a GRU network. The state S kj First, the fully connected layer is input, and then the output of the fully connected layer is used as the input of the GRU network. The GRU network outputs action information A kj ; The neural network structure of Critic also consists of three layers of fully connected layers with decreasing number of neurons and a GRU network. kj First, input the fully connected layer, and then use the output of the fully connected layer as the input of the GRU network. The GRU network outputs the time difference evaluation δ of the state; S42, based on the traffic information set of the current intersection {S kj ,A kj }, reward value r (k+1)j And the traffic information set of the next training moment {S (k+1)j ,A (k+1)j }, calculate the time difference evaluation δ of the Actor strategy, and update the Critic's network parameter ω, β is the learning rate of the Critic network, and the ω update formula is as follows: δ←r (k+1)j +γQ(S (k+1)j ,A (k+1)j )-Q(S kj ,A kj ) ω=ω+βδ S43, based on the traffic information set of the current intersection {S kj ,A kj }, and the signal light status information A kj Update Actor network parameters θ: S44, continuously update the neural network on the simulation platform built in step S3 until the number of iterations reaches the preset maximum value K and then calculate the total reward of a training cycle episode Then enter the next training cycle, S45, after completing all training cycles, select the neural network of the training cycle corresponding to the largest total reward as the final model and save it.