Multi-intersection traffic signal control method based on deep reinforcement learning

Abstract

A kind of multi-intersection traffic signal control method based on deep reinforcement learning, the traffic light in road network is regarded as an agent, the agent runs a deep reinforcement learning algorithm, the phase selection of traffic light is trained according to the specific situation of multi-intersection, and the traffic flow can be adjusted by adjusting the phase selection of traffic light. According to the traffic condition of five-way intersection, the action with the highest score value is obtained in the deep reinforcement learning algorithm, so as to adjust the phase of traffic light, and continuously adjust the action, so that the traffic flow in the road network is optimally adjusted, the traffic efficiency of the road is optimally ensured, and the invention can also be applied to 8-phase and 6-phase intersections after appropriate simplification. The invention solves the problems of long vehicle waiting time, high lane occupancy rate and unreasonable model training.

Description

TECHNICAL FIELD

[0001] The application relates to a traffic signal control method, in particular to a multi-intersection traffic signal control method based on deep reinforcement learning. BACKGROUND

[0002] With the development of technology and economy in today's society, people's life gradually improves, and cars gradually become the means of human transportation. This leads to traffic congestion as a more serious traffic problem in current society. For example: waiting time is too long, lane occupancy rate is too high, and the like. Therefore, improving traffic efficiency has become a problem worth paying attention to.

[0003] In the field of traffic signal control, the traditional traffic signal control method is basically based on manual adjustment of the length of signal conversion. Due to the inflexibility of this method, the solution to the problem is not very ideal. Therefore, the prior art needs a method that can quickly and effectively control traffic signals to deal with traffic congestion problems, and needs a method that can be trained and learned for different traffic conditions, so that the signal light makes accurate decisions, so that the traffic flow in the road network is optimally regulated, and the road traffic efficiency is optimized. SUMMARY

[0004] The application provides a multi-intersection traffic signal control method based on deep reinforcement learning, which effectively solves the problems of long vehicle waiting time, high lane occupancy rate, and unreasonable model training, and provides a new idea for the subsequent traffic control field.

[0005] The technical solution of the application is implemented as follows:

[0006] A multi-intersection traffic signal control method based on deep reinforcement learning, characterized by comprising the following steps:

[0007] Step 1: define state space S, action space A, reward value function r, design vehicle file and build simulation environment: state space S is specifically the position of vehicles and pedestrians on each lane at the intersection, the length of the vehicle queue at the intersection, and the vehicle speed, wherein the state space S is the position of all vehicles and pedestrians on the lane at the current decision point, the state s (s e S) is a binary one-dimensional array, wherein the state s is the position of a single vehicle or pedestrian, the queue length and the vehicle speed are obtained from the simulation software in real time; the action space A is specifically the phase of signal control; the reward function r is specifically the weighted sum of vehicle waiting time, queue length, vehicle speed and pedestrian waiting time, and the reward value function r obtained is:

[0008] r = mu1 * r queue + mu2 * r v_waitTime + mu3 * r speed + mu4 * rp_waitTime (1)

[0009] where μ1, μ2, μ3, μ4 are the weights of the reward function, the sum of the weights is 1, r queue denotes the total length of the vehicle queue at each intersection as the decision point, r v_waitTime denotes the sum of the waiting time of all vehicles at the current time, r speed denotes the average speed of the current vehicle, r p_waitTime denotes the waiting time of pedestrians at the intersection; the vehicle file is specifically the traffic data used for training the traffic control model; the simulation environment is specifically a three-lane five-way intersection designed in the SUMO simulation software; the intersection conditions are divided into three types; five-way intersection and 10-phase, respectively, straight right turn north, straight right turn south, north-south straight, north-south left turn, straight right turn east 1, straight right turn east 2, straight right turn west, west east 1 straight, west east 2 straight, east-west left; 8-phase crossroads; 6-phase three-way intersection;

[0010] Step two: initialize neural network parameters, experience pool D, other parameters: initialize neural network parameters to define the main network Q of the double deep Q network and the target network Q', and initialize the network parameters θ of the main network Q, and assign Q network parameters to Q', θ→θ', where the main network Q is realized by neural network technology and double deep Q network technology, and the neural network connection mode is full connection, which has 8 hidden layers to estimate the value of the current state. The target network Q' is updated from the main network Q, θ is the neural network parameter of the main network Q, and θ' is the neural network parameter of the target network Q; the experience pool D is the data record stored in the training process of the main network Q, which marks the priority of the stored record to form a priority experience pool D; other parameters are the total iteration number T of model training, the decay factor γ of the reward of the next state, the exploration rate ε that determines the random selection of action, the target network Q' update frequency p, the maximum number of steps Step of simulation, and the number of samples m randomly sampled each time when training the main network Q;

[0011] Step three: the main network stores the data record to the experience pool and sets priority to the data record, takes m records from the priority experience pool according to the priority, updates the main network parameters, then judges whether the target network parameter condition is met, and then judges whether the total iteration number is completed, specifically, the road environment is run using SUMO simulation software, the vehicle data is imported for simulation: using the traci interface in SUMO, the state S is obtained, input to the current main network Q, the corresponding Q value is calculated, then the action A corresponding to the current state is selected using the ε-greedy algorithm, then the selected action is executed to obtain the new state S' and the reward R, and {S, A, R, S'} is stored in the experience pool D, and the priority is marked for the record, if the experience pool D has reached the maximum storage capacity, the first stored record is deleted, then the y of the current Q network is calculated j Specifically, m samples are randomly sampled from D, {S j , A j , R j , S j '} j = 1, 2, 3...m, through the formula:

[0012] y j = R j + γQ'(S j ', argmax a Q(S j , A j ; θ) ; θ' ) (2)

[0013] The y j is calculated, where y j is the true Q value, γ is the decay factor, S j is the state space at time j, S j ' is the state space at time j+1, R j is the reward value at time j, A j is the action made at time j, θ is the main network parameter, θ' is the target network parameter; then using the MSE mean square loss function, the loss value Loss is calculated, the parameters θ are updated by back propagation, if t% p == 0: θ→θ', the state S' = S is updated at the same time, where t is the current training step number, p is the target network update frequency, S is the current state space, S' is the next state space; the final traffic signal control model is iteratively trained by this step;

[0014] Step four: according to the traffic signal control model obtained after iterative training in step three, apply to multiple intersections: apply the trained traffic signal control model to multiple intersections, and according to the traffic state S in the traffic environment at different time periods obtained by the SUMO simulation software, input the state S into the traffic signal control model, the model will select a reasonable phase, thereby obtaining the next traffic state S', and repeating this step.

[0015] Compared with the prior art, the advantages of the present application are obvious, mainly in:

[0016] 1、The model state input of the present application is not only the vehicle queue length and vehicle speed, but also the vehicle position and pedestrian state, including more comprehensive and more realistic traffic conditions, greatly improving the accuracy of the model;

[0017] 2、The present application can be applied to a 10-phase five-way intersection, and can also be applied to an 8-phase cross intersection and a 6-phase three-way intersection through a simplified model, changing the limitations brought by the phase;

[0018] 3、The reward function of the present application includes queue length, vehicle speed, vehicle waiting time, and pedestrian waiting time, and the reward function is more accurate for signal decision, and optimizes the incorrect decision of the prior art for pedestrians, more vehicles and less roads;

[0019] 4、The present application adds a weight priority to the data records in the experience pool, and when using a batch processing strategy, the data records with better influence are extracted with a high probability, changing the multiple use of the data records with poor influence in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0020] The present application has a total of Figure 5 sheets.

[0021] Figure 1 It is a five-way intersection diagram of the present application;

[0022] Figure 2 It is a cross intersection diagram of the present application;

[0023] Figure 3 It is a three-way intersection diagram of the present application;

[0024] Figure 4 It is a model structure diagram of the present application;

[0025] Figure 5 It is a model control training flowchart of the present application. DETAILED DESCRIPTION

[0026] As Figure 1 , 2The method shown in claim 1, 2, 3, 4, 5 is characterized in that it comprises the following steps:

[0027] Step one: defining state space S, action space A, reward value function r, designing vehicle file and building simulation environment: state space S is specifically the position of vehicles and pedestrians on each lane at the intersection, the queue length of vehicles at the intersection, and the speed of vehicles, wherein the state space S is the position of all vehicles and pedestrians on the lane at the current decision point, the state s (s∈S) is a binary one-dimensional array, wherein the state s is the position of a single vehicle or pedestrian, the queue length and the speed of vehicles are obtained from the simulation software in real time; the action space A is specifically the phase of signal control; the reward function r is specifically the weighted sum of vehicle waiting time, queue length, vehicle speed and pedestrian waiting time, and the reward value function r is:

[0028] r = μ1*r queue + μ2*r v_waitTime + μ3*r speed + μ4*r p_waitTime (1)

[0029] Wherein μ1, μ2, μ3, μ4 are the weights of the reward function, the sum of the weights is 1, r queue represents the total length of the vehicle queue at each intersection at the decision point, r v_waitTime represents the sum of the waiting time of all vehicles at the current decision point, r speed represents the average speed of the current vehicle, r p_waitTime represents the waiting time of the pedestrian at the intersection at the decision point; the designed vehicle file is specifically the traffic flow data used for training the traffic control model; the built simulation environment is specifically the intersection of a three-lane five-way intersection designed in the SUMO simulation software; the intersection conditions are divided into three types; the five-way intersection has 10 phases, which are straight right turn north, straight right turn south, north-south straight, north-south left turn, straight right turn east 1, straight right turn east 2, straight right turn west, west-east 1 straight, west-east 2 straight, and east-west left; the eight-phase crossroad; the six-phase three-way intersection;

[0030] Step 2: Initialize Neural Network Parameters, Experience Pool D, and Other Parameters: Initializing the neural network parameters involves defining the main network Q and the target network Q′ of the dual deep Q-network competition. Simultaneously, the network parameters θ of the main network Q are initialized and assigned to Q′, θ→θ′. The main network Q is implemented using neural network technology and dual deep Q-network technology, with fully connected connections and 8 hidden layers to estimate the value of the current state. The target network Q′ is updated from the main network Q. θ represents the neural network parameters of the main network Q, and θ′ represents the neural network parameters of the target network Q. The experience pool D stores data records from the training process of the main network Q, prioritizing these records to form a priority experience pool D. Other parameters include the total number of iterations T during model training, the decay factor γ that reduces the reward for the next state, the exploration rate ∈ that determines the random action selection, the update frequency p of the target network Q′, the maximum number of steps in the simulation Step, and the number of samples m randomly sampled each time the main network Q is trained.

[0031] Step 3: Train the main network by storing data records in the experience pool and assigning weights and priorities to the data records. Retrieve m records from the priority experience pool according to their priorities to update the main network parameters. Then, determine if the conditions for updating the target network parameters are met, and then determine if the total number of iterations has been completed. Specifically, use SUMO simulation software to run a road environment and import vehicle data for simulation: Use the built-in traci interface in SUMO to obtain the state S, input it into the current main network Q, calculate the corresponding Q value, and then use the ∈-greedy algorithm to select the action A corresponding to the current state. Execute the selected action to obtain the new state S′ and reward R, and store {S, A, R, S′} in the experience pool D. Simultaneously, mark the priority of the records. If the experience pool D has reached its maximum storage capacity, delete the earliest stored record. Then calculate the y of the current Q network. j Specifically, m samples are randomly selected from D, {S} j A j R j S j ′}, j=1,2,3...m, through the formula:

[0032] y j =R j +γQ′(S j ', argmax a Q(S j A j ;θ);θ′) (2)

[0033] Calculate y j , where y j The true Q value is given by γ, where γ is the decay factor, and S is the S value. j Let S be the state space at time j. jS' is the state space at j+1 time, R j R is the reward value at j time, A j A is the action made at j time, θ is the main network parameter, θ' is the target network parameter; then use the MSE mean square loss function to calculate the loss value Loss, and update the parameters θ by back propagation, if t%p==0: θ→θ', and update the state S'=S, where t is the current training step number, p is the target network update frequency, S is the current state space, and S' is the next state space; iterate the final traffic signal control model by this step;

[0034] Step four: according to the traffic signal control model obtained by iterative training in step three, apply it to multiple intersections: apply the trained traffic signal control model to multiple intersections, and according to the traffic state S in different time periods obtained by the SUMO simulation software, input the state S into the traffic signal control model, and the model will select a reasonable phase to obtain the next traffic state S', and repeat this step.

Claims

1. A traffic signal control method for multiple intersections based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Define the state space S, action space A, and reward function r; design vehicle files and build the simulation environment: The state space S specifically represents the positions of vehicles and pedestrians in each lane of the intersection, the queue length, and vehicle speed. State S represents the positions of all vehicles and pedestrians in the lanes at the current decision point. The states s (s∈S) are binary one-dimensional arrays, where each state represents the position of a single vehicle or pedestrian. Queue length and vehicle speed are obtained in real-time from the simulation software. The action space A specifically represents the phase of the signal control. The reward function r is the reward value function obtained by weighted summation of vehicle waiting time, queue length, vehicle speed, and pedestrian waiting time. r=μ1 *r queue +μ2*r v_waitTime +μ3*r speed +μ4*r p_waitTime (1) Where μ1, μ2, μ3, and μ4 are the weights of the reward function, and the sum of the weights is 1. queue Let r represent the total length of vehicle queues at each intersection at the decision point. v_waitTime Let r represent the sum of the waiting times of all vehicles currently in use. speed Let r represent the current average speed of the vehicle. p_waitTime This represents the pedestrian waiting time at the intersection at this decision point; the vehicle design file specifically refers to the traffic flow data used for training the traffic control model; the simulation environment is specifically designed in the SUMO simulation software as a three-lane, five-entry intersection; the intersection conditions are divided into three types: a five-entry intersection with 10 phases, namely, northbound straight and right turn, southbound straight and right turn, north-south straight, north-south left turn, eastbound 1 straight and right turn, eastbound 2 straight and right turn, westbound straight and right turn, west-east 1 straight, west-east 2 straight, east-west left turn; an 8-phase crossroad; and a 6-phase three-way intersection. Step 2: Initialize Neural Network Parameters, Experience Pool D, and Other Parameters: Initializing the neural network parameters involves defining the main network Q and the target network Q′ of the dual deep Q-network competition. Simultaneously, the network parameters θ of the main network Q are initialized and assigned to Q′, θ→θ′. The main network Q is implemented using neural network technology and dual deep Q-network technology, with fully connected connections and 8 hidden layers to estimate the value of the current state. The target network Q′ is updated from the main network Q. θ represents the neural network parameters of the main network Q, and θ′ represents the neural network parameters of the target network Q. The experience pool D stores data records from the training process of the main network Q, assigning priorities to the stored records to form a priority experience pool D. Other parameters include the total number of iterations T for model training, the decay factor γ for reducing the reward in the next state, the exploration rate ∈ that determines the random action selection, the update frequency p of the target network Q′, the maximum number of steps in the simulation Step, and the number of samples m randomly sampled each time when training the main network Q. Step 3: Train the main network by storing data records in the experience pool and assigning weights and priorities to the data records. Retrieve m records from the priority experience pool according to their priority to update the main network parameters. Then, determine if the conditions for updating the target network parameters are met, and then determine if the total number of iterations has been completed. Specifically, use SUMO simulation software to run a road environment and import vehicle data for simulation: Use the built-in traci interface in SUMO to obtain the state S, input it into the current main network Q, calculate the corresponding Q value, and then use the ∈-greedy algorithm to select the action A corresponding to the current state. Execute the selected action to obtain the new state S′ and reward R, and store {S, A, R, S′} in the experience pool D. Simultaneously, mark the records with priorities. If the experience pool D has reached its maximum storage capacity, delete the earliest stored record. Then calculate the y of the current Q network. j Specifically, m samples are randomly selected from D, {S} j A j ,R j ,S j ′}, j=1,2,3...m, through the formula: y j =R j +γQ′(S j ′,argmax a Q(S j ,A j ;θ);θ′) (2) Calculate y j , where y j The true Q value is given by γ, where γ is the decay factor, and S is the S value. j Let S be the state space at time j. j Let ′ be the state space at time j+1, and R be the state space. j Let A be the reward value at time j. j For the action taken at time j, θ is the principal network parameter and θ′ is the target network parameter. Then, the mean squared loss function (MSE) is used to calculate the loss value Loss, and the parameters θ are updated by backpropagation. If t%p==0∶θ→θ′, the state S′=S is updated at the same time, where t is the current training step, p is the target network update frequency, S is the current state space, and S′ is the next state space. The final traffic signal control model is trained iteratively in this manner. Step 4: Apply the traffic signal control model obtained after iterative training in Step 3 to the multi-intersection: Apply the trained traffic signal control model to the multi-intersection, and at the same time obtain the traffic flow state S under different time periods according to the SUMO simulation software. Input the state S into the traffic signal control model, and the model will select a reasonable phase to obtain the next traffic state S′. Repeat this step.

Images