A dynamic maximum pressure traffic signal control method based on reinforcement learning

Abstract

The present invention relates to a dynamic maximum pressure traffic signal control method based on reinforcement learning, and belongs to the field of traffic technology. The method includes traffic flow data collection and real-time traffic status observation; establishing a dynamic maximum pressure traffic signal control method that dynamically associates dynamic traffic pressure parameters with upstream and downstream pressures; establishing a dynamic maximum pressure traffic signal control model based on reinforcement learning; simulation environment and parameter setting, training and testing of the dynamic maximum pressure traffic signal control model; dynamic maximum pressure traffic signal timing optimization strategy. The present invention combines the advantages of data-driven methods and theoretical control models, and optimizes the maximum pressure control model based on deep reinforcement learning technology with highly adaptive characteristics. The present invention overcomes the deficiency of the maximum pressure signal control model that only considers a single pressure parameter as the pressure weight and the association between the upstream and downstream pressures of the intersection, and the model is more adaptable to complex and changeable traffic conditions.

Description

Technical Field

[0001] The invention relates to a dynamic maximum pressure traffic signal control method based on reinforcement learning, belonging to the technical field of traffic. Background Art

[0002] With the continuous increase in the number of motor vehicles and non-motor vehicles, traffic congestion has gradually become normalized. The use of advanced traffic signal control methods to regulate traffic flow is an important way to alleviate traffic congestion, improve travel efficiency, and help achieve carbon peak and carbon neutrality. Maximum pressure traffic signal control is an advanced distributed signal control method that can effectively prevent queue overflow, ensure that vehicles can pass through the network effectively and smoothly, and maximize the throughput of the road network. However, most of the existing maximum pressure traffic signal controls only consider a single traffic pressure parameter as the input of the maximum pressure controller, and the pressure correlation between adjacent intersections is not considered enough, making it difficult for maximum pressure signal control to adapt to complex and changeable traffic conditions.

[0003] At present, artificial intelligence algorithms such as reinforcement learning are a very promising data-driven method for solving the control of traffic lights in complex urban traffic networks. Therefore, in order to solve the existing limitations of maximum pressure, the present invention combines the advantages of data-driven methods and theoretical control models, establishes a dynamic maximum pressure traffic signal control model that dynamically associates dynamic traffic pressure parameters with upstream and downstream pressures, and optimizes the model using deep reinforcement learning technology with highly adaptive characteristics. The control strategy obtained by the existing reinforcement learning-based traffic signal control method is usually represented by a neural network. Due to the problem of the generalization ability of the neural network, it is not advisable to directly apply the reinforcement learning strategy learned offline in a simulated environment to the real world without additional adjustments to the trained model. The present invention combines the advantages of the theoretical control model and uses reinforcement learning to optimize the maximum pressure control model. The final control strategy is obtained from the optimized maximum pressure control model. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a dynamic maximum pressure traffic signal control method based on reinforcement learning, which is used to further optimize the maximum pressure traffic signal control method, formulate a signal control scheme that can flexibly adapt to complex and changeable traffic conditions, and study a highly adaptive reinforcement learning method to overcome the limitations of the maximum pressure signal control model that uses a single traffic pressure parameter as the pressure weight and lacks consideration of the pressure correlation between adjacent intersections.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a dynamic maximum pressure traffic signal control method based on reinforcement learning, comprising the following steps:

[0007] Step 1: Collect traffic flow data;

[0008] Step 2: Establish a dynamic maximum pressure traffic signal control method that dynamically associates dynamic traffic pressure parameters with upstream and downstream pressures;

[0009] Step 3: Establish a dynamic maximum pressure traffic signal control model based on reinforcement learning;

[0010] Step 4: Set the simulation environment and parameters to complete the training of the dynamic maximum pressure traffic signal control model based on reinforcement learning.

[0011] Furthermore, the traffic flow data is used for model training;

[0012] Furthermore, the step 2 comprises:

[0013] Step 2.1: Based on the variable cell transport model, four types of dynamic traffic pressure parameters are divided according to free flow density, congestion flow density and road section length. , expressed as:

[0014]

[0015]

[0016]

[0017]

[0018] in, is the pressure of congestion flow density; is the pressure of the combination of congestion flow density and free flow density; is the pressure of congestion flow density combined with the length of the road section; It is the pressure of the combination of congestion flow density and free flow density under the combined section length; is the congestion flow density pressure; is the free stream density pressure; , , Indicates the type of pressure; is the length of the road section; For time; , Number the road sections;

[0019] Step 2.2: According to the dynamic correlation of the pressure of the upstream and downstream adjacent intersections, use the upstream and downstream correlation parameters To characterize whether adjacent intersections are associated, The value of is 0 and 1, where 0 means no association and 1 means association;

[0020] Step 2.3: Using a fixed phase sequence, establish a dynamic maximum pressure traffic signal control method that dynamically associates dynamic traffic pressure parameters with upstream and downstream pressures. At the end of each signal cycle, calculate the green light time of each phase based on the pressure proportion of each phase.

[0021] Furthermore, the step 2.3 is specifically described as follows:

[0022] First calculate the pressure weight , at every time When , the controller calculates the pressure weight according to the selected traffic pressure parameters and upstream and downstream associated parameters:

[0023]

[0024] in, and is the traffic pressure parameter; is the type of pressure; , , Number the road sections; From the road section To section The traffic ratio, The starting section is The set of all downstream segments of ;

[0025] Calculate traffic pressure , with the maximum pressure weight Saturation flow rate The product of is used to express the traffic pressure, that is:

[0026]

[0027] Calculate phase pressure , is the phase number, each phase contains a number of traffic flows, so the phase pressure is equal to the sum of the pressures of all traffic flows contained in the phase:

[0028]

[0029] in, Indicates phase Internal arrive Turning into traffic;

[0030] Calculate the green light time of each phase according to the pressure ratio of each phase for:

[0031]

[0032] in, , Phase and Phase The sum of all traffic pressures contained in it; Number the node; , is the phase number; is the signal period; is the phase set;

[0033] To ensure pedestrians can cross the street, the minimum green light time is set to , and set the maximum green light time to , then the phase green light time The final expression is:

[0034] .

[0035] Furthermore, the step 3 comprises:

[0036] Step 3.1: Reinforcement learning agent, including the design of states, actions, and rewards;

[0037] Step 3.2: Establish a dynamic maximum pressure traffic signal control model based on reinforcement learning, including a spatiotemporal feature extraction module, a deep Q network (DQN) action decision module, and a value decomposition hybrid network module.

[0038] Furthermore, the step 3.1 comprises:

[0039] The action is specifically designed to use the four types of discrete combinations of dynamic traffic pressure parameters and upstream and downstream associated parameters as reinforcement learning actions, and after the traffic pressure parameters and associated parameters are determined through DQN actions, they are input into the dynamic maximum pressure traffic signal control method dynamically associated with the dynamic traffic pressure parameters and upstream and downstream pressures, and the green light time of each phase is output;

[0040] The state is specifically designed as follows: the state includes the number of vehicles queuing at each entrance of the current intersection , vehicle density on the import road , the current intersection exit vehicle density , previous action and intersection number , the state is represented by a state vector state={ , , , , };

[0041] The design of the reward is specifically that the reward is a combination of the number of vehicles queued at the intersection in adjacent time steps and the intersection throughput:

[0042]

[0043] in, For The reward obtained after performing an action at the intersection at the current state, for The duration of the next cycle at time, is the total number of vehicles queuing at the intersection entrance, is the vehicle throughput at the intersection, is the weight coefficient.

[0044] Furthermore, the step 3.2 includes:

[0045] The spatiotemporal feature extraction module is composed of a graph attention network GAT and a gated recurrent unit GRU in series. The state features of each intersection are input into the spatiotemporal feature extraction module, and a new intersection state feature containing adjacent intersections and historical information is output, specifically:

[0046] The graph attention network GAT is specifically constructed as follows:

[0047] Input feature matrix X, shape is N×F, N is the number of intersections, F is the intersection state dimension; for each intersection a and adjacent intersection b, the corresponding state vector is and , calculate the attention coefficient between intersection a and adjacent intersection b for:

[0048]

[0049] in, is the attention mechanism, which is a learnable parameter vector; T represents matrix transpose; LeakyReLU is the activation function; is a learnable weight vector; || represents a matrix concatenation operation;

[0050] Use the Softmax function to normalize the attention coefficient and get the attention weight of the adjacent intersection b to the intersection a. for:

[0051]

[0052] in, is the attention coefficient between intersection a and the adjacent intersection b, is the attention coefficient between intersection a and the adjacent intersection c; exp is the natural exponential function; is the set of adjacent intersections of intersection a; c is the intersection number;

[0053] Take the weighted sum of the features of all adjacent intersections of intersection a to obtain a new feature representation of intersection a, and apply the sigmoid function to the new feature vector Activate and finally output the new feature matrix Z:

[0054]

[0055] in, The new traffic state containing adjacent intersection information learned by the graph attention network is To represent the set of neighbor intersections;

[0056] A multi-head attention mechanism is used, and each attention head outputs its own attention coefficient, so the features of multiple outputs are concatenated to obtain the fusion feature. :

[0057]

[0058] in, is the attention weight of the adjacent intersection b to the intersection a, is a learnable weight vector, Number the attention heads, , is the total number of attention heads;

[0059] The gated recurrent unit GRU is specifically constructed as follows:

[0060] The new features of each intersection output by the graph attention network As input, a hidden state containing historical traffic information is also input , get the reset gate signal and update gate signal :

[0061]

[0062]

[0063] in, The new features of each intersection output by the graph attention network; , , and is the weight matrix to be learned;

[0064] By resetting the goalkeeper's hidden state at the previous moment The relevant information in is discarded and combined with the new features of the current input Combine to obtain candidate hidden states ; Then, through the update gate, combined with the hidden state of the previous moment and candidate hidden states Get the current hidden state :

[0065]

[0066]

[0067] in, , is the weight matrix to be learned; tanh is the hyperbolic tangent activation function, and ⊙ represents element-wise multiplication;

[0068] The DQN action decision module is specifically constructed as follows:

[0069] The hidden state obtained by the spatiotemporal feature extraction module Input to the Q network to obtain the action value function in the current state , is the Q network parameter, and maximizes the reward R by selecting the action with the highest Q value at each time t;

[0070] The value decomposition hybrid network module includes a Q function summation network, which calculates the global Q function by inputting the Q functions of all intersections :

[0071]

[0072] in, For the The Q function of the intersection.

[0073] Furthermore, the step 4 comprises:

[0074] Step 4.1: compile a road network file according to the actual road network, the road network file includes intersections, roads, and traffic lights; compile a traffic flow file according to the synthetic traffic flow data and the real traffic flow data, the traffic flow file includes the vehicle number, vehicle type, maximum acceleration and deceleration, maximum speed, path information, and start time of each vehicle; create a simulation environment according to the road network file and the traffic flow file;

[0075] Step 4.2: Initialize hyperparameters, including exploration rate, learning rate, discount factor, experience pool sample size, random sampling sample size, training rounds, spatiotemporal feature extraction module, and DQN action decision module hidden layer size;

[0076] Step 4.3: Train the dynamic maximum pressure traffic signal control model based on reinforcement learning according to the simulation environment and the initialized hyperparameters.

[0077] Further, the step 4.3 comprises:

[0078] Interact the agent with the simulation environment to obtain the current state , a new fusion state is obtained through the spatiotemporal feature extraction module , pass the fusion state into the Q network to output the Q value of different actions, and then press - Greedy strategy selects actions. When making action decisions, the probability Randomly choose a random action with probability 1- Select the action that maximizes the Q value. The whole process is expressed as:

[0079]

[0080]

[0081]

[0082]

[0083] in, is the state of the intersection; is the new state of the intersection containing spatial feature information output by the graph attention network GAT; It is a new feature of the intersection containing temporal feature information output by the gated recurrent unit GRU; It is a Q network with intersections formed by fully connected layers FC; is the action; i is the intersection number; A is the number of discrete actions; is the parameter that needs to be optimized for the Q network; rand(0,1) means taking a random number in the range of 0 to 1; int(0,A) means randomly selecting a total of A integers from 0 to n-1;

[0084] Collect the trajectory experience during the interaction process, which is composed of state ,action ,award , next state Composition, through tuples Indicates that the trajectory experience is stored in the experience pool, which is a queue. When the experience pool capacity reaches the upper limit, the initially stored experience is discarded to store new experience. A batch of experience samples are randomly sampled from the experience pool for learning, and the target Q value of each experience sample is calculated. :

[0085]

[0086] in, is the discount factor; is the next action; are the learned model parameters;

[0087] Calculate the loss function and use backpropagation and gradient descent to update the network parameters of the graph attention network, gated recurrent unit, and DQN action module to minimize the loss function :

[0088]

[0089] in, is a batch experience sample; is the size of the batch experience sample; It is the batch sample status; for batch sample actions; is the batch sample number; is the target Q value; is the Q value of the global Q network.

[0090] The beneficial effects of the present invention are:

[0091] (1) The present invention adopts graph attention network and gated recurrent unit to construct spatiotemporal feature extraction module, fully considers the influence of adjacent intersections, and effectively extracts the road network status containing spatiotemporal features.

[0092] (2) The present invention utilizes a deep reinforcement learning technique with a high degree of adaptability to optimize the model, thereby overcoming the limitations of the maximum pressure signal control model, which uses a single traffic pressure parameter as the pressure weight and does not adequately consider the pressure correlation of adjacent intersections.

[0093] (3) The present invention combines the advantages of the theoretical control model and uses reinforcement learning to optimize the maximum pressure control model. The final control strategy is obtained from the optimized maximum pressure control model, which solves the problem of neural network generalization to a certain extent. This is of practical significance for practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0095] Figure 1 It is a schematic flow chart of the method of the present invention;

[0096] Figure 2 It is the overall method framework diagram of the present invention;

[0097] Figure 3It is a schematic diagram of a variable-length cellular transmission model consisting of two types of cells;

[0098] Figure 4 It is a schematic diagram of the intersection signal phase diagram;

[0099] Figure 5 It is a schematic diagram of maximum pressure phase pressure calculation;

[0100] Figure 6 It is the spatiotemporal feature extraction module;

[0101] Figure 7 It is a simulation training flow chart of the present invention;

[0102] Figure 8 It is the SUMO simulation road network diagram;

[0103] Fig. 9 It is the input demand profile distribution diagram of the training simulation environment;

[0104] Fig.10 It is the comparison of average queue length under the six control methods;

[0105] Fig.11 It is the comparison of average vehicle delay under the six control methods;

[0106] Fig.12 It is a comparison of the total vehicle throughput under the control of 6 methods. DETAILED DESCRIPTION

[0107] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0108] Embodiment 1: The present invention is used to disclose a dynamic maximum pressure traffic signal control method based on reinforcement learning. The overall method flow and framework are as follows: Figure 1 and Figure 2 As shown, the specific steps of the method are as follows:

[0109] Step 1: Collect traffic flow data;

[0110] The traffic flow data is used for subsequent model training;

[0111] Step 2: Establish a dynamic maximum pressure traffic signal control method that dynamically associates dynamic traffic pressure parameters with upstream and downstream pressures;

[0112] The step 2 includes three steps:

[0113] Step 1: Read Figure 3 , based on the variable cell transport model, according to the free flow density , congestion flow density and road section length, four types of dynamic traffic pressure parameters are divided , expressed as:

[0114]

[0115]

[0116]

[0117]

[0118] in, is the pressure of congestion flow density; is the pressure of the combination of congestion flow density and free flow density; is the pressure of congestion flow density combined with the length of the road section; It is the pressure of the combination of congestion flow density and free flow density under the combined section length; is the congestion flow density pressure; is the free stream density pressure; , , Indicates the type of pressure; is the length of the road section; For time; , Number the road segment.

[0119] Step 2: According to the dynamic correlation of the pressure of the upstream and downstream adjacent intersections, use the upstream and downstream correlation parameters To characterize whether adjacent intersections are associated, The value of is 0 and 1, where 0 means no association and 1 means association;

[0120] Step 3: Read Figure 4 and Figure 5 , a fixed phase sequence is adopted to establish a dynamic maximum pressure traffic signal control method that dynamically associates dynamic traffic pressure parameters with upstream and downstream pressures. At the end of each signal cycle, the green light time of each phase is calculated based on the pressure proportion of each phase.

[0121] First calculate the pressure weight , at every time When , the controller calculates the pressure weight according to the selected traffic pressure parameters and upstream and downstream associated parameters:

[0122]

[0123] in, and is the traffic pressure parameter; is the type of pressure; , , Number the road sections; From the road section To the road section The traffic ratio, The starting section is The set of all downstream segments of ;

[0124] Calculate traffic pressure , with the maximum pressure weight Saturation flow rate The product of is used to express the traffic pressure, that is:

[0125]

[0126] Calculate phase pressure , is the phase number, each phase contains a number of traffic flows, so the phase pressure is equal to the sum of the pressures of all traffic flows contained in the phase:

[0127]

[0128] in, Indicates phase Internal arrive Turning into traffic;

[0129] Calculate the green light time of each phase according to the pressure ratio of each phase for:

[0130]

[0131] in, , Phase and Phase The sum of all traffic pressures contained in it; Number the node; , is the phase number; is the signal period; is the phase set;

[0132] To ensure pedestrians can cross the street, the minimum green light time is set to , and set the maximum green light time to , then the phase green light time The final expression is:

[0133] .

[0134] Step 3: Establish a dynamic maximum pressure traffic signal control model based on reinforcement learning;

[0135] The step 3 includes two steps:

[0136] Step 1: Reinforcement learning agent, including the design of states, actions, and rewards;

[0137] The action is specifically designed to use the four types of discrete combinations of dynamic traffic pressure parameters and upstream and downstream associated parameters as reinforcement learning actions, and after the traffic pressure parameters and associated parameters are determined through DQN actions, they are input into the dynamic maximum pressure traffic signal control method dynamically associated with the dynamic traffic pressure parameters and upstream and downstream pressures, and the green light time of each phase is output;

[0138] The state is specifically designed as follows: the state includes the number of vehicles queuing at each entrance of the current intersection , vehicle density on the import road , the current intersection exit vehicle density , previous action and intersection number , the state is represented by a state vector state={ , , , , };

[0139] The design of the reward is specifically that the reward is a combination of the number of vehicles queued at the intersection in adjacent time steps and the intersection throughput:

[0140]

[0141] in, For The reward obtained after performing an action at the intersection at the current state, for The duration of the next cycle at time, is the total number of vehicles queuing at the intersection entrance, is the vehicle throughput at the intersection, is the weight coefficient.

[0142] Step 2: Read Figure 6 , a dynamic maximum pressure traffic signal control model based on reinforcement learning is established, including a spatiotemporal feature extraction module, a DQN action decision module and a value decomposition hybrid network module.

[0143] The spatiotemporal feature extraction module is composed of a graph attention network GAT and a gated recurrent unit GRU in series. The state features of each intersection are input into the spatiotemporal feature extraction module, and a new intersection state feature containing adjacent intersections and historical information is output, specifically:

[0144] The graph attention network GAT is specifically constructed as follows:

[0145] Input feature matrix X, shape is N×F, N is the number of intersections, F is the intersection state dimension; for each intersection a and adjacent intersection b, the corresponding state vector is and , calculate the attention coefficient between intersection a and adjacent intersection b for:

[0146]

[0147] in, is the attention mechanism, which is a learnable parameter vector; T represents matrix transpose; LeakyReLU is the activation function; is a learnable weight vector; || represents a matrix concatenation operation;

[0148] Use the Softmax function to normalize the attention coefficient and get the attention weight of the adjacent intersection b to the intersection a. for:

[0149]

[0150] in, is the attention coefficient between intersection a and the adjacent intersection b, is the attention coefficient between intersection a and the adjacent intersection c; exp is the natural exponential function; is the set of adjacent intersections of intersection a; c is the intersection number;

[0151] Take the weighted sum of the features of all adjacent intersections of intersection a to obtain a new feature representation of intersection a, and apply the sigmoid function to the new feature vector Activate and finally output the new feature matrix Z:

[0152]

[0153] in, The new traffic state containing adjacent intersection information learned by the graph attention network is To represent the set of neighbor intersections;

[0154] A multi-head attention mechanism is used, and each attention head outputs its own attention coefficient, so the features of multiple outputs are concatenated to obtain the fusion feature. :

[0155]

[0156] in, is the attention weight of the adjacent intersection b to the intersection a, is a learnable weight vector, Number the attention heads, , is the total number of attention heads;

[0157] The gated recurrent unit GRU is specifically constructed as follows:

[0158] The new features of each intersection output by the graph attention network As input, a hidden state containing historical traffic information is also input , get the reset gate signal and update gate signal :

[0159]

[0160]

[0161] in, The new features of each intersection output by the graph attention network; , , and is the weight matrix to be learned;

[0162] By resetting the goalkeeper's hidden state at the previous moment The relevant information in is discarded and combined with the new features of the current input Combine to obtain candidate hidden states ; Then, through the update gate, combined with the hidden state of the previous moment and candidate hidden states Get the current hidden state :

[0163]

[0164]

[0165] in, , is the weight matrix to be learned; tanh is the hyperbolic tangent activation function, and ⊙ represents element-wise multiplication;

[0166] The DQN action decision module is specifically constructed as follows:

[0167] The hidden state obtained by the spatiotemporal feature extraction module Input to the Q network to obtain the action value function in the current state , is the Q network parameter, and maximizes the reward R by selecting the action with the highest Q value at each time t;

[0168] The value decomposition hybrid network module includes a Q function summation network, which calculates the global Q function by inputting the Q functions of all intersections :

[0169]

[0170] in, For the The Q function of the intersection.

[0171] Step 4: Simulation environment and parameter setting and training of dynamic maximum pressure traffic signal control model;

[0172] The step 4 includes three steps:

[0173] Step 1: compile a road network file according to the actual road network, the road network file includes intersections, roads, and traffic lights; compile a traffic flow file according to the synthetic traffic flow data and the real traffic flow data, the traffic flow file includes the vehicle number, vehicle type, maximum acceleration and deceleration, maximum speed, path information, and start time of each vehicle; create a simulation environment according to the road network file and the traffic flow file;

[0174] Step 2: Initialize hyperparameters, including exploration rate, learning rate, discount factor, experience pool sample size, random sampling sample size, training rounds, spatiotemporal feature extraction module, and DQN action decision module hidden layer size.

[0175] Step 3: Read Figure 7 , according to the simulation environment and initialized hyperparameters, a dynamic maximum pressure traffic signal control model based on reinforcement learning is trained.

[0176] Interact the agent with the simulation environment to obtain the current state , a new fusion state is obtained through the spatiotemporal feature extraction module , pass the fusion state into the Q network to output the Q value of different actions, and then press - Greedy strategy selects actions. When making action decisions, the probability Randomly choose a random action with probability 1- Select the action that maximizes the Q value. The whole process is expressed as:

[0177]

[0178]

[0179]

[0180]

[0181] in, is the state of the intersection; is the new state of the intersection containing spatial feature information output by the graph attention network GAT; It is a new feature of the intersection containing temporal feature information output by the gated recurrent unit GRU; It is a Q network with intersections formed by fully connected layers FC; is the action; i is the intersection number; A is the number of discrete actions; is the parameter that needs to be optimized for the Q network; rand(0,1) means taking a random number in the range of 0 to 1; int(0,A) means randomly selecting a total of A integers from 0 to n-1;

[0182] Collect the trajectory experience during the interaction process, which is composed of state ,action ,award , next state Composition, through tuples Indicates that the trajectory experience is stored in the experience pool, which is a queue. When the experience pool capacity reaches the upper limit, the initially stored experience is discarded to store new experience. A batch of experience samples are randomly sampled from the experience pool for learning, and the target Q value of each experience sample is calculated. :

[0183]

[0184] in, is the discount factor; is the next action; are the learned model parameters;

[0185] Calculate the loss function and use backpropagation and gradient descent to update the network parameters of the graph attention network, gated recurrent unit, and DQN action module to minimize the loss function :

[0186]

[0187] in, is a batch experience sample; is the size of the batch experience sample; It is the batch sample status; for batch sample actions; is the batch sample number; is the target Q value; is the Q value of the global Q network.

[0188] The experiment takes 9 intersections as examples. Figure 8 For the SUMO simulation road network, the network topology is directly extracted from the OpenStreetMap dataset. The traffic demand is calibrated based on historical data collected from videos during the evening rush hour. Fig. 9 Input demand profile for training simulation environment. Each round is 60 minutes long and is divided into 6 different periods according to different demand levels or arrival rates. Vehicle arrival follows a Poisson process with a fixed arrival rate in each period. The proportion of vehicles turning left, going straight and turning right is 20%, 60% and 20% respectively. The relative demand level refers to the ratio of realized demand to the calibrated peak hour demand value. For signal controllers, each movement (controlled lane) will have a mandatory 3 seconds yellow time. A total of 6 different controls were tested in the simulation test: maximum pressure controller using queue length as input (Q-MP), maximum pressure controller using the ratio of queued vehicles to segment capacity as input (QL-MP), maximum pressure controller using travel time as input (TT-MP) and maximum pressure controller using density queue as input (C-MP), delay-based maximum pressure controller (D-MP) using crowdsourced data and dynamic maximum pressure controller (SGMA-DMP). The final hyperparameter settings for reinforcement learning training are shown in Table 1:

[0189] Table 1 Final hyperparameter settings for reinforcement learning training

[0190]

[0191] The average queue length, delay time and total throughput are selected as evaluation indicators. The results are shown in Table 2:

[0192] Table 2 Comparison of different controllers

[0193]

[0194] Fig.10 , Fig.11 , Fig.12Table 2 shows that under low traffic demand, the performance of each controller is similar, and the controllers with different pressure parameters have similar control effects on network queue length and average vehicle delay; however, as traffic demand increases, the dynamic maximum pressure controller has better performance in controlling queue length and average vehicle delay than the single pressure parameter and associated maximum pressure control. Compared with the other five control methods, the dynamic maximum pressure controller reduces queue length by more than 14% and delay by more than 10% on average. At the same time, the dynamic maximum pressure controller has a higher vehicle throughput.

[0195] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A dynamic maximum pressure traffic signal control method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Collect traffic flow data; Step 2: Establish a dynamic maximum pressure traffic signal control method that dynamically associates dynamic traffic pressure parameters with upstream and downstream pressures; Step 3: Establish a dynamic maximum pressure traffic signal control model based on reinforcement learning; Step 4: Set the simulation environment and parameters to complete the training of the dynamic maximum pressure traffic signal control model based on reinforcement learning; The step 2 comprises: Step 2.1: Based on the variable cell transport model, four types of dynamic traffic pressure parameters are divided according to free flow density, congestion flow density and road section length. , expressed as: ； ； ； ； in, is the pressure of congestion flow density; is the pressure of the combination of congestion flow density and free flow density; is the pressure of congestion flow density combined with the length of the road section; It is the pressure of the combination of congestion flow density and free flow density under the combined section length; is the congestion flow density pressure; is the free stream density pressure; , , Indicates the type of pressure; is the length of the road section; For time; , Number the road sections; Step 2.2: According to the dynamic correlation of the pressure of the upstream and downstream adjacent intersections, use the upstream and downstream correlation parameters To characterize whether adjacent intersections are associated, The value of is 0 and 1, where 0 means no association and 1 means association; Step 2.3: Using a fixed phase sequence, a dynamic maximum pressure traffic signal control method that dynamically associates dynamic traffic pressure parameters with upstream and downstream pressures is established. At the end of each signal cycle, the green light time of each phase is calculated based on the pressure proportion of each phase; The step 2.3 is specifically described as follows: First calculate the pressure weight , at every time When , the controller calculates the pressure weight according to the selected traffic pressure parameters and upstream and downstream associated parameters: ； in, and is the traffic pressure parameter; is the type of pressure; , , Number the road sections; From the road section To the road section The traffic ratio, The starting section is The set of all downstream segments of ; Calculate traffic pressure , with the maximum pressure weight Saturation flow rate The product of is used to express the traffic pressure, that is: ； Calculate phase pressure , is the phase number, each phase contains a number of traffic flows, so the phase pressure is equal to the sum of the pressures of all traffic flows contained in the phase: ； in, Indicates phase Internal arrive Turning into traffic; Calculate the green light time of each phase according to the pressure ratio of each phase for: ； in, , Phase and Phase The sum of all traffic pressures contained in it; Number the node; , is the phase number; is the signal period; is the phase set; To ensure pedestrians can cross the street, the minimum green light time is set to , and set the maximum green light time to , then the phase green light time The final expression is: ； The step 3 comprises: Step 3.1: Reinforcement learning agent, including the design of states, actions, and rewards; Step 3.2: Establish a dynamic maximum pressure traffic signal control model based on reinforcement learning, including a spatiotemporal feature extraction module, a DQN action decision module, and a value decomposition hybrid network module; The step 3.1 comprises: The action is specifically designed to use the four types of discrete combinations of dynamic traffic pressure parameters and upstream and downstream associated parameters as reinforcement learning actions, and after the traffic pressure parameters and associated parameters are determined through DQN actions, they are input into the dynamic maximum pressure traffic signal control method dynamically associated with the dynamic traffic pressure parameters and upstream and downstream pressures, and the green light time of each phase is output; The state is specifically designed as follows: the state includes the number of vehicles queuing at each entrance of the current intersection , vehicle density on the import road , the current intersection exit road vehicle density , previous action and intersection number , the state is represented by a state vector state={ , , , , }; The design of the reward is specifically that the reward is a combination of the number of vehicles queued at the intersection in adjacent time steps and the intersection throughput: ； in, For The reward obtained after performing an action at the intersection at the current state, for The duration of the next cycle at time, is the total number of vehicles queuing at the intersection entrance, is the vehicle throughput at the intersection, is the weight coefficient; The step 3.2 comprises: The spatiotemporal feature extraction module is composed of a graph attention network GAT and a gated recurrent unit GRU in series. The state features of each intersection are input into the spatiotemporal feature extraction module, and a new intersection state feature containing adjacent intersections and historical information is output, specifically: The graph attention network GAT is specifically constructed as follows: Input feature matrix X, shape is N×F, N is the number of intersections, F is the intersection state dimension; for each intersection a and adjacent intersection b, the corresponding state vector is and , calculate the attention coefficient between intersection a and adjacent intersection b for: ； in, is the attention mechanism, which is a learnable parameter vector; T represents matrix transpose; LeakyReLU is the activation function; is a learnable weight vector; || represents a matrix concatenation operation; Use the Softmax function to normalize the attention coefficient and get the attention weight of the adjacent intersection b to the intersection a. for: ； in, is the attention coefficient between intersection a and the adjacent intersection b, is the attention coefficient between intersection a and the adjacent intersection c; exp is the natural exponential function; is the set of adjacent intersections of intersection a; c is the intersection number; Take the weighted sum of the features of all adjacent intersections of intersection a to obtain a new feature representation of intersection a, and apply the sigmoid function to the new feature vector Activate and finally output the new feature matrix Z: ； in, The new traffic state containing adjacent intersection information learned by the graph attention network is To represent the set of neighbor intersections; A multi-head attention mechanism is used, and each attention head outputs its own attention coefficient, so the features of multiple outputs are concatenated to obtain the fusion feature. : ； in, is the attention weight of the adjacent intersection b to the intersection a, is a learnable weight vector, Number the attention heads, , is the total number of attention heads; The gated recurrent unit GRU is specifically constructed as follows: The new features of each intersection output by the graph attention network As input, a hidden state containing historical traffic information is also input , get the reset gate signal and update gate signal : ； ； in, The new features of each intersection output by the graph attention network; , , and is the weight matrix to be learned; By resetting the goalkeeper's hidden state at the previous moment The relevant information in is discarded and combined with the new features of the current input Combine to obtain candidate hidden states ; Then, through the update gate, combined with the hidden state of the previous moment and candidate hidden states Get the current hidden state : ； ； in, , is the weight matrix to be learned; tanh is the hyperbolic tangent activation function, and ⊙ represents element-wise multiplication; The DQN action decision module is specifically constructed as follows: The hidden state obtained by the spatiotemporal feature extraction module Input to the Q network to obtain the action value function in the current state , is the Q network parameter, and maximizes the reward R by selecting the action with the highest Q value at each time t; The value decomposition hybrid network module includes a Q function summation network, which calculates the global Q function by inputting the Q functions of all intersections : ； in, For the The Q function of the intersection.

2. The method for dynamic maximum pressure traffic signal control based on reinforcement learning according to claim 1 is characterized in that: The step 4 comprises: Step 4.1: compile a road network file according to the actual road network, the road network file includes intersections, roads, and traffic lights; compile a traffic flow file according to the synthetic traffic flow data and the real traffic flow data, the traffic flow file includes the vehicle number, vehicle type, maximum acceleration and deceleration, maximum speed, path information, and start time of each vehicle; create a simulation environment according to the road network file and the traffic flow file; Step 4.2: Initialize hyperparameters, including exploration rate, learning rate, discount factor, experience pool sample size, random sampling sample size, training rounds, spatiotemporal feature extraction module, and DQN action decision module hidden layer size; Step 4.3: Train the dynamic maximum pressure traffic signal control model based on reinforcement learning according to the simulation environment and the initialized hyperparameters.

3. The method for dynamic maximum pressure traffic signal control based on reinforcement learning according to claim 2 is characterized in that: The step 4.3 includes: Interact the agent with the simulation environment to obtain the current state , a new fusion state is obtained through the spatiotemporal feature extraction module , pass the fusion state into the Q network to output the Q value of different actions, and then press - Greedy strategy selects actions. When making action decisions, the probability Randomly choose a random action with probability 1- Select the action that maximizes the Q value. The whole process is expressed as: ； ； ； ； in, is the state of the intersection; is the new state of the intersection containing spatial feature information output by the graph attention network GAT; It is a new feature of the intersection containing temporal feature information output by the gated recurrent unit GRU; It is a Q network with intersections formed by fully connected layers FC; is the action; i is the intersection number; A is the number of discrete actions; is the parameter that needs to be optimized for the Q network; rand(0,1) means taking a random number in the range of 0 to 1; int(0,A) means randomly selecting a total of A integers from 0 to n-1; Collect the trajectory experience during the interaction process, which is composed of state ,action ,award , next state Composition, through tuples Indicates that the trajectory experience is stored in the experience pool, which is a queue. When the experience pool capacity reaches the upper limit, the initially stored experience is discarded to store new experience. A batch of experience samples are randomly sampled from the experience pool for learning, and the target Q value of each experience sample is calculated. : ； in, is the discount factor; is the next action; are the learned model parameters; Calculate the loss function and use backpropagation and gradient descent to update the network parameters of the graph attention network, gated recurrent unit, and DQN action module to minimize the loss function : ； in, is a batch experience sample; is the size of the batch experience sample; It is the batch sample status; for batch sample actions; is the batch sample number; is the target Q value; is the Q value of the global Q network.

Images