Regional traffic signal control method based on graph attention and value decomposition reinforcement learning

Abstract

The present invention provides a regional traffic signal control method based on graph attention and value decomposition reinforcement learning. By collecting road traffic information, a road network simulation model and a graph model are established, and the spatial and temporal characteristics of the road network traffic information are fully utilized. A multi-head graph attention network and a gated recurrent unit are used for feature extraction. The fused spatiotemporal features are passed as input to the D3QN network to calculate the current optimal phase. During the training process, a value decomposition network under the condition of multi-agent shared parameters is used to mix the state action values ​​of all agents in the road network to achieve global training. The present invention can enhance the agent's perception of road network characteristics, effectively utilize the road network's temporal and spatial information, improve the overall control effect of regional traffic, and improve the road network traffic operation efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent traffic control, and in particular to a regional traffic signal control method based on graph attention and value decomposition reinforcement learning. Background Art

[0002] The current traffic congestion problem in cities has caused huge economic and time waste. Traffic congestion is caused by many factors, such as unreasonable signal control schemes, lane structure defects, etc. Effective traffic signal control is the most direct and lowest-cost way to improve lane traffic efficiency. The most widely used traffic signal control method in cities is manual time-sharing. This method relies on traditional signal control principles and a lot of manual design experience. It can alleviate traffic congestion to a certain extent. However, there is still a problem that fixed timing cannot adapt to changing traffic flow, and the timing efficiency is relatively low. Therefore, the current traffic signal control method still has a lot of room for improvement.

[0003] With the rapid development of artificial intelligence technology, traffic control methods based on reinforcement learning are the current research focus in the field of adaptive traffic signal control. Such methods can output the optimal signal control strategy based on the current traffic status, which further improves traffic efficiency and road network throughput compared to fixed timing. In addition, they can also be applied to various complex traffic scenarios such as peak hours and intersection congestion.

[0004] Traffic control methods based on reinforcement learning face the important problem of how to design traffic states. Existing studies usually combine inherent states such as queue length, speed, and waiting time, which increases the difficulty of state design on the one hand, and ignores a large amount of road network topology information on the other hand. In addition, multi-agent reinforcement learning is often used when facing regional signal control problems, but multi-agent collaborative decision-making has value function estimation problems and action selection difficulties. Therefore, how to extract traffic state features and collaboratively train multiple agents has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to propose a regional traffic signal control method based on graph attention and value decomposition reinforcement learning. On the one hand, the spatiotemporal characteristics of the current road network state are obtained through a feature fusion module including a multi-head graph attention network and a gated recurrent unit, which effectively reduces the difficulty of state design. On the other hand, value decomposition network training is used to alleviate the value function estimation problem and action selection problem of multiple agents.

[0006] The technical solution to achieve the purpose of the present invention is: a regional traffic signal control method based on graph attention and value decomposition reinforcement learning, comprising the following steps:

[0007] Step 1: Establish a simulation environment based on the road network environment and traffic flow data, and calculate the adjacency matrix as a graph structure based on the road network;

[0008] Step 2: Collect traffic flow information, including phase, queue length, vehicle density, phase switching, and design reinforcement learning states, actions, and rewards;

[0009] Step 3, construct a regional traffic signal control model, including a feature fusion module, a D3QN decision module and a value decomposition network VDN. The feature fusion module consists of a multi-head graph attention network and a gated recurrent unit. The state record of each intersection is obtained through the multi-head graph attention network to obtain the output features of each intersection, and then the final features of the fused historical information are obtained through the gated recurrent unit; the D3QN decision module consists of two dueling networks, which take the final features of each intersection spliced ​​with the corresponding intersection identification number as input to obtain the state action value; VDN contains only one summation network, which approximates the global state action value by summing the state action values ​​of all input intersections;

[0010] Step 4: Collect the interaction experience between the agent and the traffic simulation environment and store it in the experience pool;

[0011] Step 5: Sample the experience pool and use the samples obtained to train the regional traffic signal control model to complete the regional traffic signal control.

[0012] Further, in step 1, a simulation environment is established according to the road network environment and traffic flow data, and an adjacency matrix is ​​calculated according to the road network as a graph structure. The specific method is:

[0013] Step 11, compile a road network file and a traffic flow file according to the actual control area requirements. The road network file includes the connection information between all intersections and roads, and the traffic flow file includes the maximum acceleration, maximum speed, routing information and departure time of each vehicle;

[0014] Step 12, establishing a traffic simulation environment according to the road network file and the traffic flow file;

[0015] Step 13, calculate the adjacency matrix based on the road network connection information. The row and column numbers of the two-dimensional matrix represent the corresponding intersection numbers. The elements in the matrix indicate whether the intersections with the corresponding row and column numbers are connected. If there is a connection, the element is set to 1, otherwise it is set to 0.

[0016] Furthermore, in step 2, traffic flow information is collected, including phase, queue length, vehicle density, phase switching, and the state, action, and reward of reinforcement learning are designed. The specific method is as follows:

[0017] (1) Status

[0018] The state is represented by a one-dimensional matrix s = {P, min, d, q}, which contains four sub-state matrices: phase, queue length, vehicle density, and phase switching permission. The phase P is represented by one-hot encoding. The i-th phase P i Denoted as {0,…,p i ,…,0},p i is 1, and the rest are 0; the queue length q of the lane is represented by {q1,…,q i ,…,q n}, where the queue length of the i-th lane is q i =n si (l v +2.5) / l l , n si is the number of vehicles stopped on the lane, l v is the average length of vehicles on the lane, 2.5 is the minimum distance between vehicles, l l is the length of the lane; the vehicle density d of the lane is expressed as {d1,…,d i ,…,d n}, where the vehicle density of the i-th lane is d i =n i (l v +2.5) / l l , n i is the number of all vehicles on the lane; the phase switching permission min is recorded as {1} or {0};

[0019] (2) Action

[0020] The action is set to the next phase selection of the signal light, the action space is all phases of the signal light, and the signal light is set to random phase switching, that is, switching from the current phase to any phase;

[0021] (3) Rewards

[0022] The reward r is the negative of the sum of the queue lengths of all lanes at the intersection.

[0023] Furthermore, in step 3, a regional traffic signal control model is constructed, including a feature fusion module, a D3QN decision module and a value decomposition network VDN. The specific method is as follows:

[0024] Step 31, construct a feature fusion module, which is a series structure of a multi-head graph attention network and a gated recurrent unit, wherein the multi-head graph attention network is composed of multiple graph attention networks with consistent structure and input in parallel;

[0025] (1) Multi-head Graph Attention Network

[0026] For a single graph attention network, the state of each intersection is recorded as the original feature s iThrough a 33×64 fully connected layer, the embedded features h are obtained respectively i , input all embedded features into a 64×64 fully connected layer to obtain enhanced features, concatenate the enhanced features of any two intersections to obtain joint features, and output all joint features through a 128=1 fully connected layer to obtain the similarity coefficient e between the intersections ij , including the similarity coefficient e of the intersection itself ii ; Arrange the similarity coefficients into a similarity coefficient matrix. Similar to the adjacency matrix, the elements in the similarity coefficient matrix represent the similarity coefficients between the intersections with the corresponding row and column numbers; After the similarity coefficient matrix is ​​masked by the adjacency matrix, it is input into a softmax function to obtain the attention coefficient α between the intersections ij , use the attention coefficient to weight the embedded features and activate them using the elu function to get the fused feature h′ i ; The above are the fusion features obtained by a single graph attention network. The fusion features output by multiple graph attention networks are spliced ​​to obtain the output features of each intersection;

[0027] (2) Gated Recurrent Unit

[0028] The gated recurrent unit first inputs the output features of all intersections in the multi-head graph attention network into a 64×64 fully connected layer to obtain spatial features of uniform size. Before using the gated recurrent unit at the initial moment, it is necessary to initialize a historical time series feature with the same size as the spatial feature. All elements of the initialized historical time series feature are 0. The spatial features and historical time series features of all intersections are input into the GRU layer, and the final feature that integrates the historical information is output. The final feature is used as the historical time series feature at the next moment.

[0029] Step 32, constructing a D3QN decision module under the condition of multi-agent shared parameters. The constructed D3QN decision module includes two dueling networks with the same structure and different parameters. Only the first dueling network is used when outputting decisions, and both dueling networks are used simultaneously when updating parameters.

[0030] In order to distinguish different intersections, the duel network needs to add unique hot encoding of the intersection identification number to the features. The final features of each intersection output by the feature fusion module are spliced ​​with the corresponding intersection identification number, and input into an 80×128 fully connected layer to obtain the transition feature. The duel network has two network branches, the value function and the advantage function. The transition features are input into the value function network and the advantage function network respectively. The value function network is composed of two fully connected layers in series, with sizes of 128×128 and 128×1 respectively. The advantage function network is composed of two fully connected layers in series, with sizes of 128×128 and 128×8 respectively. The average difference of the output of the advantage function network is calculated and summed with the output of the value function network to obtain the state action value. According to the ε-greedy strategy, the action corresponding to the maximum value of the state action value is taken as the output strategy;

[0031] Step 33, constructing a value decomposition network VDN used in training. The VDN only contains a summation network, which approximates the global state-action value by summing the state-action values ​​of all input intersections;

[0032] Step 34, set the loss function to Huber loss function, the optimizer to Adam optimizer, and initialize the hyperparameters, including learning rate, discount factor, number of samples, experience pool size, time series length, decision frequency, training frequency, and number of training rounds.

[0033] Furthermore, in step 4, the interaction experience between the agent and the traffic simulation environment is collected and stored in the experience pool, where:

[0034] The experience pool is a python queue. The number of elements is at most the size of the experience pool. Each element is a python list. Each list represents a complete simulation process. Each element of the list represents a state transition {s,a,s next ,r}, s is the current state of all intersections, a is the action selected by all intersections, s next is the next state of all intersections after the action is executed, r is the reward calculated based on the next state, s and s next At intervals of a certain number of simulation steps, depending on the decision frequency, the agent interacts with the traffic simulation environment to generate experience. When the amount of experience reaches the upper limit of the experience pool size, the oldest experience is discarded.

[0035] Furthermore, in step 5, sampling is performed on the experience pool, and the samples obtained by sampling are used to train the regional traffic signal control model to complete the regional traffic signal control. The specific method is as follows:

[0036] Step 51, randomly extract experience samples from the experience pool according to the sampling quantity. If the number of samples in the experience pool is less than 1, then exit the training and continue sampling; otherwise, randomly intercept a continuous state transition from the extracted experience samples according to the time series length as input;

[0037] Step 52: The original states s and s before and after the state transfer next The final features h and h before and after state transfer are obtained through the feature fusion module respectively. next ;

[0038] Step 53, using the two constructed duel networks, input h and a into the first duel network to obtain the state action value Q(s, a; θ) in the current state, and then input h next Input the first dueling network to get the best action for the next state h next and a′ are input into the second duel network to obtain θ and are the network parameters of the two dueling networks respectively;

[0039] Step 54, using the constructed VDN, input the current and next state action values ​​of all agents into the value decomposition network to obtain the global state action value estimate Q(S,A;θ)=∑Q(s,a;θ), Among them, S and S next Indicates the current and next global states;

[0040] Step 55, calculate the target value based on the reward Where γ is the discount factor;

[0041] Step 56, calculate the loss Loss = Q(S, A; θ) - T of the target value and the current state action value, and back-propagate based on the loss to update the network parameters of the multi-head attention network, gated recurrent unit and D3QN module.

[0042] A regional traffic signal control system based on graph attention and value decomposition reinforcement learning implements the regional traffic signal control method based on graph attention and value decomposition reinforcement learning to achieve regional traffic signal control based on graph attention and value decomposition reinforcement learning.

[0043] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, which implements the regional traffic signal control method based on graph attention and value decomposition reinforcement learning to achieve regional traffic signal control based on graph attention and value decomposition reinforcement learning.

[0044] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method for regional traffic signal control based on graph attention and value decomposition reinforcement learning is implemented to realize regional traffic signal control based on graph attention and value decomposition reinforcement learning.

[0045] Compared with the prior art, the present invention has the following significant advantages:

[0046] (1) The feature fusion module adopts a serial architecture of a multi-head graph attention network and a gated recurrent unit to effectively extract the spatiotemporal characteristics of the current traffic network state and reduce the difficulty of state design.

[0047] (2) The D3QN decision module adopts a dual dueling deep Q network, which effectively alleviates the problem of overestimation of the value of a single network and effectively improves the accuracy of estimating the pros and cons of different actions.

[0048] (3) Use a value decomposition network under shared parameter conditions to assist training, effectively alleviating the value function estimation and action selection problems of multiple agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the overall method flow framework of the present invention.

[0050] Figure 2 It is a single graph attention network structure implemented by the present invention.

[0051] Figure 3 It is a gated cycle unit structure implemented by the present invention.

[0052] Figure 4 This is the structure of the D3QN decision module implemented by the present invention.

[0053] Figure 5 This is a SUMO road network scenario of an embodiment of the present invention.

[0054] Figure 6 is the adjacency matrix of the embodiment of the present invention.

[0055] Figure 7 It is the comparison result of the average travel time of the embodiment of the present invention.

[0056] Figure 8 It is the comparison result of the average travel time of the embodiment of the present invention.

[0057] Fig. 9 It is the comparison result of the normalized average travel time of the embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. After reading the present invention, those skilled in the art will modify the various equivalent forms of the present invention within the scope defined by the appended claims of the present application.

[0059] The present invention provides a regional traffic signal control method based on graph attention and value decomposition reinforcement learning. The overall method flow is as follows: Figure 1 shown.

[0060] Step 1: Establish a simulation environment based on the road network environment and traffic flow data, and calculate the adjacency matrix as a graph structure based on the road network, including the following specific steps:

[0061] Step 11, compile the road network file and traffic flow file according to the actual control area requirements. The road network file includes the connection information between all intersections and roads, and the traffic flow file includes the maximum acceleration, maximum speed, routing information and departure time of each vehicle.

[0062] Step 12: Establish a traffic simulation environment based on the road network file and the traffic flow file.

[0063] Step 13, calculate the adjacency matrix based on the road network connection information. The row and column numbers of the two-dimensional matrix represent the corresponding intersection numbers. The elements in the matrix indicate whether the intersections with the corresponding row and column numbers are connected. If there is a connection, the element is set to 1, otherwise it is set to 0.

[0064] Step 2: Design the states, actions, and rewards of reinforcement learning based on the traffic flow information that can be collected in the simulation environment, including phase, queue length, vehicle density, and phase switching permission.

[0065] (1) Status

[0066] The state is represented as a one-dimensional matrix s = {P, min, d, q}, which contains four sub-state matrices: phase, queue length, vehicle density, and phase switching permission. Phase P is represented using one-hot encoding. The i-th phase P i Denoted as {0,…,p i ,…,0},p i is 1, and the rest are 0; the queue length q of the lane is represented by {q1,…,q i ,…,q n}, where the queue length of the i-th lane is q i =n si (l v +2.5) / l l , n si is the number of vehicles stopped on the lane, l v is the average length of vehicles on the lane, 2.5 is the minimum distance between vehicles, l l is the length of the lane; the vehicle density d of the lane is expressed as {d1,…,d i ,…,d n}, where the vehicle density of the i-th lane is d i =n i (l v+2.5) / l l , n i is the number of all vehicles on the lane; the phase switching permission min is recorded as {1} or {0}.

[0067] (2) Action

[0068] The action is set to the next phase selection of the signal light, the action space is all phases of the signal light, and the signal light is set to random phase switching, that is, it can switch from the current phase to any phase.

[0069] (3) Rewards

[0070] The reward r is the negative of the sum of the queue lengths of all lanes at the intersection. q i Same meaning as in Status.

[0071] Step 3: Taking the states of all intersections in step 2 as input and the actions of each agent as output, a multi-agent shared parameter network model is established, which includes a feature fusion module for extracting spatiotemporal characteristics of traffic flow and a D3QN decision module for outputting control actions, constructs a value decomposition network for training, and initializes the corresponding module parameters. Specifically, the following steps are included:

[0072] Step 31, construct a feature fusion module, whose basic structure is the series connection of a multi-head graph attention network and a gated recurrent unit. The multi-head graph attention network is composed of multiple graph attention networks with consistent structure and input in parallel. The gated recurrent unit is GRU, which is a type of recurrent neural network.

[0073] (1) Multi-head Graph Attention Network

[0074] The state of each intersection in step 2 is recorded as the original feature s i Through a 33×64 fully connected layer, the embedded features h are obtained respectively i , the following is an explanation of the single graph attention network structure, such as Figure 2 As shown in Figure 1, all embedded features are input into a 64×64 fully connected layer to obtain enhanced features respectively, and the enhanced features of any two intersections are concatenated to obtain joint features. All joint features are output through a 128×1 fully connected layer to obtain the similarity coefficient e between the intersections. ij , including the similarity coefficient e of the intersection itself ii The similarity coefficients are arranged into a similarity coefficient matrix. Similar to the adjacency matrix, the elements in the similarity coefficient matrix represent the similarity coefficients between the intersections with the corresponding row and column numbers. The similarity coefficient matrix is ​​input into a softmax function after the mask operation of the adjacency matrix to obtain the attention coefficient α between the intersections. ij, use the attention coefficient to weight the embedded features and activate them using the elu function to get the fused feature h′ i The above are the fusion features obtained by a single graph attention network. The fusion features output by multiple graph attention networks are concatenated to obtain the output features of each intersection.

[0075] (2) Gated Recurrent Unit

[0076] like Figure 3 As shown in the figure, the output features of all intersections in the multi-head graph attention network are first input into a 64×64 fully connected layer to obtain spatial features of uniform size. Before using the gated recurrent unit at the initial moment, it is necessary to initialize a historical time series feature with a size consistent with the spatial feature. All elements of the initialized historical time series feature are 0. The spatial features and historical time series features of all intersections are input into the GRU layer, and the final feature that integrates the historical information is output and used as the historical time series feature at the next moment.

[0077] Step 32, taking the final features of the feature fusion module as input, constructing a D3QN decision module under the condition of multi-agent shared parameters. The constructed D3QN decision module contains two duel networks with the same structure and different parameters. Only the first duel network is used when outputting the decision, and the two duel networks are used simultaneously when updating the parameters.

[0078] D3QN Decision Module

[0079] Since the agents corresponding to each intersection share the same decision module, in order to distinguish different intersections, it is necessary to add a unique hot encoding of the intersection identification number to the features. The final features of each intersection output by the feature fusion module are concatenated with the corresponding intersection identification number and input into an 80×128 fully connected layer to obtain the transition features, such as Figure 4 As shown in the figure, the duel network has two network branches, the value function and the advantage function. The transition features are input into the value function network and the advantage function network respectively. The value function network consists of two fully connected layers in series, with sizes of 128×128 and 128×1 respectively. The advantage function network consists of two fully connected layers in series, with sizes of 128×128 and 128×8 respectively. The average difference of the output of the advantage function network is calculated and summed with the output of the value function network to obtain the state action value. According to the ε-greedy strategy, the action corresponding to the maximum value of the state action value is taken as the output strategy. The second duel network structure is consistent with the above and is only used when updating network parameters.

[0080] Step 33, using the state-action value calculated by the D3QN decision module as input, construct a value decomposition network VDN used in training. The VDN only contains a summation network, which approximates the global state-action value by summing the state-action values ​​of all input intersections.

[0081] Step 34, set the loss function to Huber loss function, the optimizer to Adam optimizer, and initialize the hyperparameters, including learning rate, discount factor, number of samples, experience pool size, time series length, decision frequency, training frequency, and number of training rounds.

[0082] Step 4: Set the experience pool according to the experience pool size in step 3, collect the interaction experience between the agent and the traffic simulation environment according to the decision frequency, and store it in the experience pool.

[0083] (1) Experience Pool

[0084] The experience pool is a python queue. The number of elements is at most the size of the experience pool. Each element is a python list. Each list represents a complete simulation process. Each element of the list represents a state transition {s,a,s next ,r}, s is the current state of all intersections, a is the action selected by all intersections, s next is the next state of all intersections after the action is executed, r is the reward calculated based on the next state, s and s next At intervals of a certain number of simulation steps, depending on the decision frequency, the agent interacts with the traffic simulation environment to generate experience. When the amount of experience reaches the upper limit of the experience pool size, the oldest experience is discarded.

[0085] Step 5: Sample the experience pool according to the training frequency in step 3, and use the sampled samples to train the model, which specifically includes the following steps:

[0086] Step 51, randomly extract experience samples from the experience pool according to the sampling quantity in step 3. If the number of samples in the experience pool is less than 1, then exit the training and continue to step 4; otherwise, randomly intercept a continuous state transition from the extracted experience samples according to the time series length in step 3 as input;

[0087] Step 52: The original states s and s before and after the state transfer next The final features h and h before and after state transfer are obtained through the feature fusion module respectively. next ;

[0088] Step 53, using the two duel networks constructed in step 32, input h and a into the first duel network to obtain the state action value Q(s, a; θ) in the current state, and then input h next Input the first dueling network to get the best action for the next state h next and a′ are input into the second duel network to obtain θ and are the network parameters of the two dueling networks respectively;

[0089] Step 54, using the VDN constructed in step 33, input the current and next state action values ​​of all agents into the value decomposition network to obtain an estimate of the global state action value Q(S,A;θ) = ∑Q(s,a;θ), Among them, S and S next Indicates the current and next global states;

[0090] Step 55, calculate the target value based on the reward Where γ is the discount factor;

[0091] Step 56, calculate the loss Loss = Q(S, A; θ) - T of the target value and the current state action value, and back-propagate based on the loss to update the network parameters of the multi-head attention network, gated recurrent unit and D3QN module.

[0092] Step 6: Repeat steps 4-5 until the reward converges, and use the trained model to implement regional traffic signal control.

[0093] Example

[0094] In order to verify the effectiveness of the solution of the present invention, the following simulation experiment is carried out.

[0095] The simulation environment used in this embodiment is provided by SUMO, which is an open source urban traffic simulation software used to simulate and analyze issues such as traffic flow, traffic signal control and traffic management. The Traci interface can be used in the Python environment to interact with SUMO, such as obtaining vehicle data, obtaining lane data, and controlling traffic lights. The simulation experiment scene is a 4×4 road network scene rebuilt according to the real road network. The vehicle data used comes from a public data set, which contains a total of 3600 seconds of traffic data. In order to show the control effect of the present invention, three comparison standards are used for evaluation: vehicle arrival rate, average travel time, and normalized average travel time. The vehicle arrival rate represents the ratio of the number of vehicles that completely leave the road network to the number of all initialized vehicles within the simulated 3600 seconds, which can measure the traffic efficiency of the road network. The average travel time represents the average time difference between all vehicles entering and leaving the road network. However, the average travel time only counts the vehicles that completely leave the road network, ignoring the vehicles that still exist in the road network, and is only illustrative when the arrival rate is high. The normalized average travel time represents the ratio of the average travel time to the vehicle arrival rate, thereby balancing the first two indicators and more comprehensively evaluating the traffic conditions of the road network. The goal of this embodiment is to maximize the reinforcement learning reward by controlling the phase of the traffic lights, reduce the normalized average travel time, and thus improve the traffic efficiency of the road network.

[0096] Based on the above objectives, the specific steps of this embodiment are as follows:

[0097] Step S1: Model the road network environment and traffic flow data, and after initializing the simulation environment, establish a graph structure by calculating the adjacency matrix of the road network. The modeling simulation environment is completed in SUMO, and the corresponding road network file and traffic flow file are required, where the road network file includes the connection information between all intersections and roads, and the traffic flow file contains the maximum acceleration, maximum speed, routing information, departure time, etc. of each vehicle in the road network. The graph structure is represented by an adjacency matrix, which can describe the topological information of the road network nodes and edges, where the road network nodes refer to the intersections controlled by a traffic light, and the edges refer to the undirected connections between intersections formed by roads. The number of rows and columns of the two-dimensional adjacency matrix is ​​equal, which is the number of nodes in the road network. Each element in the matrix indicates whether there is an edge connection between the corresponding nodes. If there is an edge connection between the i-th and j-th nodes, the element of the i-th row and j-th column of the matrix is ​​set to 1. If there is no edge connection, the corresponding position is set to 0. In addition, the connection of the node to itself is set to 1.

[0098] Specifically for the 4×4 network scenario in the embodiment, such as Figure 5 As shown in the figure, there are 16 intersections in total. For experimental considerations, each intersection is isomorphic, and the physical structure and control phase set of the intersection are the same. Each intersection has 24 lanes in four directions, divided into 12 import lanes and 12 exit lanes, and the number of lanes in all directions is the same. Every 3 lanes constitute a set of exits or entrances, which are responsible for left turns, straight driving and right turns of vehicles respectively. Each intersection is controlled by a set of phase sets specified by a set of traffic lights. There are 8 green light phases and 8 yellow light phases in the phase set. The green light phase is responsible for the straight driving and turning of vehicles, and the yellow light phase is the transition between green light phases. In order to meet the traffic needs of pedestrians and reduce the stop-start loss of vehicles during phase switching, the shortest phase duration of each green light phase is 10 seconds, that is, each phase lasts at least 10 seconds before switching to the next appropriate phase. In addition, the yellow light phase is fixed to 3 seconds.

[0099] The specific vehicle parameters of the public data set used in the embodiment are as follows: all vehicles have the same basic type, with a body length of 5.0m, a width of 2.0m, a minimum vehicle spacing of 2.5m, and a maximum vehicle acceleration of 2.0m / s 2 , the maximum deceleration is -4.5m / s 2 , with a maximum speed of 11.11 m / s. Each vehicle data contains three key information: departure time, vehicle name, and route track. The route track records all lanes that the vehicle passes through from the starting point to the end point.

[0100] Specifically for the 4×4 network scenario in the embodiment, such as Figure 6 As shown, the adjacency matrix is ​​a 16×16 two-dimensional matrix.

[0101] Step S2: Design the three elements of reinforcement learning for the agent: state, action, and reward. The state matrix consists of four sub-states, as follows:

[0102] The hot coding represents the phase P of the traffic light at the previous intersection. In this embodiment, the current traffic light at the intersection has 8 phases, so each phase is represented by a 1×8 matrix. The specific i-th phase P i It is represented as {0,…,p i ,…,0}, where the element p i is 1, and the i-th element p i is 1, and the rest are 0; the queue lengths of all lanes at the current intersection are arranged in sequence to form a queue length matrix q. In this embodiment, the intersection controls 12 import lanes, so the matrix q = {q1,…,q i ,…,q 12}, the specific queue length q of the i-th lane i =n si (l v +2.5) / l l , where n si is the number of vehicles stopped on the lane, l v is the average length of vehicles on the lane, 2.5 is the minimum distance between vehicles, l l is the length of the lane; the vehicle density of all lanes at the current intersection is arranged in sequence to form a vehicle density matrix d. In this embodiment, the intersection controls 12 import lanes, so the matrix d = {d1,…,d i ,…,d 12}, the specific vehicle density d of the i-th lane i =n i (l v +2.5) / l l , where n i is the number of vehicles on the lane, l v , 2.5, l l It is consistent with the variables in the queue length; if the duration of the current intersection phase is greater than the minimum green light time, it can switch to the next phase, and take the phase switching permission min={1}, otherwise take min={0}; the above four sub-states are spliced ​​to form a complete state matrix s={P,min,d,q}; the action of the agent is set to the next phase selection of the intersection signal light, so its action space is all phases of the signal light. In this embodiment, the action space is 8, and the signal light is set to random phase switching, that is, it can switch from a certain phase to any phase. If the phase duration is less than the minimum green light time, the phase will not be changed; the reward r of the agent is the negative of the sum of the queue lengths of all lanes at the current intersection, where q i Has the same meaning as the variables in state.

[0103] Step S3: Set hyperparameters and establish a graph attention decision network, which includes a feature fusion module and a D3QN decision module

[0104] The hyperparameter initialization in this embodiment includes learning rate, discount factor, number of samples, experience pool size, time series length, decision frequency, training frequency, number of training rounds, feature fusion module and hidden layer size of decision module; wherein the training frequency and decision frequency are parameters different from most training frameworks, the decision frequency refers to the number of simulation steps between two actions of the network output, and the training frequency refers to the number of decisions between two trainings of the entire network. The specific parameters are set as follows: the learning rate is 0.001, the discount factor is 0.99, the number of samples is 32, the experience pool size is 50, the time series length is 8, the decision frequency is 5, the training frequency is 1, and the number of training rounds is 200.

[0105] The feature fusion module in this embodiment includes two feature extraction networks, namely a multi-head graph attention network and a gated recurrent unit. The multi-head graph attention network can fuse spatial features according to the graph structure, and the gated recurrent unit can fuse historical temporal features. The two networks are connected in series to obtain the final spatiotemporal features. The graph attention network includes a total of 4 attention heads.

[0106] The D3QN decision module in this embodiment is composed of a deep Q network, a dual deep Q network, and a dueling deep Q network, which together form two dueling deep Q networks; the deep Q network estimates the state action value Q(s) through a neural network. t ,a t θ t ), where s t 、a t ,θ t are the state at time t, the selected action, and the network parameters to be trained; the dual-depth Q network reduces the overestimation problem caused by a single network decision by constructing an additional target network. The structure of the target network is the same as that of the original deep Q network, but the update frequency of the network parameters is different; the duel deep Q network decomposes the state value function into the state value function V(s) by inputting the features extracted by the fully connected layer into two sub-networks respectively. t ) and advantage function A(s t ), and finally sum the state value function and advantage function to output the state action value Q(s t ,a t θ t ,α t ,β t )=V(s t θ t ,α t )+A(s t ,a t θt ,β t ), where α t and β t are the parameters of the two sub-networks respectively, which enables the network to more accurately estimate the pros and cons of different actions.

[0107] Step S4: After the agent obtains the original observation state, it uses the feature fusion module to extract advanced spatiotemporal features, which are input into the D3QN decision module to obtain the agent action at the next moment. The agent executes the action and collects the reward and state of the road network environment feedback. The samples obtained by the interaction between the agent and the road network environment are stored in the experience pool. The data utilization rate is improved through the experience replay mechanism. The experience replay mechanism includes an experience pool with a maximum storage of 50. When the number exceeds the size of the experience pool, the first stored experience is discarded. Each experience in the experience pool represents a finite number of state transitions {s→s next} complete simulation process, the number of state transitions for each experience is In this embodiment, it is 720, so the entire experience pool has a total of 36,000 state transitions, s, s next They represent the states of the intersection before and after the transfer, with a certain number of simulation steps in between, which is determined by the hyperparameter decision frequency; each specific state transfer combination is represented by {s, a, s next ,r}, where s is the current intersection state, a is the selected action, and s next is the next state of the intersection after the action is executed, and r is the reward calculated based on the next state after the action is executed.

[0108] Step S5: During the training process, the agent's Q value needs to be further input into the value decomposition network, and stochastic gradient descent is used to update the agent's feature fusion module and D3QN decision module. Finally, the entire graph attention decision network is trained to achieve regional signal control.

[0109] In this embodiment, a centralized training and distributed execution training method is adopted. Centralized training means that all agents reuse a network and use the observed states of all agents to train the reused network. Distributed execution means that the trained reused network is deployed on all agents to make decisions separately. In addition, the training frequency of the network is 1, that is, all modules are trained once after each decision. The specific training includes the following steps:

[0110] Step S5.1: Set the value decomposition network used in training to VDN, the loss function to Huber loss function, and the optimizer to Adam optimizer;

[0111] Step S5.2: If the total number of experiences is less than the training start threshold, the training is terminated and the next round of simulation is continued. Otherwise, complete simulation experiences are randomly extracted from the experience pool according to the sampling number, and then a certain length of continuous state transition experiences are randomly extracted from each complete simulation experience according to the time series length. Finally, the experiences are processed into tensors that can be directly input into the network.

[0112] Step S5.3: Input the original features in the experience tensor into the multi-head graph attention network, and calculate the spatial fusion features according to the road network graph structure. The specific steps are as follows:

[0113] First, all the original features s of the agents are i Mapped to the embedding feature h i , set a linear mapping with a shared parameter W to embed the features h of all agents i Mapped to the shared linear space Wh i , obtain enhanced features;

[0114] For any two agents i and j, the enhanced features of the two are concatenated to obtain the joint feature Wh i ||Wh j ;

[0115] Set a linear mapping with a shared parameter a to map the joint features to the real number space to obtain the similarity coefficient e of agent i to all agents ij =a(Wh i ||Wh j ), thus the similarity coefficient matrix between all agents can be obtained;

[0116] Use the adjacency matrix as a mask to perform a mask operation on the similarity coefficient matrix, leaving only the similarity coefficients of agent i with its adjacent agents;

[0117] The similarity coefficient matrix after the mask operation is normalized using the softmax function to obtain the attention coefficient

[0118] According to the attention coefficient, the embedded features are weighted and summed to obtain new fusion features. Where σ is the activation function;

[0119] For the multi-head attention mechanism, each attention head outputs its own independent attention coefficient, that is, there are K independent attention coefficients between any agents i and j Among them, 1≤k≤K, so multiple fusion features are concatenated to obtain the output feature

[0120] Step S5.4: input the time-continuous fusion features in the experience sequence into the gated recurrent unit to obtain the final features of the fusion history information;

[0121] Step S5.5: Transform each state transition experience {s,a,s next ,r} corresponding to the final feature input D3QN decision network to obtain the state action value Q(s,a;θ) under the current state s, and obtain the best action for the next state through the target network and the maximum state-action value Step S5.6: Input the current and next state action values ​​of all agents into the value decomposition network to obtain an estimate of the global state action value Q(S,A;θ) = ∑Q(s,a;θ), Where S and S represent the current and next global states;

[0122] Step S5.7: Calculate target value based on reward Where γ is the discount factor;

[0123] Step S5.8: Calculate the loss Loss = Q(S, A; θ) - T of the target value and the current state action value and reversely update the network parameters of all modules according to the loss to obtain the final target model.

[0124] In this embodiment, in addition to the method proposed in the present invention, three different methods are additionally used as comparison objects, and the training simulation environment of all methods is the road network environment constructed as described above. The method proposed in the present invention is TSV, and the methods used for comparison are fixed timing, PPO and D3QN. Among them, the fixed timing is that the traffic lights execute a pre-set timing scheme, including a fixed phase sequence and phase duration, and the phase of the fixed timing is consistent with the phase of the reinforcement learning method. D3QN is the decision module of the method proposed in this article, which removes the feature fusion module before the D3QN decision module and the value decomposition network used for training. PPO is a proximal policy optimization algorithm, which is a policy-based reinforcement learning algorithm.

[0125] The above methods were trained 5 times, each with 150 rounds of training. Different random seeds were used for each training. The average of the 5 training results was used as the final effect comparison of the methods. The comparison indicators were the three indicators mentioned above, average travel time, vehicle arrival rate and standard average travel time. The comparison results of the three indicators are shown in the figure below. Figure 7 , Figure 8 and Fig. 9 shown. Figure 7 It shows that TSV and PPO obtain the smallest and similar average travel time, and TSV has a very small average travel time in the early stage of training. This is due to the aforementioned statistical error, so the vehicle arrival rate needs to be considered. Figure 8 It shows that TSV has the highest vehicle arrival rate. Considering the average travel time and vehicle arrival rate, by observing Fig. 9It can be seen that TSV obtains the minimum standard average travel time. In addition, it can be observed that TSV has the fastest convergence speed. This is because the feature fusion module integrates some historical information and observation information of other nodes in the road network, so that the decision network can calculate the local optimal action based on more comprehensive features, and the value decomposition network can effectively alleviate the value function estimation problem and action selection problem of multi-agents, and accelerate the convergence of the network from a global perspective.

[0126] The trained model can be used in the corresponding road network scenario in SUMO to achieve regional traffic control.

[0127] It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit it. For those of ordinary skill in the art, without departing from the concept of the present application, several modifications may be made, or some or all of the technical features may be replaced by equivalents, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A regional traffic signal control method based on graph attention and value decomposition reinforcement learning, characterized in that: The steps include: Step 1: Establish a simulation environment based on the road network environment and traffic flow data, and calculate the adjacency matrix as a graph structure based on the road network; Step 2: Collect traffic flow information, including phase, queue length, vehicle density, and phase switching permission, and design the state, action, and reward of reinforcement learning; Step 3: Construct a regional traffic signal control model, including a feature fusion module, a D3QN decision module, and a value decomposition network (VDN). The feature fusion module consists of a multi-head graph attention network and a gated recurrent unit. The state of each intersection is obtained through the multi-head graph attention network to obtain the output features of each intersection, and then the final features of the fused historical information are obtained through the gated recurrent unit. The D3QN decision module consists of two dueling networks, which take the final features of each intersection concatenated with the corresponding intersection identification number as input to obtain the state-action value; the VDN contains only one summation network, which approximates the global state-action value by summing the state-action values ​​of all input intersections; Step 4: Collect the interaction experience between the agent and the traffic simulation environment and store it in the experience pool; Step 5, sampling the experience pool, using the sampled samples to train the regional traffic signal control model, and completing the regional traffic signal control; Step 3: Construct a regional traffic signal control model, including a feature fusion module, a D3QN decision module, and a value decomposition network VDN. The specific method is as follows: Step 31, construct a feature fusion module, which is a series structure of a multi-head graph attention network and a gated recurrent unit, wherein the multi-head graph attention network is composed of multiple graph attention networks with consistent structures and inputs in parallel; (1) Multi-head Graph Attention Network For a single graph attention network, the state of each intersection is recorded as the original feature, and the embedded features are obtained through a 33×64 fully connected layer. All embedded features are input into a 64×64 fully connected layer to obtain enhanced features. The enhanced features of any two intersections are concatenated to obtain the joint feature. All joint features are output through a 128×1 fully connected layer to obtain the similarity coefficients between intersections, including the similarity coefficient of the intersection to itself. The similarity coefficients are arranged into a similarity coefficient matrix. Similar to the adjacency matrix, the elements in the similarity coefficient matrix represent the similarity coefficients between the intersections with corresponding row and column numbers. The similarity coefficient matrix is ​​masked through the adjacency matrix and then input into a softmax function to obtain the attention coefficient between intersections. The attention coefficient is used to weight the embedded features and activated using the elu function to obtain the fusion feature. The above are the fusion features obtained by a single graph attention network. The fusion features output by multiple graph attention networks are spliced ​​to obtain the output features of each intersection; (2) Gated Recurrent Unit The gated recurrent unit first inputs the output features of all intersections in the multi-head graph attention network into a 64×64 fully connected layer to obtain spatial features of uniform size. Before using the gated recurrent unit at the initial moment, it is necessary to initialize a historical time series feature with the same size as the spatial feature. All elements of the initialized historical time series feature are 0. The spatial features and historical time series features of all intersections are input into the GRU layer, and the final feature that integrates the historical information is output. The final feature is used as the historical time series feature at the next moment. Step 32, constructing a D3QN decision module under the condition of multi-agent shared parameters. The constructed D3QN decision module includes two dueling networks with the same structure and different parameters. Only the first dueling network is used when outputting decisions, and both dueling networks are used simultaneously when updating parameters. In order to distinguish different intersections, the duel network needs to add unique hot encoding of the intersection identification number to the features. The final features of each intersection output by the feature fusion module are spliced ​​with the corresponding intersection identification number, and input into an 80×128 fully connected layer to obtain the transition feature. The duel network has two network branches, the value function and the advantage function. The transition features are input into the value function network and the advantage function network respectively. The value function network is composed of two fully connected layers in series, with sizes of 128×128 and 128×1 respectively. The advantage function network is composed of two fully connected layers in series, with sizes of 128×128 and 128×8 respectively. The average difference of the output of the advantage function network is calculated and summed with the output of the value function network to obtain the state action value. According to the ε-greedy strategy, the action corresponding to the maximum value of the state action value is taken as the output strategy; Step 33, constructing a value decomposition network VDN used in training. The VDN only contains a summation network, which approximates the global state-action value by summing the state-action values ​​of all input intersections; Step 34, set the loss function to Huber loss function, the optimizer to Adam optimizer, and initialize the hyperparameters, including learning rate, discount factor, number of samples, experience pool size, time series length, decision frequency, training frequency, and number of training rounds.

2. The regional traffic signal control method based on graph attention and value decomposition reinforcement learning as claimed in claim 1, characterized in that: Step 1: Establish a simulation environment based on the road network environment and traffic flow data, and calculate the adjacency matrix as a graph structure based on the road network. The specific method is as follows: Step 11, compile a road network file and a vehicle flow file according to the actual control area requirements. The road network file includes the connection information between all intersections and roads, and the vehicle flow file includes the maximum acceleration, maximum speed, routing information and departure time of each vehicle; Step 12, establishing a traffic simulation environment according to the road network file and the traffic flow file; Step 13, calculate the adjacency matrix based on the road network connection information. The row and column numbers of the two-dimensional matrix represent the corresponding intersection numbers. The elements in the matrix indicate whether the intersections with the corresponding row and column numbers are connected. If there is a connection, the element is set to 1, otherwise it is set to 0.

3. The regional traffic signal control method based on graph attention and value decomposition reinforcement learning as claimed in claim 1, characterized in that: Step 2: Collect traffic flow information, including phase, queue length, vehicle density, and phase switching permission, and design the state, action, and reward of reinforcement learning. The specific method is as follows: (1) Status The state is represented by a one-dimensional matrix s = {P, min, d, q}, which contains four sub-state matrices: phase, queue length, vehicle density, and phase switching permission. Phase P is represented by one-hot encoding. The i'th phase P i’ Denoted as {0,...,P i’ ,...,0},P i’ is 1, and the rest are 0; the queue length q of the lane is represented by {q1,…,q i ,…,q n }, where the queue length of the i-th lane is q i =n si (l v +2.5) / l l , n si is the number of vehicles stopped on the lane, l v is the average length of vehicles on the lane, 2.5 is the minimum distance between vehicles, l l is the length of the lane; the vehicle density d of the lane is expressed as {d1,…,d i ,…,d n }, where the vehicle density of the i-th lane is d i =n i (l v +2.5) / l l , n i is the number of all vehicles on the lane; the phase switching permission min is recorded as {1} or {0}; (2) Action The action is set to the next phase selection of the signal light, the action space is all phases of the signal light, and the signal light is set to random phase switching, that is, switching from the current phase to any phase; (3) Rewards The reward r is the negative of the sum of the queue lengths of all lanes at the intersection.

4. The regional traffic signal control method based on graph attention and value decomposition reinforcement learning as claimed in claim 1, characterized in that: Step 4: Collect the interaction experience between the agent and the traffic simulation environment and store it in the experience pool, where: The experience pool is a python queue. The number of elements is at most the size of the experience pool. Each element is a python list. Each list represents a complete simulation process. Each element of the list represents a state transition {s,a,s next ,r}, s is the current state of all intersections, a is the action selected by all intersections, s next is the next state of all intersections after the action is executed, r is the reward calculated based on the next state, s and s next At intervals of a certain number of simulation steps, depending on the decision frequency, the agent interacts with the traffic simulation environment to generate experience. When the amount of experience reaches the upper limit of the experience pool size, the oldest experience is discarded.

5. The regional traffic signal control method based on graph attention and value decomposition reinforcement learning as claimed in claim 1, characterized in that: Step 5: Sample the experience pool and use the samples obtained to train the regional traffic signal control model to complete the regional traffic signal control. The specific method is as follows: Step 51, randomly extract experience samples from the experience pool according to the sampling quantity. If the number of samples in the experience pool is less than 1, then exit the training and continue sampling; otherwise, randomly intercept a continuous state transition from the extracted experience samples according to the time series length as input; Step 52: The original states s and s before and after the state transfer next The final features h and h before and after state transfer are obtained through the feature fusion module respectively. next ; Step 53, using the two constructed duel networks, input h and a into the first duel network to obtain the state action value Q(s, a; θ) in the current state, and then input h next Input the first dueling network to get the best action for the next state h next and a′ are input into the second duel network to obtain θ and are the network parameters of the two dueling networks respectively; Step 54, using the constructed VDN, input the current and next state action values ​​of all agents into the value decomposition network to obtain the global state action value estimate Q(S,A;θ)=∑Q(s,a;θ), Among them, S and S next Indicates the current and next global states; Step 55, calculate the target value based on the reward Where γ is the discount factor; Step 56, calculate the loss Loss = Q(S, A; θ) - T of the target value and the current state action value, and back-propagate based on the loss to update the network parameters of the multi-head graph attention network, gated recurrent unit and D3QN module.

6. A regional traffic signal control system based on graph attention and value decomposition reinforcement learning, characterized in that: Implement the regional traffic signal control method based on graph attention and value decomposition reinforcement learning as described in any one of claims 1-5 to realize regional traffic signal control based on graph attention and value decomposition reinforcement learning.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, implementing the method for regional traffic signal control based on graph attention and value decomposition reinforcement learning as described in any one of claims 1 to 5, and realizing regional traffic signal control based on graph attention and value decomposition reinforcement learning.

8. A computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for regional traffic signal control based on graph attention and value decomposition reinforcement learning according to any one of claims 1 to 5 is implemented to realize regional traffic signal control based on graph attention and value decomposition reinforcement learning.

Images