Traffic signal control method, device, electronic device and storage medium

Abstract

The present application provides a traffic signal control method, device, electronic device and storage medium, wherein the traffic signal control method includes: determining the influence weight of adjacent intersections on the target intersection; determining a first aggregation state for characterizing the influence of adjacent intersections on the target intersection; determining a second aggregation state for characterizing the traffic state of the target intersection taking into account the influence of adjacent intersections on the target intersection; determining the traffic signal change action of the target intersection; and generating a traffic control signal for the target intersection. The use of the aggregation state taking into account the influence of adjacent intersections on the target intersection to characterize the traffic state of the target intersection makes the depiction of the traffic conditions of the target intersection more accurate, realizes efficient and high-quality aggregation of multi-intersection traffic information, enables the above-mentioned traffic signal control method to better consider the traffic conditions of the target intersection, thereby obtaining the most preferred signal light change action, and effectively improves the effect of road network coordination and scheduling.

Description

Technical Field

[0001] The present application relates to the technical field of traffic signal control, and in particular to a traffic signal control method, device, electronic device and storage medium. Background Art

[0002] With the continuous development of information technology and the improvement of the level of urban information application, the construction of smart cities has emerged. As the top priority of smart city development, the realization of smart transportation is crucial. Smart transportation involves traffic control, disaster management, vehicle information and navigation, driver assistance, etc. The intelligent control of traffic signals can effectively alleviate traffic congestion, ensure people's daily travel safety, improve travel efficiency, promote social development, reduce environmental pollution, and ultimately achieve sustainable development. Therefore, the research and implementation of intelligent control algorithms for traffic lights is a very meaningful and practical direction in intelligent transportation. However, the existing traffic signal control methods have slow convergence speed and poor effect on road network coordination and scheduling. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a traffic signal control method, device, electronic device and storage medium to improve the effect of road network coordination and scheduling.

[0004] In a first aspect, an embodiment of the present application provides a traffic signal control method, comprising: determining an influence weight of the adjacent intersection on the target intersection based on a first traffic state of a target intersection and a second traffic state of an adjacent intersection; determining a first aggregation state for characterizing the influence of the adjacent intersection on the target intersection based on the second traffic state and the influence weight; determining a second aggregation state for characterizing the traffic state of the target intersection taking into account the influence of the adjacent intersection on the target intersection based on the first traffic state and the first aggregation state; determining a traffic signal change action of the target intersection based on the second aggregation state; and generating a traffic control signal of the target intersection based on the traffic signal change action of the target intersection.

[0005] In the process of implementing the above scheme, the aggregation state that takes into account the influence of adjacent intersections on the target intersection is used to characterize the traffic state of the target intersection, so that the characterization of the traffic conditions of the target intersection is more accurate, and efficient and high-quality aggregation of traffic information from multiple intersections is achieved, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, effectively improving the effect of road network coordination and scheduling.

[0006] In an implementation of the first aspect, the influence weight of the adjacent intersection on the target intersection is determined based on the first traffic state of the target intersection and the second traffic state of the adjacent intersection, including: determining the embedding layer weights of the target intersection and the adjacent intersection; using a graph attention network to learn the influence weight of the adjacent intersection on the target intersection based on the embedding layer weights of the target intersection and the first traffic state and the embedding layer weights and the second traffic state of the adjacent intersection.

[0007] In the process of implementing the above scheme, the influence of adjacent intersections on the target intersection is taken into consideration, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, effectively improving the effect of road network coordination and scheduling.

[0008] In an implementation of the first aspect, determining the first aggregation state based on the second traffic state and the influence weight includes: using a graph attention network to obtain the first aggregation state based on the embedding layer weights of the adjacent intersections, the influence weights and the second traffic state.

[0009] In the process of implementing the above scheme, the traffic status of the target intersection is characterized by the traffic status of the adjacent intersections of the target intersection, and the influence of the adjacent intersections on the target intersection is fully considered, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, and effectively improving the effect of road network coordination and scheduling.

[0010] In an implementation manner of the first aspect, determining a second aggregation state based on the first traffic state and the first aggregation state includes: aggregating the first traffic state and the first aggregation state to obtain a second aggregation state.

[0011] In the implementation process of the above scheme, the purpose of obtaining the second traffic state after aggregating the first traffic state and the second traffic state is to use the graph attention network to fit the influencing factors of the adjacent intersections on the target intersection, rather than fitting the complex future traffic conditions of the target intersection. The influence of the adjacent intersections on the target intersection is considered, and the state of the target intersection itself is added, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most preferred signal light change action, effectively improving the effect of road network coordination and scheduling.

[0012] In an implementation of the first aspect, determining the traffic signal change action at the target intersection based on the second aggregation state includes: using the second aggregation state as the input of the linear layer to obtain the sum of future reward values ​​that can be obtained by taking a certain action at the target intersection under the second aggregation state; and adopting the action that maximizes the sum of the future reward values ​​as the traffic signal change action at the target intersection.

[0013] In the implementation process of the above scheme, a second aggregated state that takes into account the impact of adjacent intersections on the target intersection and also aggregates the state of the target intersection itself is used as the input of the linear layer, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, effectively improving the effect of road network coordination and scheduling.

[0014] In an implementation manner of the first aspect, the first traffic state and the second traffic state both include: a traffic light phase at a current intersection, and a congestion parameter of an entry lane at the current intersection.

[0015] In the process of implementing the above scheme, the congestion parameters of the entrance lane in the current intersection are added to the current intersection state, so that the intersection state can better characterize the traffic conditions of the intersection and more accurately describe the traffic conditions of the target intersection, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, effectively improving the effect of road network coordination and scheduling.

[0016] In an implementation of the first aspect, the congestion parameter of the entry lane at the current intersection includes: the number of vehicles in the sensing section of the entry lane at the current intersection; the method for determining the number of vehicles in the sensing section of the entry lane at the current intersection is: determining the maximum speed Sp that can be reached by the vehicle in the entry lane at the current intersection max , the duration of the current traffic signal state Δt, and the number of vehicles currently queuing in the lane at the current intersection l; if the number of vehicles currently queuing in the lane at the current intersection l is less than the preset vehicle queue number threshold k, then Sp max ×Δt is determined as the sensing section, and the number of vehicles on the sensing section is obtained; if the number of vehicles currently queuing in the lane at the current intersection is not less than the preset vehicle queue threshold k, then Sp max ×Δt+R is determined as the sensing section, and the number of vehicles on the sensing section is obtained; wherein R is the dynamically adjusted length, and R is determined by the current number of queued vehicles l in the lane entering the current intersection, the preset vehicle queue number threshold k and the preset road congestion sensitivity parameter r.

[0017] In the process of implementing the above scheme, the number of vehicles in the sensing section of the lane entering the current intersection is added to the intersection state. This state can dynamically focus on the key section according to the congestion of the lane, taking into account the location information of the vehicle, and at the same time better characterize the degree of congestion of the target intersection, and more accurately characterize the traffic state of the target intersection, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, and effectively improving the effect of road network coordination and scheduling; at the same time, compared with the situation of using multiple parameters as the intersection state, the embodiment of the present application uses the intersection signal light phase and the number of vehicles in the sensing section of the lane entering the current intersection as the state, which ensures the simplicity of the state and is conducive to the rapid convergence of the above traffic signal control method.

[0018] In a second aspect, an embodiment of the present application provides a traffic signal control device, comprising:

[0019] An influence weight determination module, used to determine the influence weight of the adjacent intersection on the target intersection based on the first traffic state of the target intersection and the second traffic state of the adjacent intersection;

[0020] A first aggregation state determination module, configured to determine, based on the second traffic state and the influence weight, a first aggregation state for characterizing the influence of the adjacent intersection on the target intersection;

[0021] A second aggregation state determination module, configured to determine, based on the first traffic state and the first aggregation state, a second aggregation state for characterizing the traffic state of the target intersection taking into account the influence of the adjacent intersection on the target intersection;

[0022] A traffic signal change action determination module, configured to determine a traffic signal change action at the target intersection based on the second aggregation state;

[0023] The traffic control signal generating module is used to generate the traffic control signal of the target intersection based on the traffic signal changing action of the target intersection.

[0024] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the method provided by the first aspect or any possible implementation of the first aspect is executed.

[0025] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are read and run by the processor, the method provided by the first aspect or any possible implementation of the first aspect is executed.

[0026] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 A flow chart of a traffic signal control method provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the structure of a traffic signal control model provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of the structure of a traffic signal control device provided in an embodiment of the present application;

[0031] Figure 4 Represents each enhancement algorithm in the simulated data set Grid 3×3 Convergence effect diagram on ;

[0032] Figure 5 Represents each enhancement algorithm in the simulated data set Grid 6×6 Convergence effect diagram on ;

[0033] Figure 6 Represents each enhancement algorithm in the real data set C 16×3 -1 on the convergence effect diagram;

[0034] Figure 7 Represents each enhancement algorithm in the real data set B 3×4 -1 on the convergence effect diagram;

[0035] Figure 8 Represents each enhancement algorithm in the real data set B 3×4 -2 on the convergence effect diagram;

[0036] Fig. 9 Represents each enhancement algorithm in the real data set B 3×4 -3 on the convergence effect diagram;

[0037] Fig.10 Represents each enhancement algorithm in the real data set A 4×4-1 on the convergence effect diagram;

[0038] Fig.11 Represents each enhancement algorithm in the real data set A 4×4 -2 on the convergence effect diagram;

[0039] Fig.12 Represents each enhancement algorithm in the real data set A 4×4 -3 on the convergence effect diagram;

[0040] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0043] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0044] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0045] At present, many important achievements have been made in the research field of intelligent traffic signal control. Traditional traffic signal control algorithms can be traced back to SCOOT (Split Cycle Offset Optimizing Technique) and SCATS (Sydney Coordinated Adaptive Traffic System), which rely on artificial pre-designed rules and promote a series of control algorithms based on predefined rules, which can be divided into single-intersection independent control methods and multi-intersection coordinated control methods. However, this method based on empirical observation cannot cope with today's complex and changeable traffic conditions. Later, different artificial intelligence technologies were applied to traffic signal control, such as fuzzy logic algorithms, swarm intelligence, and the most widely used reinforcement learning (RL). RL-based methods model the traffic signal control problem as a Markov decision problem and select actions through state information collected in the environment to maximize rewards without prior knowledge. In recent years, thanks to breakthroughs in deep learning research, RL combined with deep learning (DL), called deep reinforcement learning (DRL), has become the most effective learning framework in traffic signal control. It combines the powerful nonlinear fitting ability of deep neural networks with the advantages of reinforcement learning in solving control and decision-making problems, and extends the reinforcement learning method to more complex tasks. After that, researchers mainly improved the performance of DRL-based methods by extracting valid states and promoting the effective aggregation of information between intersections.

[0046] There is a fundamental trade-off in state design. The more comprehensive the information contained in the state, the more difficult it is to collect and preprocess the state. In addition, as the exploration space of the state becomes larger, the convergence speed of the model will slow down or even fail to converge, especially for multi-intersection models with exponential growth in space. In terms of model design, the existing technology has promoted more efficient information aggregation between multiple intersections through centralized control, communication between adjacent intersections, etc., especially in combination with the emerging graph neural network (GNN). RL combined with GNN has become a better choice for joint control of multiple intersections, and has reached advanced levels in many public data sets.

[0047] The method based on graph neural network regards each intersection as a node in the graph, collects effective traffic information from the environment as the state of the node, and learns the relationship between the node and the adjacent nodes through a neural network with shared weights, thereby achieving effective information aggregation between multiple intersections. However, in actual application, it is found that the use of GNN combined with RL cannot achieve a good road network coordination and scheduling effect. After a lot of research, the inventor found that the key to not being able to achieve a good scheduling effect is:

[0048] The mutual influence between the intersections in the road network is not considered: Since traffic intersections do not exist independently, vehicles in adjacent intersections adjacent to the target intersection have a certain probability of entering the target intersection, thereby affecting the target intersection. However, the traffic signal control method in the prior art does not consider the influence of adjacent intersections on the target intersection.

[0049] In view of the above defects in the prior art, the inventor proposes the following ideas:

[0050] In the aggregation of multi-intersection information, the representation state of the target intersection obtained by aggregating the traffic information of adjacent intersections is superimposed on the traffic state of the target intersection itself. At the same time, the model is changed from fitting complex future states based on the states of adjacent intersections to considering the impact of adjacent intersections on the target intersection, thereby improving the road network coordination and scheduling effect of the traffic signal control method.

[0051] See also Figure 1 , an embodiment of the present application provides a traffic signal control method, comprising:

[0052] Step S110: determining the influence weight of the adjacent intersection on the target intersection based on the first traffic state of the target intersection and the second traffic state of the adjacent intersection;

[0053] Step S120: determining a first aggregation state for characterizing the influence of the adjacent intersection on the target intersection based on the second traffic state and the influence weight;

[0054] Step S130: Based on the first traffic state and the first aggregation state, determining a second aggregation state for characterizing the traffic state of the target intersection taking into account the influence of the adjacent intersection on the target intersection;

[0055] Step S140: determining a traffic signal change action at a target intersection based on the second aggregation state;

[0056] Step S150: Generate a traffic control signal for the target intersection based on the traffic signal change action for the target intersection.

[0057] In the process of implementing the above scheme, the aggregation state that takes into account the influence of adjacent intersections on the target intersection is used to characterize the traffic state of the target intersection, so that the characterization of the traffic conditions of the target intersection is more accurate, and efficient and high-quality aggregation of traffic information from multiple intersections is achieved, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, effectively improving the effect of road network coordination and scheduling.

[0058] The above steps S110 to S150 are described in detail below:

[0059] First, step S110 is described in detail:

[0060] As an optional implementation of the above traffic signal control method, the first traffic state and the second traffic state in step S110 both include: the signal light phase of the current intersection, and the congestion parameter of the lane entering the current intersection. This implementation, for example, takes the combination of the signal states of each lane in each direction within the signal light release time as the signal light phase of the current intersection within the signal light release time; the congestion parameter may be the number of vehicles waiting in the lane entering, the number of vehicles in a fixed road section length, or the length of the queue waiting to pass through the intersection.

[0061] It should be noted that the above-mentioned entry lane refers to the lane where the vehicle enters the intersection. Corresponding to the entry lane is the exit lane, which refers to the lane where the vehicle exits the intersection. In the road network, the exit lane of the current intersection is also the entry lane of the intersection adjacent to the current intersection.

[0062] In the process of implementing the above scheme, the congestion parameters of the entrance lane in the current intersection are added to the current intersection state, so that the intersection state can better characterize the traffic conditions of the intersection and more accurately describe the traffic conditions of the target intersection, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, effectively improving the effect of road network coordination and scheduling.

[0063] As an optional implementation of the above traffic signal control method, the congestion parameter of the current entry lane in the intersection can be: the number of vehicles in the sensing section of the current entry lane in the intersection. The method for determining the number of vehicles in the sensing section of the current entry lane in the intersection is: determining the maximum speed Sp that the vehicles in the current entry lane can reach max , the duration of the current traffic signal state Δt, and the number of vehicles currently queuing in the lane at the current intersection l; if the number of vehicles currently queuing in the lane at the current intersection l is less than the preset vehicle queue number threshold k, then Sp max ×Δt is determined as the sensing section, and the number of vehicles on the sensing section is obtained; if the number of vehicles currently queuing in the lane at the current intersection is not less than the preset vehicle queue threshold k, then Spmax ×Δt+R is determined as the sensing section, and the number of vehicles on the sensing section is obtained; wherein R is the dynamically adjusted length, and R is determined by the current number of queued vehicles l in the lane entering the current intersection, the preset vehicle queue number threshold k, and the preset road congestion sensitivity parameter r. For example, in this implementation, the number of vehicles in the sensing section of the lane entering the current intersection is added to the intersection state, and the number of vehicles in the sensing section of the lane entering the current intersection is called NOV-LADLE (Number Of Vehicle on LAnewith Dynamic Length, the number of vehicles in the dynamic length section), wherein the method for determining the sensing section DL is:

[0064]

[0065] Among them, Sp max is the maximum vehicle speed that the vehicles in the lane entering the current intersection can theoretically reach; Δt is the duration of the current traffic signal state, that is, the decision time; l is the number of vehicles currently queuing in the lane entering the current intersection; k is the preset vehicle queue number threshold; r is the preset road congestion sensitivity parameter; the dynamic adjustment length R = r×(lk).

[0066] In the above embodiment, Sp max ×Δt is called the effective distance, which means that vehicles within this distance have a chance to pass through the intersection. The core idea of ​​NOV-LADLE is to focus on these vehicles when the road is unobstructed; the preset vehicle queue number threshold k is set to half the effective distance Sp max ×Δt is the number of vehicles that can be accommodated within the length range; r is a hyperparameter used to adjust the dynamic adjustment length R. When the number of vehicles currently queuing in the lane at the current intersection increases, the larger the parameter r is, the larger the value of the dynamic adjustment length R is.

[0067] In the process of implementing the above scheme, the number of vehicles in the sensing section of the lane entering the current intersection is added to the intersection state. This state can dynamically focus on the key section according to the congestion of the lane, taking into account the location information of the vehicle, and at the same time better characterize the degree of congestion of the target intersection, and more accurately characterize the traffic state of the target intersection, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, and effectively improving the effect of road network coordination and scheduling; at the same time, compared with the situation of using multiple parameters as the intersection state, the embodiment of the present application uses the intersection signal light phase and the number of vehicles in the sensing section of the lane entering the current intersection as the state, which ensures the simplicity of the state and is conducive to the rapid convergence of the above traffic signal control method.

[0068] As an optional implementation of the above-mentioned traffic signal control method, step S110 determines the influence weight of the adjacent intersection on the target intersection based on the first traffic state of the target intersection and the second traffic state of the adjacent intersection, including: determining the embedding layer weights of the target intersection and the adjacent intersection; using a graph attention network, according to the embedding layer weight of the target intersection and the first traffic state and the embedding layer weight and the second traffic state of the adjacent intersection, learning the influence weight of the adjacent intersection on the target intersection. This implementation, for example: input the road network connection diagram and the traffic state of each intersection into the graph attention network, obtain the embedding layer weight of each intersection, and then calculate the influence weight of intersection j on intersection i by the following formula:

[0069] e ij =a[(h i W t ),(h j W s )]

[0070] Among them, h i and h j are the traffic status of intersection i and intersection j respectively; W t and W s are the embedding layer weights of intersection i and intersection j in the graph attention network respectively; a[] represents the shared attention mechanism, and a[] can be in the form of inner product, that is:

[0071] e ij =(h i W t )·(h j W s )

[0072] In the process of implementing the above scheme, the influence of adjacent intersections on the target intersection is taken into consideration, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, effectively improving the effect of road network coordination and scheduling.

[0073] As an optional implementation of the above traffic signal control method, before step S110, the above traffic signal control method further includes: using a multilayer perceptron (MLP) to encode the traffic state of the intersection, specifically:

[0074]

[0075] Among them, h i is the coded traffic status; represents the state of the i-th intersection at time t, including the signal light phase of the intersection and the number of vehicles in the sensing section of the current intersection; W e and be is the weight matrix of the multilayer perceptron; σ is the activation function, and the ReLU function can be used.

[0076] The following describes step S120 in detail:

[0077] As an optional implementation of the above-mentioned traffic signal control method, step S120 determines the first aggregation state based on the second traffic state and the influence weight, including: using a graph attention network to obtain the first aggregation state based on the embedding layer weights, influence weights and second traffic state of adjacent intersections. This implementation, for example, uses the following formula to obtain the first aggregation state:

[0078]

[0079] Among them, H i is the first aggregation state; σ is the activation function; α ij In order to normalize the influence weight of intersection j on intersection i, it should be pointed out that intersection i in the above formula is the target intersection, intersection j is the adjacent intersection of target intersection i, and N i is the set of all adjacent intersections of the target intersection i; W q and b q are weight parameters, and these two parameters are obtained by learning the graph attention network.

[0080] In the process of implementing the above scheme, the traffic status of the target intersection is characterized by the traffic status of the adjacent intersections of the target intersection, and the influence of the adjacent intersections on the target intersection is fully considered, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, and effectively improving the effect of road network coordination and scheduling.

[0081] The influence weight e of intersection j on intersection i ij Normalize to get α ij The method is:

[0082]

[0083] Among them, softmax is a normalized exponential function.

[0084] It should be noted that the first aggregation state can be aggregated using a layer of graph neural network, or a multi-layer graph neural network can be used to expand the receptive field of the aggregated information:

[0085]

[0086] …

[0087]

[0088]

[0089] in, is the output of the L-th layer graph neural network, representing the vector used to characterize the traffic state of the target intersection obtained by aggregating the traffic states of adjacent intersections of the target intersection L times.

[0090] The following describes step S130 in detail:

[0091] As an optional implementation of the above traffic signal control method, step S130 determines the second aggregation state based on the first traffic state and the first aggregation state, including: aggregating the first traffic state with the first aggregation state to obtain the second aggregation state. This implementation is for example:

[0092] In the field of traffic signal control, the joint modeling of multiple intersections is to share information and promote cooperation. Its essence is to make each intersection consider the impact of other intersections in the future. The current traffic conditions of the intersection are still the main basis for decision-making. Therefore, the first traffic state and the first aggregated state of the target intersection are aggregated in the following way to obtain the second aggregated state:

[0093] H′ i =H i +h i =GAT(h i )+h i

[0094] Among them, H i The first aggregated state is aggregated using the second traffic state of the adjacent intersection; i is the first traffic state of the target intersection; H′ i The second aggregated state is obtained by aggregating the first traffic state and the first aggregated state.

[0095] In the implementation process of the above scheme, the purpose of obtaining the second traffic state after aggregating the first traffic state and the second traffic state is to use the graph attention network to fit the influencing factors of the adjacent intersections on the target intersection, rather than fitting the complex future traffic conditions of the target intersection. The influence of the adjacent intersections on the target intersection is considered, and the state of the target intersection itself is added, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most preferred signal light change action, effectively improving the effect of road network coordination and scheduling.

[0096] The following describes step S140 and step S150 in detail:

[0097] As an optional implementation of the above-mentioned traffic signal control method, step S140 determines the traffic signal change action of the target intersection based on the second aggregation state, including: using the second aggregation state as the input of the linear layer, obtaining the sum of future reward values ​​that can be obtained by taking a certain action at the target intersection under the second aggregation state; and adopting the action that maximizes the sum of future reward values ​​as the traffic signal change action at the target intersection. This implementation, for example: modeling the traffic signal control problem as a Markov decision problem. Each intersection with a traffic light in the road network is regarded as a road network node, and the execution signal of the next road network node is selected according to its intersection state and the traffic state of its adjacent intersections. The purpose of this problem is to minimize the average travel time of all vehicles. The above problem can be described as (S, A, P, R, γ), where:

[0098] State space S, state s t ∈S represents the traffic state of the intersection at time t, which is the decision basis for making a signal change action. The embodiment of the present application selects the signal light phase of the current intersection and the congestion parameter of the entry lane in the current intersection. At the same time, considering the state extraction, model learning difficulty, and the final optimization effect, the congestion parameter of the entry lane in the previous intersection is improved, and the number of vehicles in the sensing section of the entry lane in the current intersection is used as the congestion parameter of the entry lane in the current intersection.

[0099] Action Space A, Action At the intersection at time t, choose to open the corresponding lane at the next time.

[0100] Transition probability P, state s at a given time t t and action a t , the intersection will be based on the transition probability P(s t+1 |s t ,a t )Transfer to the next state s t+1 .

[0101] Reward R, The reward obtained at the intersection at time t.

[0102] The discount factor γ represents the degree of influence of controlling future rewards. The method of calculating the total reward using the discount factor is:

[0103]

[0104] in, is the total reward obtained by intersection i at time t; T is the total time step of a complete round; γ t-τ The discount rate for the reward of the tth step thereafter.

[0105] The state in the state space S is the second aggregated state. By inputting the above (S, A, P, R, γ) into the linear layer, the sum of the future reward values ​​that can be obtained by taking a certain action at the target intersection can be obtained. Then, the action that maximizes the sum of the future reward values ​​is used as the traffic signal change action of the target intersection, so that the target intersection can obtain the maximum future reward value.

[0106] The linear layer calculates the sum of future rewards, or Q values, that can be obtained by taking an action at the target intersection in the following way:

[0107] q i =H i 'W p +b p

[0108] Among them, W p and b p is the linear layer parameter, H i ' is the second polymerization state.

[0109] In the implementation process of the above scheme, a second aggregated state that takes into account the impact of adjacent intersections on the target intersection and also aggregates the state of the target intersection itself is used as the input of the linear layer, so that the above traffic signal control method can better consider the traffic conditions of the target intersection, thereby obtaining the most optimal signal light change action, effectively improving the effect of road network coordination and scheduling.

[0110] After the action that maximizes the sum of future reward values ​​is used as the traffic signal change action of the target intersection in step S140, the action is converted into a traffic light control signal to control the traffic light change of the target intersection in step S150.

[0111] See also Figure 2 The above traffic signal control method can be summarized as a traffic signal control model. In the embodiment of the present application, the traffic signal control model is called KeyLight model 200, which includes:

[0112] A multi-layer perceptron 210 is used to encode the traffic status of the intersection;

[0113] GAT220 with residual structure, used to aggregate the traffic status of intersections;

[0114] The linear layer 230 is used to further process the output of the GAT 220 with a residual structure to obtain a Q value that meets the final dimension.

[0115] It should be pointed out that the above model can be trained for reinforcement learning based on the Ape-X framework. Ape-X is a distributed training framework for offline reinforcement learning algorithms such as DQN (Deep Q-learning) and DDPG (Deep Deterministic Policy Gradient). It uses multiple Actors that contain both the model and the environment for sampling, and then puts the collected sample data into an experience replay pool (Replay Buffer) for learner training. For specific training methods, please refer to the existing technology.

[0116] In order to verify the performance of the above traffic signal control method, the following comparative experiments were conducted in the embodiment of the present application:

[0117] The proposed method is experimentally verified using CityFlow, an open-source and efficient traffic simulator that supports flexible customization of road networks and traffic flows and is used to test traditional or reinforcement learning-based traffic control algorithms.

[0118] (1) Dataset selection

[0119] In order to fully compare the performance of the existing methods and the method proposed in the embodiment of the present application, the embodiment of the present application uses a classic public data set in the prior art, which includes 2 sets of simulation data and 3 sets of real scene data sets from City A, City B and City C.

[0120] In the simulated data set, the generation of traffic conforms to uniform distribution. In order to better compare the differences in scheduling results, the traffic in the horizontal direction is denser than the traffic in the vertical direction. It should be noted that the embodiment of the present application defines the lanes from west to east or from east to west as horizontal lanes, and the lanes from north to south or from south to north as vertical lanes. In contrast, the traffic on the lane is defined as horizontal traffic or vertical traffic. Its data attributes are shown in Table 1:

[0121] Table 1 Data attributes of the simulated data set

[0122]

[0123] Among them, Grid m×n The subscript m×n indicates the intersection distribution, Bi means both bidirectional lanes are open, and Uni means only one-way lane is open;

[0124] In the real data set, the two data sets from City A and City B both contain three data sets with different traffic flow scales and distributions. Their data attributes are shown in Table 2:

[0125] Table 2 Data attributes of the real dataset

[0126]

[0127] (2) Environment Settings

[0128] The environment settings used in the comparative experiments involved in the embodiments of the present application are shown in Table 3:

[0129] Table 3 Comparative experimental environment settings

[0130] property value unit length 5.0 m width 2.0 m minimum acceleration 2.0 <![CDATA[m / s 2 ]]> Maximum acceleration 4.5 <![CDATA[m / s 2 ]]> Normal acceleration 2.0 <![CDATA[m / s 2 ]]> Normal deceleration 4.5 <![CDATA[m / s 2 ]]> Minimum acceptable distance to the vehicle ahead 2.5 m Maximum speed 11.11 m / s

[0131] (3) Comparison method settings

[0132] In order to demonstrate the superiority of the traffic signal control method proposed in the embodiment of the present application, experiments were conducted on the following traditional traffic signal control methods and a recently proposed reinforcement learning-based algorithm:

[0133] The traditional method of setting up is as follows:

[0134] a.FixedTime: timing signal control;

[0135] This method sets the next action according to a predefined sequence and period and is widely used in stable traffic flow.

[0136] b.MaxPressure: One of the classic traffic signal control methods based on mature traffic theory. It releases the lane with the greatest pressure. The lane pressure is calculated by subtracting the number of vehicles entering the lane from the number of vehicles leaving the lane.

[0137] c.SOTL (Self organizing traffic lights): self-organizing traffic light control;

[0138] This method changes the intersection phase if and only if the number of vehicles on the entering lane exceeds a predefined threshold.

[0139] The reinforcement learning method set up is as follows:

[0140] a.GCN (Graph Convolutional Network): Integrate information of adjacent intersections based on graph convolutional neural network to promote cooperation between multiple intersections.

[0141] b.CoLight: It is a method in the prior art that uses a graph attention network to control traffic lights at multiple intersections. This method uses rules to determine the adjacent intersections of the target intersection, and the number of adjacent intersections of each target intersection is predefined as 3. This method integrates the information of adjacent intersections based on the graph attention network to promote cooperation between multiple intersections.

[0142] c.CoLight+Res: Adds a residual structure based on the above CoLight method;

[0143] d.CoLight+NOV-LADLE: Based on the above CoLight method, NOV-LADLE is used for the intersection status.

[0144] (4) Evaluation indicators

[0145] According to existing research, the performance of the above method is evaluated using average travel time and throughput in seconds. Vehicle travel time is defined as the time difference between a vehicle entering the system and leaving the system. Throughput is the number of vehicles that complete the journey in the road network within a period of time, which is set to 3600s in the embodiment of the application. All reinforcement learning-based algorithms are tested using more than three different random seeds, and all the following results represent their best performance.

[0146] (5) Performance comparison

[0147] a. Average travel time and throughput

[0148] Tables 4 and 5 show the performance comparison between the traffic signal control method proposed in the embodiment of the present application, namely the KeyLight algorithm and other traditional algorithms and the RL-based model.

[0149] Table 4 Results of each algorithm on real data sets of City A and City B

[0150]

[0151] Table 5 Results of each algorithm on the simulated dataset and the real dataset of City C

[0152]

[0153] It can be seen from Table 4 and Table 5 that:

[0154] 1. The traffic signal control method proposed in the embodiment of the present application, namely the KeyLight method, achieved the best performance on both simulated data sets and real data sets, proving the superiority and stability of the KeyLight method in multiple scenarios.

[0155] 2. The real data sets of City A and City B are compared with the simulated data sets. The traffic flow in the real data sets is complex and changeable, and the average travel time of KeyLight is reduced by more than 10%. Especially in City A, where the average and instantaneous traffic flow is the largest, 4×4 -2 and A 4×4-3 dataset, the KeyLight method achieved great improvements. Compared with CoLight, the maximum reduction rate of average travel time reached 23.44%, and the maximum increase in throughput was as high as 114.

[0156] 3. The application of access residual structure and NOV-LADLE state respectively significantly improves the model performance, and using both can further improve the performance.

[0157] b. Training efficiency

[0158] See also Figures 4 to 12 , the embodiment of the present application provides the convergence effect diagram of each reinforcement learning algorithm on each data set. It can be seen that the application of the residual structure and the NOV-LADLE state makes the model training more stable and the convergence speed faster, especially in the real data set. And the traffic signal control method proposed in the embodiment of the present application uses the residual structure and the NOV-LADLE state at the same time to achieve the best training effect.

[0159] (6) Summary of comparative experiments

[0160] The embodiment of the present application provides a traffic signal control method, which uses a new state NOV-LADLE for reinforcement learning method to control traffic signals. NOV-LADLE can dynamically focus on key sections according to the congestion of the lanes, taking into account the location information of the vehicle to ensure the simplicity of the state. The introduction of the residual structure in the GAT model explicitly increases the importance of the state of the intersection itself, which is more in line with reality, making the model training more stable, while also improving the model performance and promoting the efficient aggregation of traffic information at multiple intersections. A large number of experimental results on simulated data sets and real-world data sets verify the effectiveness of the traffic signal control method proposed in the embodiment of the present application. In general, the above-mentioned traffic signal control method uses reinforcement learning to avoid artificial induction of prior knowledge, directly learns the optimal scheduling strategy from attempts, and realizes intelligent control of coordinated scheduling of multiple intersections; based on the graph deep neural network, the road network is modeled at multiple intersections, so as to coordinate and schedule the road network, and realize efficient cooperative scheduling of the road network.

[0161] See also Figure 3 Based on the same inventive concept, the embodiment of the present application further provides a traffic signal control device 300, comprising:

[0162] An influence weight determination module 310, configured to determine an influence weight of an adjacent intersection on the target intersection based on a first traffic state of the target intersection and a second traffic state of an adjacent intersection;

[0163] A first aggregation state determination module 320, configured to determine a first aggregation state for characterizing the influence of the adjacent intersection on the target intersection based on the second traffic state and the influence weight;

[0164] A second aggregation state determination module 330, configured to determine, based on the first traffic state and the first aggregation state, a second aggregation state for characterizing the traffic state of the target intersection taking into account the influence of the adjacent intersection on the target intersection;

[0165] A traffic signal change action determination module 340, configured to determine a traffic signal change action at the target intersection based on the second aggregation state;

[0166] The traffic control signal generating module 350 is used to generate a traffic control signal for the target intersection based on the traffic signal changing action for the target intersection.

[0167] As an optional implementation of the above-mentioned traffic signal control device, the influence weight determination module 310 determines the influence weight of the adjacent intersection on the target intersection based on the first traffic state of the target intersection and the second traffic state of the adjacent intersection, including: determining the embedding layer weights of the target intersection and the adjacent intersection; using a graph attention network to learn the influence weight of the adjacent intersection on the target intersection according to the embedding layer weights and the first traffic state of the target intersection and the embedding layer weights and the second traffic state of the adjacent intersection.

[0168] As an optional implementation of the above-mentioned traffic signal control device, the first aggregation state determination module 320 determines the first aggregation state based on the second traffic state and the influence weight, including: using a graph attention network to obtain the first aggregation state based on the embedding layer weights of the adjacent intersections, the influence weights and the second traffic state.

[0169] As an optional implementation of the above-mentioned traffic signal control device, the second aggregation state determination module 330 determines the second aggregation state based on the first traffic state and the first aggregation state, including: aggregating the first traffic state and the first aggregation state to obtain the second aggregation state.

[0170] As an optional implementation of the above-mentioned traffic signal control device, the traffic signal change action determination module 340 determines the traffic signal change action of the target intersection based on the second aggregation state, including: using the second aggregation state as the input of the linear layer, obtaining the sum of future reward values ​​that can be obtained by taking a certain action at the target intersection under the second aggregation state; and adopting the action that maximizes the sum of the future reward values ​​as the traffic signal change action of the target intersection.

[0171] As an optional implementation manner of the above-mentioned traffic signal control device, the first traffic state and the second traffic state both include: a signal light phase at a current intersection, and a congestion parameter of an entry lane at the current intersection.

[0172] As an optional implementation of the above-mentioned traffic signal control device, the congestion parameter of the current entry lane at the intersection includes: the number of vehicles in the sensing section of the current entry lane at the intersection. The traffic signal control device 300 also includes: a vehicle number determination unit in the sensing section of the current entry lane at the intersection, specifically: determining the maximum speed Sp that the vehicle in the current entry lane can reach max , the duration of the current traffic signal state Δt, and the number of vehicles currently queuing in the lane at the current intersection l; if the number of vehicles currently queuing in the lane at the current intersection l is less than the preset vehicle queue number threshold k, then Sp max ×Δt is determined as the sensing section, and the number of vehicles on the sensing section is obtained; if the number of vehicles currently queuing in the lane at the current intersection is not less than the preset vehicle queue threshold k, then Sp max ×Δt+R is determined as the sensing section, and the number of vehicles on the sensing section is obtained; wherein R is the dynamically adjusted length, and R is determined by the current number of queued vehicles l in the lane entering the current intersection, the preset vehicle queue number threshold k and the preset road congestion sensitivity parameter r.

[0173] Fig.13 A schematic diagram of an electronic device provided in an embodiment of the present application. Fig.13 The electronic device 400 includes: a processor 410, a memory 420 and a communication interface 430. These components are interconnected and communicate with each other through a communication bus 440 and / or other forms of connection mechanisms (not shown).

[0174] The memory 420 includes one or more (only one is shown in the figure), which may be, but not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The processor 410 and other possible components may access the memory 420 and read and / or write data therein.

[0175] The processor 410 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 410 can be a general-purpose processor, including a central processing unit (CPU), a micro control unit (MCU), a network processor (NP) or other conventional processors; it can also be a special-purpose processor, including a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0176] The communication interface 430 includes one or more (only one is shown in the figure), which can be used to communicate directly or indirectly with other devices to exchange data. For example, the communication interface 430 can be an Ethernet interface; it can be a mobile communication network interface, such as an interface of a 3G, 4G, or 5G network; or it can be other types of interfaces with data transceiving functions.

[0177] One or more computer program instructions may be stored in the memory 420 , and the processor 410 may read and execute these computer program instructions to implement the traffic signal control method provided in the embodiment of the present application and other desired functions.

[0178] Understandably, Fig.13 The structure shown is for illustration only. The electronic device 400 may also include Fig.13 More or fewer components as shown, or with Fig.13 Different configurations are shown. Fig.13 Each component shown in the figure can be implemented by hardware, software or a combination thereof. For example, the electronic device 400 can be a single server (or other device with computing processing capabilities), a combination of multiple servers, a cluster of a large number of servers, etc., and can be both a physical device and a virtual device.

[0179] The present application also provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are read and executed by a computer processor, the traffic signal control method provided by the present application is executed. For example, the computer-readable storage medium can be implemented as Fig.13The memory 420 in the electronic device 400.

[0180] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0181] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0183] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A traffic signal control method, characterized in that: include: Determining, based on a first traffic state of a target intersection and a second traffic state of an adjacent intersection, an influence weight of the adjacent intersection on the target intersection; Based on the second traffic state and the influence weight, determining a first aggregation state for characterizing the influence of the adjacent intersection on the target intersection; Based on the first traffic state and the first aggregation state, determining a second aggregation state for characterizing the traffic state of the target intersection taking into account the influence of the adjacent intersection on the target intersection; Based on the second aggregation state, determining a traffic signal change action at the target intersection; generating a traffic control signal for the target intersection based on the traffic signal change action for the target intersection; The determining of the first aggregate state based on the second traffic state and the influence weight comprises: using a graph attention network to obtain the first aggregate state based on the embedding layer weights of the adjacent intersections, the influence weights and the second traffic state; The method for obtaining the first aggregation state is: in, is the first polymerization state; is the activation function; The intersection after normalization j On the intersection i The influence weight of intersection i The target intersection is j Target intersection i The adjacent intersections, By target intersection i The set of all adjacent intersections of ; and is the weight parameter; For intersection The second traffic state; Determining a second aggregate state based on the first traffic state and the first aggregate state includes: aggregating the first traffic state and the first aggregate state to obtain a second aggregate state; The method for obtaining the second aggregation state is: in, A first aggregated state aggregated using a second traffic state of an adjacent intersection; is the first traffic state of the target intersection; A second aggregated state obtained by aggregating the first traffic state and the first aggregated state; For the graph attention network; The determining of the traffic signal changing action of the target intersection based on the second aggregation state includes: taking the second aggregation state as the input of the linear layer, obtaining the sum of future reward values ​​that can be obtained by taking a certain action at the target intersection under the second aggregation state; and adopting the action that maximizes the sum of the future reward values ​​as the traffic signal changing action of the target intersection.

2. The traffic signal control method according to claim 1, characterized in that: The determining, based on the first traffic state of the target intersection and the second traffic state of the adjacent intersection, the influence weight of the adjacent intersection on the target intersection comprises: Determining embedding layer weights of the target intersection and the adjacent intersections; A graph attention network is adopted to learn the influence weight of the adjacent intersection on the target intersection according to the embedding layer weight and the first traffic state of the target intersection and the embedding layer weight and the second traffic state of the adjacent intersection.

3. The traffic signal control method according to any one of claims 1 to 2, characterized in that: The first traffic state and the second traffic state both include: The signal light phase at the current intersection, and the congestion parameters of the entry lane at the current intersection.

4. The traffic signal control method according to claim 3, characterized in that: The congestion parameters of the lane in the current intersection include: The number of vehicles in the sensing section of the current intersection entering the lane; The method for determining the number of vehicles in the sensing section of the lane entering the current intersection is: Determine the maximum speed that a vehicle can reach in the current intersection entering the lane , Current traffic signal status duration , and the number of vehicles currently queuing in the lane at the current intersection l ; If the number of vehicles currently queuing in the lane at the current intersection l Less than the preset vehicle queue threshold k , then Determine the sensing section and obtain the number of vehicles on the sensing section; If the number of vehicles currently queuing in the lane at the current intersection l Not less than the preset vehicle queue threshold k , then Determine the sensing section and obtain the number of vehicles on the sensing section; in, To dynamically adjust the length, The number of vehicles currently queuing in the lane entering the current intersection l , preset vehicle queue number threshold k and preset road congestion sensitivity parameters r jointly determined; The dynamically adjusted length The calculation method is: .

5. A traffic signal control device, characterized in that: include: An influence weight determination module, used to determine the influence weight of the adjacent intersection on the target intersection based on the first traffic state of the target intersection and the second traffic state of the adjacent intersection; A first aggregation state determination module, configured to determine, based on the second traffic state and the influence weight, a first aggregation state for characterizing the influence of the adjacent intersection on the target intersection; A second aggregation state determination module, configured to determine, based on the first traffic state and the first aggregation state, a second aggregation state for characterizing the traffic state of the target intersection taking into account the influence of the adjacent intersection on the target intersection; A traffic signal change action determination module, configured to determine a traffic signal change action at the target intersection based on the second aggregation state; A traffic control signal generating module, used for generating a traffic control signal for the target intersection based on the traffic signal changing action of the target intersection; The first aggregation state determination module determines a first aggregation state for characterizing the influence of the adjacent intersection on the target intersection based on the second traffic state and the influence weight, including: using a graph attention network to obtain a first aggregation state based on the embedding layer weights of the adjacent intersection, the influence weight and the second traffic state; The method for obtaining the first aggregation state is: in, is the first polymerization state; is the activation function; The intersection after normalization j On the intersection i The influence weight of intersection i The target intersection is j Target intersection i The adjacent intersections, By target intersection i The set of all adjacent intersections of ; and is the weight parameter; For intersection The second traffic state; The second aggregation state determination module determines a second aggregation state based on the first traffic state and the first aggregation state, including: aggregating the first traffic state and the first aggregation state to obtain a second aggregation state; The method for obtaining the second aggregation state is: in, A first aggregated state aggregated using a second traffic state of an adjacent intersection; is the first traffic state of the target intersection; A second aggregated state obtained by aggregating the first traffic state and the first aggregated state; For the graph attention network; The traffic signal change action determination module determines the traffic signal change action of the target intersection based on the second aggregation state, including: taking the second aggregation state as the input of the linear layer, obtaining the sum of future reward values ​​that can be obtained by taking a certain action at the target intersection under the second aggregation state; and adopting the action that maximizes the sum of the future reward values ​​as the traffic signal change action of the target intersection.

6. An electronic device, characterized in that: include: processor, memory and bus, wherein, The processor and the memory communicate with each other via the bus; The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 4 by calling the program instructions.

7. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 4.

Images