Exploration and Optimization Control Method for the Cooperative Structure of Traffic Lights Based on Causal Influence

Abstract

The present invention discloses a method for exploring and optimizing the control of a traffic light group collaboration structure based on causal influence, including initialization; converting the original observation of traffic light i into f(oi); making a decision based on the converted f(oi) and the behavior policy model πi to obtain the traffic light action ai, the local reward ri of the traffic light, and the next observation f(f(oi)); performing learning on traffic light i based on (f(oi), traffic light action ai, local reward ri of the traffic light, f(f(oi))); putting the observations of other traffic lights into the global observation sequence set Ui; calculating the causal influence relationship model Gi between traffic light i and other traffic lights in a period of time; appending it to the causal influence graph sample set Ωi of traffic light i; screening to obtain the optimal causal influence graph sample Gi* of traffic light i; clearing the global observation sequence set Ui of traffic light i. Through instant information interaction, the present invention enables traffic lights to have a predictive effect on the possible impacts on other traffic lights, and can take appropriate decisions in advance to relieve regional traffic congestion.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic light optimization control, and particularly relates to a method for exploring and optimizing the control of a traffic light group collaboration structure based on causal influence. Background Technique

[0002] With the continuous advancement of urbanization, the number of motor vehicles has risen rapidly, and the urban road congestion has become increasingly serious. To ensure the normal operation of vehicles on urban roads, the traffic signal control at road intersections is particularly important. The traffic signal control at intersections is mainly achieved through traffic signal controllers. Therefore, traffic signal controllers play an increasingly important role in daily life.

[0003] In the prior art, the control of traffic lights at intersections is usually carried out according to preset time, and the green light passing time for each direction at the intersection remains unchanged during operation once it is set. In this case, the traffic conditions at road intersections vary widely. When a certain direction is congested or multiple directions are congested, the traffic signal control at the road intersection still proceeds in a predetermined manner, not only failing to relieve the traffic conditions but often leading to more serious congestion.

[0004] Based on this, in the process of urban traffic scheduling, the present invention explores the causal influence relationship between traffic light groups, providing a theoretical guidance for the later optimization of traffic light control. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for exploring and optimizing the control of a traffic light group collaboration structure based on causal influence in view of the above deficiencies in the prior art, so as to solve the problem that the prior art lacks the exploration of the causal relationship between traffic light groups.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A method for exploring and optimizing the control of a traffic light group collaboration structure based on causal influence, which includes the following steps:

[0008] S1. Initialize the observation o of traffic light i i , the behavior strategy model π i , the causal influence graph model G i , the causal influence graph sample set Ω i and the global observation sequence set U i ;

[0009] S2. Based on the initial optimal causal influence graph sample G i * convert the original observation o of traffic light i i into the corrected observation f(o i );

[0010] S3. Make a decision based on the transformed f(o i ) and the behavioral policy model π i to obtain the traffic light action a i , the local reward r of the traffic light i and f(f(o i ));

[0011] S4. Learn traffic light i based on (f(o i ), the traffic light action a i , the local reward r of the traffic light i , f(f(o i )));

[0012] S5. Put the observations of other traffic lights into the global observation sequence set U i , where U i = U i ∪(O1, O2,..., O n );

[0013] S6. Use the global observation sequence set U i to calculate the causal influence relationship model G i between traffic lights in a period;

[0014] S7. Append to the causal influence diagram sample set Ω i , where is the average global reward of all traffic lights in this period, and is the average local reward of traffic light i in this period;

[0015] S8. Screen out the optimal causal influence diagram sample G i from the causal influence diagram sample set Ω i * , and replace the currently used causal influence diagram model with the optimal causal influence diagram sample G i * ;

[0016] S9. Clear the global observation sequence set U i , and return to step S2 until the iteration reaches the specified number of times.

[0017] Furthermore, the observation o i of traffic light i is:

[0018] A comprehensive evaluation and quantification of the congestion situation of the lanes under its jurisdiction; the comprehensive evaluation of its state is reflected by two parts: the phase factor and the lane occupancy factor.

[0019] Furthermore, the traffic light phase factor is:

[0020] One or several vehicle flows in each phase period within a signal cycle receive exactly the same traffic light color indication at the same time;

[0021] Lane occupancy factor: The lane occupancy factor focuses on lanes and calculates the final lane occupancy jam through a congestion value algorithm:

[0022] jam = α × move + β × halt

[0023] The state s of the traffic lights i is calculated as:

[0024] s i = (phase + duration) × jam

[0025] where move is the ratio of the number of vehicles in motion to the lane capacity on the approach lane of the intersection where the traffic lights are located at time t; halt is the ratio of the number of vehicles in a stalled state to the lane capacity on the approach lane of the intersection where the traffic lights are located at time t; α is the driving attenuation factor; β is the parking enhancement factor; phase is the traffic light phase number; and duration is the phase timing.

[0026] Furthermore, the setting of the traffic light action a includes Keep and Change. Among them, Keep is to keep the current phase unchanged until the next state update or reach the maximum flexible timing of phase timing reset; Change is to switch the current phase to the next phase.

[0027] Furthermore, the behavior policy model π i is a neural network based on DQN. The input of this neural network is the final state s of the traffic lights i , and the outputs of the neural network are the two actions of Keep and Change of the traffic lights, represented by 0 and 1 respectively.

[0028] Furthermore, the causal influence diagram model G i is:

[0029] G i [[ID=B1]]=(Causality(X1,X i ),Causality(X2,X i ),...,Causality(X N ,X i ))=(G 1,i ,G 2,i ,...,G N,i )

[0030] where i and j are any two traffic lights in the traffic light group, and i ≠ j, Xi and X j are the state sequence segments of i and j on the time segment (t, t + L), respectively. X i =(o i,t, o i,t+1 ,...o i,t+L ), X j =(o j,t, o j,t+1 ,...o j,t+L ), where any o i,t+k or o j,t+k (0 ≤ k ≤ L) is the observation of the set traffic lights; The setting of Causality(X i , X j ) is as follows. Let X i =(o i,t, o i,t+1 ,...o i,t+L ), X i ’={x i,t ’, x i,t+1 ’,...x i,L-2 ’}:

[0031]

[0032] The conversion of X i to X i ’ is used to determine whether X i is concentrated in a relatively concentrated exploration range;

[0033] Causality(X j , X i ) = dist(X j ’, X i ’)

[0034] where dist represents the edit distance between sequences.

[0035] Furthermore, step S2 specifically includes:

[0036] f(o i )={o1’, o2’,..., o n ’}, if G j,i =1, then o j ’ = o j ; if G j,i =0, then

[0037] Furthermore, in step S7, the local reward of the traffic lights is jointly determined by the congestion situation and the lane phase factor;

[0038] Among them, the lane phase factor: The influence of the phase on the traffic light reward is determined by the lane phase factor. The lane phase factor is the phase signal (G for green light, Y for yellow light, R for red light) on the lane at the intersection where the traffic light is located in the current phase. Denote the lane phase factor as p, and its value is as follows:

[0039]

[0040] Congestion situation: There are various combinations of lane connections under a traffic light. In each combination, there is an entrance lane and an exit lane.

[0041] Combining the two factors of the lane phase factor and the congestion situation, the local reward of each traffic light is calculated:

[0042]

[0043] Among them, {1, 2..., M} is the set of import lanes connected to the intersection where a certain traffic light is located;...

[0044] The global reward of the traffic light group is:

[0045]

[0046] Among them, r i (1 ≤ i ≤ N) is the local reward of the i-th traffic light.

[0047] Furthermore, the causal influence graph sample set Ω in step S7 i is:

[0048]

[0049] Among them, τ k (1 ≤ k ≤ L) is the time period (t (k-1)*L+1 , t (k-1)*L+2 ,..., t k*L ), L is the length of this time period, G i,τk is the causal influence graph of which traffic lights the i-th traffic light will be affected by other traffic lights in the time period τ k , is the average global reward in the time period τ k , is the average local reward of the i-th traffic light in the time period τ k .

[0050] Furthermore, the optimal causal influence graph sample G i * is:

[0051] G i *∈{G i,τ1 ,G i,τ2 ,...,G i,τk};

[0052] in:

[0053]

[0054] The traffic light group collaborative structure exploration and optimization control method based on causal influence provided by the present invention has the following beneficial effects:

[0055] This invention improves and explores the collaborative structure between traffic lights by analyzing the relationship between the causal influence relationship between multiple intelligent agents and the global reward, and improves the global reward by optimizing the collaborative structure. In the process of urban traffic scheduling, this paper explores the causal influence relationship of collaboration between traffic light groups and provides theoretical guidance for the subsequent optimization of traffic light control.

[0056] At the same time, the present invention enables the intelligent agent to foresee the upcoming traffic flow through real-time information interaction, and can take appropriate decisions in advance to alleviate traffic congestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of traffic light group reinforcement learning based on the causal influence graph model of the present invention.

[0058] Figure 2 This is the causal influence relationship between the traffic lights of the present invention.

[0059] Figure 3 This is the map and traffic light distribution of the present invention.

[0060] Figure 4 The distribution of congested intersections of each algorithm and the total congestion time of each intersection.

[0061] Figure 5 Comparison of congestion time distribution and congestion value of typical traffic lights of each algorithm. DETAILED DESCRIPTION

[0062] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0063] Example 1

[0064] The method for exploring and optimizing the control of the traffic light group collaboration structure based on causal influence in this embodiment aims to form a sufficient collaboration relationship among agents to obtain a better global reward, and further realize the exploration and optimization of the traffic light group collaboration structure. The specific steps are as follows:

[0065] Step S1: Initialize the observation o of traffic light i i , the behavior policy model π i , the causal influence graph model G i , the causal influence graph sample set Ω i and the global observation sequence set U i ;

[0066] In the following, each parameter initialized in this step will be described in detail in this embodiment:

[0067] The observation o of traffic light i i is:

[0068] A comprehensive evaluation and quantification of the congestion situation of the lanes under the jurisdiction of traffic light i. The comprehensive evaluation of its state is reflected in two parts: the phase factor and the lane occupancy factor.

[0069] Phase factor: The traffic light phase refers to one or several streams of traffic in each phase period within a signal cycle, and at the same time, they obtain exactly the same traffic light color indication. For example, the "green - yellow - red" time period is called a signal phase; the traffic light controls the red and green operation logic by setting a series of phase combinations and setting the operation timing. The phase and its timing of a certain traffic light are as follows in the table:

[0070]

[0071] According to the above table, two quantified data, the traffic light phase number phase and the phase timing duration, are designed to reflect the phase factor in the traffic light state.

[0072] Lane occupancy factor: The lane occupancy factor focuses on the lane, and a congestion value set is obtained through the congestion value algorithm.

[0073] Congestion value algorithm:

[0074] The congestion value is calculated based on the parking value and the driving value. The parking value halt is the ratio of the number of vehicles in a stagnant state on the import lane of the intersection where the traffic light is located at time t to the lane capacity. The driving value move is the ratio of the number of vehicles in a driving state on the import lane of the intersection where the traffic light is located at time t to the lane capacity.

[0075] In this embodiment, the average vehicle length is set to 5 meters, and the value obtained by dividing the length of each lane in the import lane by 5 is the current lane capacity.

[0076] Introduce a driving attenuation factor α (0 < α < 1) to reduce the enhancement effect of the driving value on the congestion value, and a parking enhancement factor β (0 < β < 3) to enhance the enhancement effect of the parking value on the congestion value.

[0077] The final lane occupancy ratio is denoted as jam: jam = α × move + β × halt

[0078] The final state setting of traffic light i is: s i = (phase + duration) × jam

[0079] The traffic light action a is:

[0080] The traffic light phase setting is selected through the timing cycle of the phase list, so the action setting will also focus on the phase. The action setting includes two types: Keep and Change; Keep means maintaining the current phase unchanged until the next state update or reaching the maximum flexible timing of the phase timing reset; Change means switching the current phase to the next phase.

[0081] The behavior policy model π i is:

[0082] The π in this embodiment i is a neural network based on DQN. The input of this neural network is the state setting of the final traffic light, and the output of the neural network is the two actions of Keep and Change of the traffic light, which are represented by 0 and 1 respectively.

[0083] Specifically, for the first layer of this neural network, construct a two-dimensional convolution to define the dimension space and define the stride of the convolution along the width and height directions; define the activation function Relu; for the second layer of the neural network, construct a two-dimensional convolution to define the dimension space and define the stride of the convolution along the width and height directions; define the activation function Relu; define the third layer of fully connected network.

[0084] The calculation process of the causal influence diagram model G i is:

[0085] Let i and j be any two traffic lights in the traffic light group, and i ≠ j, X i and X j are respectively the state sequence segments of i and j in the time segment (t, t + L), X i = (o i,t, o i,t+1 ,... o i,t+L ), X j = (o j,t, o j,t+1 ,... o j,t+L ), where any o i,t+k or oj,t+k (0 ≤ k ≤ L) is the observation of the set traffic lights.

[0086] G i = (Causality(X1, X i ), Causality(X2, X i ),..., Causality(X N , X i )) = (G 1,i , G 2,i ,..., G N,i ) is

[0087] The causal influence relationship between any X j and X i (i ≠ j). The greater Causality(X j , X i ) is, the greater the influence of X j on X j . The setting of Causality(X i , X j ) is as follows. Let X i = (o i,t, o i,t+1 ,... o i,t+L ), X i ’ = {x i,t ’, x i,t+1 ’,... x i,L-2 ’},

[0088]

[0089] The purpose of converting X i to X i ’ is to determine whether X i is concentrated in a relatively concentrated exploration range:

[0090] Causality(X j , X i ) = dist(X j ’, X i ’)

[0091] where dist represents the edit distance between sequences.

[0092] Reference Figure 2, which illustrates how to analyze the causal influence relationship between traffic lights. From the perspective of event-based causality, there is no clear causal relationship between traffic light 1 and traffic light 2. When traffic light 1 tends to be in a certain state, traffic light 2 may tend to be in the same state or another state, and vice versa. However, their changes are not random or correlated. If the variables are directly calculated, it will be found that their correlation coefficients are changing. If the symbol e represents a certain fixed exploration pattern within the neighborhood of a certain state of the traffic light, and the symbol * represents that the agent does not have a certain fixed exploration pattern within this time slice, it can be found that traffic light 1 and traffic light 2 follow the trend of (e, *, e) in three time slices, and traffic light 3 follows the trend of (*, *, *). In such an expression, it is easy to know that traffic light 1 and traffic light 2 influence each other, but the exploration process of traffic light 3 is not affected.

[0093] Step S2. Based on the influence diagram model G i Convert the original observation of traffic light i into the corrected f(o i ), specifically:

[0094] f(o i ) = {o1', o2',..., o n '};

[0095] If G j,i = 1, then o j ’ = o j ; if G j,i = 0, then

[0096] Step S3. Based on the converted f(o i ) and the behavior strategy model π i Make a decision to obtain the traffic light action a i , the traffic light local reward r i and f(f(o i ));

[0097] In this step, the traffic light action a i is the output obtained by inputting the local observation o i of traffic light i into the DQN-based policy network π i . The reward r i and f(f(o i )) are the reward values collected in the traffic simulation environment and the next observation obtained after taking the action a i ;

[0098] Step S4. Based on (f(o i ), the traffic light action a i , the traffic light local reward r i , f(f(oi )))Learn about traffic light i;

[0099] The learning in this step is the neural network training of the traffic light policy model based on DQN in step S3;

[0100] Step S5: Put the observations of other traffic lights into the global observation sequence set U i in, U i = U i ∪(O1, O2,..., O n );

[0101] Step S6: Use the global observation sequence set U i to calculate the causal influence relationship model G between traffic lights in a period of time i ;

[0102] Step S7: Append to the causal influence diagram sample set Ω i in, where is the average global return of all traffic lights in this period, is the average local return of traffic light i in this period;

[0103] Specifically, the local return of the traffic light is set to be determined by the congestion situation and the lane phase factor together.

[0104] Lane phase factor: The influence of the phase on the traffic light reward is determined by the lane phase factor, and the lane phase factor is the phase signal (G green light, Y yellow light, R red light) on the lane at the intersection where the traffic light is located in the current phase; Denote the lane phase factor as p, and its value standard is as follows:

[0105] [[ID=...]] [[ID=...]]

[0106] Congestion situation: There are various combinations of lane connections under a traffic light, and there is an entrance lane and an exit lane in each combination; Both lanes are intuitive manifestations of the congestion value.

[0107] Finally, by combining the two factors of the lane phase factor and the congestion situation, the local return of each traffic light can be calculated. Let {1, 2..., M} be the set of import lanes connected to the intersection where a certain traffic light is located, and the local return of traffic light i is:

[0108]

[0109] Global return of traffic light group is set as:

[0110] Note: Some of the tags in the original text seem to be incomplete or have placeholder-like ellipses in the middle of the text. I've translated the text as accurately as possible while maintaining the integrity of the tags. If there are specific instructions or corrections regarding those incomplete tags, it would be helpful for a more precise translation.

[0111] Among them, r i (1≤i≤N) is the local reward of the i-th traffic light;

[0112] Causal influence diagram sample set Ω i for:

[0113]

[0114] Among them, τ k (1≤k≤L) represents the time period (t (k-1)*L+1 ,t (k-1)*L+2 ,...,t k*L ), L is the length of the time period, G i,τk is the time period τ k Traffic light i i A causal influence diagram showing which other traffic lights will be affected. is the time period τ k The average global return in is the time period τ k The average local return of traffic light i.

[0115] Step S8: From the causal influence diagram sample set Ω i The optimal causal influence diagram sample G is obtained by screening i * And replace the currently used causal influence graph model with the optimal causal influence graph sample G i * ;

[0116] Optimal causal influence diagram sample G i * for:

[0117] G i * ∈{G i,τ1 ,G i,τ2 ,...,G i,τk}

[0118] in:

[0119]

[0120] Formula (2) gives how to get from {G i,τ1 ,G i,τ2 ,...,G i,τk} to select the optimal causal influence diagram model G i * , G i * is Ω i Can maximize Samples;

[0121] First, G i * Needs to be maximized That is, balance the global return and the local return. Second, G i * Needs to minimize Sim(G i,τi ). Equation (3) indicates that G i * is the one with the smallest average distance from other causal influence diagrams in {G i,τ1 , G i,τ2 ,..., G i,τk}. The more frequently a diagram similar to G appears in all time periods and can obtain a higher i * statistical significance compared to the diagram selected only based on Compared with selected diagrams.

[0122] Step S9: Clear the global observation sequence set U i , and then return to step S2 for iteration, and re - assign values to U i .

[0123] Embodiment 2

[0124] This embodiment is used to verify the algorithm in Embodiment 1, and specifically includes the following content:

[0125] Experimental analysis

[0126] In this experiment, SUMO is used as a traffic simulation platform to implement traffic simulation. SUMO is an open - source traffic simulation framework that can provide road network construction, traffic behavior, and collection of relevant data required in simulation experiments. Pychram is used as an IDE tool to write Python code to complete traffic light optimization and corresponding data collection, and relevant data operation libraries such as numpy and matplotlib are used to realize the visualization of experimental data.

[0127] Experimental data construction

[0128] To reflect the authenticity of the experiment, the comparative experiment will use the actually collected map of the Mianyang Yuanshan area and the road network vehicle operation trajectories collected within 1 hour on a certain day for simulation in the sumo map. [[ID=5i]]

[0129] Map data construction: Map data mainly includes lane data, intersection data, traffic light data, phase logic data, and lane association data. In the project cooperation with Mianyang traffic police, relevant personnel collected the above - mentioned data in the Mianyang Yuanshan area and constructed visual map data through sumo software for subsequent use.

[0130] Traffic flow data construction: Approximately 15,000 pieces of traffic flow general directions were obtained from the traffic flow photo data of each intersection in the Yuanshan area provided by the traffic police within one hour. Then, the missing parts were filled in according to the road network connectivity information of the Yuanshan map using the provided data to form traffic flow data suitable for simulation on SUMO.

[0131] Traffic evaluation indicators

[0132] In the actual road network, the most intuitive traffic intervention effect is reflected in the number of vehicles on each intersection and the lanes under the jurisdiction of traffic lights at different times, and then extended to the queue length at intersections. To reflect this effect, in this embodiment, the queue length at each intersection of each traffic light to be optimized is continuously collected during the road simulation process and their average values are calculated. The queue length data at intersections can only show the optimization effect in terms of intersections and cannot comprehensively reflect the optimization effect of the entire road network. Therefore, the total number of vehicles in the current road network needs to be collected at each moment, and the ratio of the average queue length at intersections to the total number of vehicles is used as an indicator to evaluate the comprehensive optimization effect of traffic lights on the entire road network. The queue length at intersections and the total congestion ratio index data are comprehensively used as traffic evaluation indicators.

[0133] The map file is constructed in SUMO and the operation logic of each traffic light phase is constructed according to the actual traffic light data in Yuanshan, Mianyang. The congestion-prone areas of the map are found through actual simulation, and the key nodes with the peak queue length during the peak period are selected as the optimization objects for traffic lights in this area for optimization and relevant traffic evaluation indicators are collected.

[0134] Reference Figure 3 , is a schematic diagram of the effect of a certain moment in the congestion-prone area of Yuanshan in the SUMO simulation. Among them, the intersections with light-colored nodes are the distribution of traffic lights in this area, and the dark-colored nodes are the actual optimization objects.

[0135] The roads reflect their different congestion degrees according to the depth of their colors. The relevant data of roads and intersection traffic lights at each moment are collected during the simulation process to provide implementation data for subsequent optimization evaluation.

[0136] Experimental comparison index 1: Distribution of congested intersections of each algorithm and total congestion time of each intersection

[0137] Reference Figure 4 , are respectively the comparison of the distribution of congested intersections and the congestion time data of each intersection for the original data, Q-Learning, DQN and the algorithm of the present invention:

[0138] The four bar charts shown in the above figure are the original data without optimization, the data optimized by the Q-Learning algorithm, the data optimized by the DQN algorithm, and the data optimized by the algorithm in this paper. The abscissa is the list of intersections with obvious congestion after optimization by each algorithm, and the ordinate is the total congestion time of specific traffic lights. The peak value of the total congestion time of the original data is more than 8000 seconds, and the total congestion times of traffic lights numbered 306, 32, 33, 361, and 630 are all at a high level.

[0139] For the data optimized by the Q-learnging algorithm compared with the original data, the peak value of the total congestion time is about 4000 seconds, and the peak optimization effect is improved by 50% compared with the original data. The reduction range of the total congestion time of traffic lights numbered 32, 361, and 630, which were in a high congestion state for a long time in the metadata, is very large in the data optimized by Q-Learning. It can be said that the complete optimization of these two traffic lights has been achieved.

[0140] However, traffic lights numbered 28, 35, and 37, whose total congestion times were at a low level in the metadata, have a large increase in the data optimized by Q-Learning. By comparing the above two types of data, it can be found that the Q-Learing algorithm, compared with the original data, while achieving an improvement in the comprehensive congestion relief effect, diverts the scheduling pressure of some traffic lights that were originally in a high congestion state to some traffic lights with relatively low scheduling pressure.

[0141] By observing the optimization results of the DQN optimization algorithm, it is found that most of its traffic lights are in a state of relatively low total congestion time. However, traffic lights numbered 306 and 309 are in a state of extremely high total congestion time. Among them, the total time of traffic light 306 exceeds the highest level of the original data by more than 2000 seconds. At the same time, traffic light 309, which was not in an obvious congestion state in the original data and the data of the Q-Learning algorithm, also reaches a total time of about 7000 seconds. From the above data, it can be concluded that the optimization effect of the DQN algorithm compared with the metadata is mainly manifested in transferring the scheduling pressure of some intersections with relatively frequent congestion to one or two traffic lights. The biggest drawback of this optimization is that it may cause the traffic relief in a small area of the traffic system to completely collapse. However, the overall optimization effect of the DQN algorithm on the total congestion time is still greatly improved compared with the original data, but the improvement compared with the Q-Learning algorithm is not large. The DQN algorithm's handling of the peak value of the total congestion time is far less than that of Q-Learning. Therefore, generally speaking, the optimization effect of the DQN algorithm is not as good as that of Q-Learning.

[0142] In the algorithm of the present invention, the peak value of the total congestion duration of traffic lights is about 400 seconds, which shows a significant improvement compared to the other three data. At the same time, the distribution of the total congestion duration data of each traffic light in the congestion state is relatively average, and there is no situation where some traffic lights have very good effects while some have very poor effects. Generally speaking, the optimization effect of the algorithm in this paper on the congestion state is much greater than the other three data.

[0143] Experimental comparison index 2: Comparison of the congestion time distribution and congestion values of typical traffic lights of each algorithm

[0144] In Experimental Index 1, the overall congestion time of traffic lights optimized by each algorithm was compared overall. Such a comparison can only show the macroscopic effect changes. However, in the actual operation of the road network, a slight change in a traffic light has a huge impact on the entire traffic system. Therefore, the optimization effect needs to be refined to the comparison of each second. As Figure 5 shown in the comparison of the congestion time distribution of typical traffic lights of each algorithm:

[0145] Figure 5 For each algorithm in, the traffic light with the median total congestion time was selected as the sampling object for comparison from the traffic lights with obvious congestion situations. In the above graph, the abscissa is the traffic operation time from 1 to 12000, and the ordinate is the information on whether the traffic light is congested at the current time, where 0 indicates non-congested and 1 indicates congested.

[0146] Observing the original data, it can be seen that from the 2000th moment to the 10000th moment is the peak congestion time of this traffic light, and the congestion time is continuous and not scattered. This means that once the original congestion data occurs, it is not easy to solve, and the specific degree of congestion will continue to increase over time, which greatly hinders the operation efficiency of the traffic road network.

[0147] Observing the Q-Learning data, it is found that its traffic light is in the peak congestion state from the 5000th to the 8000th moment, showing a huge improvement compared to the total congestion time of the original data. However, it also has the problem of continuous and non-scattered congestion time. However, since its duration is relatively low compared to the original data, its impact on the overall traffic operation efficiency is relatively small.

[0148] Observing the DQN data, it is found that the entire time distribution in the congested state is split into three segments. And the overall peak congestion period is from the 2000th to the 4000th moment, and the occurrence time of the congestion peak period has been greatly advanced. At the same time, observing the data, it is found that within most moments of the first, second, and third segments, the ordinate has both 0 values and 1 values. This problem occurs because the span of the abscissa is too large so that the details cannot be fully reflected. The actual situation is that 0 and 1 alternate repeatedly in several segments and the duration of each state is not long, resulting in data that should not overlap looking overlapping. From this point, it shows that the congestion time distribution of DQN is discontinuous. Compared with Q-Learning, it has a better effect and does not have the problem of continuous superposition of congestion levels.

[0149] Observing the data of the algorithm of the present invention, it is found that the total congestion duration of the data of the present invention has been greatly reduced compared with the other three algorithms, and the durations of the three congestion time periods are short and their distribution positions are also very scattered. At the same time, for each individual time period, its congestion state also changes alternately repeatedly. Therefore, the influence effect of the algorithm of the present invention on the superposition of congestion levels is also the lowest.

[0150] Although the specific implementation manners of the invention have been described in detail with reference to the drawings, it should not be construed as a limitation on the protection scope of this patent. Within the scope described in the claims, various modifications and deformations that can be made by those skilled in the art without creative efforts still fall within the protection scope of this patent.

Claims

1. A traffic light group collaborative structure exploration and optimization control method based on causal influence, characterized by: The following steps are involved: S1. Initialize the observation o of traffic light i i , behavioral strategy model π i , causal influence diagram model G i , causal influence diagram sample set Ω i and the global observation sequence set U i ; S2, based on the initial optimal causal influence diagram sample G i * The original observation o of traffic light i i Converted into the corrected observation f(o i ); S3, based on the conversion of f(o i ) and behavioral strategy model π i Make a decision and get the traffic light action a i 、Traffic light local report r i and f(f(o i )); S4, based on (f(o i ), traffic light action a i , traffic light local return r i ,f(f(o i )))Learn about traffic lights i; S5. Put the observations of other traffic lights into the global observation sequence set U i Middle,U i =U i ∪(o1,o2,...,o n ); S6, using the global observation sequence set U i Calculate the causal influence relationship model G between traffic lights in a time period i ; S7, will Append to the causal influence diagram sample set Ω i Among them, is the average global return of all traffic lights in this time period, is the average local return of traffic light i during this period; S8. From the causal influence diagram sample set Ω i The optimal causal influence diagram sample G is obtained by screening i * And replace the currently used causal influence graph model with the optimal causal influence graph sample G i * ; S9. Clear the global observation sequence set U i , and return to step S2 until the specified number of iterations is reached.

2. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 1 is characterized in that: The observation o of the traffic light i i for: Comprehensive evaluation and quantification of the congestion situation of the lanes under jurisdiction; the comprehensive evaluation of its status is reflected by two parts: phase factor and lane share factor.

3. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 2 is characterized in that: The traffic light phase factor is: In each phase of a signal cycle, one or more traffic flows receive exactly the same traffic light color indication at the same time. Lane occupation factor: The lane occupation factor takes the lane as the focus and calculates the final lane occupation jam through the congestion value algorithm: jam=α×move+β×halt Traffic light status i Calculated as: s i =(phase+duration)×jam Among them, move is the ratio of the number of vehicles in a moving state in the entrance lane of the intersection where the traffic light is located at time t to the lane capacity; halt is the ratio of the number of vehicles in a stationary state in the entrance lane of the intersection where the traffic light is located at time t to the lane capacity; α is the driving reduction factor; β is the parking enhancement factor; phase is the traffic light phase number; duration is the phase timing; the traffic light phase number and phase timing are quantitative data.

4. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 3 is characterized by: The settings of the traffic light action a include Keep and Change, wherein Keep means keeping the current phase unchanged until the next state update or reaching the maximum elastic timing of the phase timing reset; Change means switching the current phase to the next phase.

5. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 4 is characterized by: The behavior strategy model π i It is a DQN-based neural network whose input is the final state s of the traffic light. i ,The output of the neural network is the two actions of the traffic light, Keep and Change, which are represented by 0 and 1 respectively.

6. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 5 is characterized in that: The causal influence graph model G i for: G i =(Causality(X1,X i ),Causality(X2,X i ),...,Causality(X N ,X i ))=(G 1,i ,G 2,i ,...,G N,i ) Where i and j are any two traffic lights in the traffic light group, and i≠j, X i and X j are the state sequence segments of i and j at the time segment (t, t+L), X i =(o i,t ,o i,t+1 ,...o i,t+L ),X j =(o j,t ,o j,t+1 ,...o j,t+L ), where any o i,t+k or j,t+k is the observation of the set traffic light, 0≤k≤L; Causality(X i ,X j ) is set as follows, let X i =(o i,t ,o i,t+1 ,...o i,t+L ),X i '={x i,t ',x i,t+1 ',...x i,L-2 '}: like Then x i,t '=e; if Then x i,t '=*; X i Convert to X i 'Used to judge X i Whether it is concentrated in a relatively concentrated exploration range; Causality(X j ,X i )=dist(X j ’,X i ’) Here, dist represents the edit distance between sequences.

7. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 6 is characterized in that: The step S2 specifically includes: f(o i )={o1',o2',...,o n '}, if G j,i =1, then o j '=o j ; If G j,i =0, then 1≤j≤N, j≠i.

8. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 7 is characterized in that: The average local return of traffic light i in step S7 Determined by congestion conditions and lane phase factors; Among them, lane phase factor: The effect of phase on traffic light reward is determined by the lane phase factor. The lane phase factor is the phase signal of the lane at the intersection where the traffic light is located at the current phase. The lane phase factor is denoted as p, and its value is: Congestion: There are multiple lane combinations at a traffic light, and each combination has There is an entry lane and an exit lane; The average local return of each traffic light is calculated by combining the lane phase factor and congestion conditions: Among them, {1,2...,M} is the set of import lanes connected to the intersection where a traffic light is located; Among them, r i is the local reward of the i-th traffic light, 1≤i≤N.

9. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 8 is characterized in that: The causal influence diagram sample set Ω in step S7 i for: Among them, τ k is the time period (t (k-1)*L+1 ,t (k-1)*L+2 ,...,t k*L ), 1≤k≤L, L is the length of the time period, G i,τk is the time period τ k The causal influence diagram in which traffic light i is affected by other traffic lights, is the time period τ k The average global return of all traffic lights in , r i,τk is the time period τ k The average local return of traffic light i.

10. The traffic light group collaborative structure exploration and optimization control method based on causal influence according to claim 9 is characterized in that: The optimal causal influence diagram sample G in step S8 i * for: G i * ∈{G i,τ1 ,G i,τ2 ,...,G i,τk }; in:

Images