Traffic signal coordination method and system based on deep reinforcement learning and situation prediction

Abstract

The present invention provides a traffic signal coordination method and system based on deep reinforcement learning and situation prediction, which belongs to the field of smart transportation. The system includes: a simulation interaction environment module, a multi-intersection signal coordination optimization model integrating situation prediction, and a multi-task training module based on uncertainty weighting; wherein, the simulation interaction environment module is used to configure the traffic system simulation software SUMO according to the actual topological structure of the road network and the corresponding vehicle routing data, and build a simulation interaction environment similar to the actual traffic scene; the multi-intersection signal coordination optimization model integrating situation prediction is used to achieve effective decision-making based on current and future traffic conditions; the multi-task training module based on uncertainty weighting is used to measure the importance of the prediction task integrating uncertainty and the multi-intersection signal light control task by introducing uncertainty, thereby dynamically adjusting the weight of each task in the overall loss function.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation, and specifically relates to a traffic signal collaboration method and system based on deep reinforcement learning and situation prediction. Background Art

[0002] As cities expand, urban transportation plays an increasingly important role in people's lives, travel, and work efficiency. Urban traffic generally uses traffic lights at intersections to scientifically allocate the right of way for vehicles and pedestrians, ensuring safe and orderly urban traffic, improving road use efficiency, and improving traffic conditions. Typically, traffic light change times are pre-set by traffic light equipment and adjusted appropriately based on road traffic volume and pedestrian demand. These lights typically include motor vehicle lights, lane lights, and pedestrian crossing lights, each using different light combinations and indicating directions to collaboratively guide vehicles and pedestrians through safe and orderly traffic.

[0003] In the existing technology, most traffic light control methods tend to only focus on the current traffic status of the intersection and give the green light phase. However, due to the spatiotemporal dynamics of the traffic state, the signal control decisions made based on the current environmental status information cannot be accurately and efficiently applied to future traffic conditions. Therefore, the current signal control methods cannot effectively solve the problem of frequent phase switching caused by drastic traffic flow fluctuations, thereby increasing traffic safety hazards, from a long-term perspective. For example, Chinese patent application number 202310489368.8 discloses a traffic light control method based on transfer reinforcement learning. This method applies intelligent traffic control technology, uses a model-based pre-training algorithm and a graph convolutional network, and combines a multi-head attention mechanism to acquire intersection feature information and determine its impact. This method better reflects traffic flow patterns, accelerates the training process for new intersections using pre-training strategies, selects strategies suitable for the current environment, improves data efficiency, and reduces training costs. Chinese patent application number 202310385798.5 provides an adaptive traffic signal control system and method for traffic flow prediction. The control system includes a traffic state perception module, a data pre-processing module, a traffic feature extraction module, a traffic flow prediction module, and an adaptive signal control and communication module. However, none of the above methods consider the impact of the uncertainty of future traffic states at signalized intersections on signal control, which may lead to inadaptability and inefficiency in traffic management, increase traffic safety risks, and reduce the driver's travel experience. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present invention aims to provide a traffic signal coordination method and system based on deep reinforcement learning and situation prediction, emphasizing the important impact of future traffic situations on control, integrating situation prediction into the process of deep reinforcement learning, and constructing a multi-intersection signal coordinated control model, while achieving reliable prediction and effective decision-making, and improving the accuracy and effectiveness of signal coordination effects.

[0005] In order to achieve the above objectives, the embodiments of the present invention adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a traffic signal coordination system based on deep reinforcement learning and situation prediction, the system comprising: a simulation interactive environment module, a multi-intersection signal coordination optimization model integrating situation prediction, and a multi-task training module based on uncertainty weighting; wherein,

[0007] The simulation interaction environment module is used to configure the traffic system simulation software SUMO according to the actual topology of the road network and the corresponding vehicle routing data, and build a simulation interaction environment similar to the actual traffic scene;

[0008] The multi-intersection signal collaborative optimization model integrating situation prediction is used to achieve effective decision-making based on current and future traffic conditions;

[0009] The uncertainty-weighted multi-task training module is used to measure the importance of the uncertainty-integrated prediction task and the multi-intersection signal light control task by introducing uncertainty, thereby dynamically adjusting the weight of each task in the overall loss function.

[0010] Preferably, the simulation interactive environment module is further used to:

[0011] Using the Traci interface and sumolib tool of the traffic system simulation software SUMO, traffic operation information of vehicles on the road network and traffic intersections controlled by traffic lights are obtained during the simulation interaction process. At the same time, the signal control plan output by the multi-intersection signal collaborative optimization model that integrates situation prediction is sent to the traffic lights through the Traci interface to adjust the traffic phase. The operation of the actual traffic environment is simulated through SUMO, and the traffic status information of each signal intersection in the road network is recorded. This information is fed back to the multi-intersection signal collaborative optimization model that integrates situation prediction for calculation of the next round of control plans.

[0012] The traffic phase is composed of one or more traffic turns that obtain the right of way at the same time without causing traffic conflicts.

[0013] Preferably, the multi-intersection signal collaborative optimization model integrating situation prediction includes a multi-intersection complex traffic dynamic graph representation module, a fine-grained traffic flow prediction module integrating uncertainty, and a multi-intersection traffic signal phase collaborative selection module; wherein,

[0014] The multi-intersection complex traffic dynamic graph representation module is composed of a first multi-layer perceptron and a graph attention mechanism submodule, which is used to realize information exchange of multi-intersection traffic status through the adjacency relationship between intersections and extract the status features of adjacent intersections;

[0015] The uncertainty-integrated fine-grained traffic flow prediction module includes an instance normalization layer, an encoder, a convolutional layer, a summation operator, and a second multi-layer perceptron, and is used to achieve lane-level situational awareness by predicting future traffic flow on the road, thereby assisting the control decision-making process of traffic lights;

[0016] The multi-intersection traffic light phase collaborative selection module is composed of a third multi-layer perceptron, which is used to receive complex dynamic graph representations of multi-intersection traffic; it is also used to integrate the uncertainty factors of future traffic flow changes through the prediction data provided by the fine-grained traffic flow prediction module that integrates uncertainty, so as to further improve the accuracy and robustness of decision-making.

[0017] Preferably, the first multi-layer perceptron MLP(·) is used to represent the state of each intersection Perform feature extraction to obtain the state characteristics of each intersection

[0018] The graph attention mechanism submodule is used to use the state feature H′ t and the intersection adjacency matrix A as input, aggregate the state features of each adjacent intersection, and thus obtain the intersection feature representation that integrates the states of adjacent intersections

[0019] Preferably, the input of the multi-intersection complex traffic dynamic graph representation module is the N intersection states at time t and intersection adjacency matrix The intersection status of each intersection Including the current signal phase p of the intersection i and the number of vehicles on the import lane v i , expressed as The adjacency matrix consists of a 0-1 matrix Indicates that when intersection i and intersection j are adjacent, a ij =1, otherwise a ij =0.

[0020] Preferably, the input data of the fine-grained traffic flow prediction module integrating uncertainty is the historical traffic flow sequence of a certain intersection. The specific execution operations are as follows:

[0021] The input historical traffic flow data is instance-normalized and fed into an encoder for traffic flow prediction. The encoder consists of a traffic flow prediction model that uses two-dimensional convolution operations to predict the changing trends of future multi-step traffic flow. The time dimension is then compressed using a sum operator to obtain the range of changes in the new traffic flow in the import lanes within a predetermined time period in the future, thereby obtaining the predicted results of future traffic flow. Finally, a third multi-layer perceptron is used to extract features from the predicted results.

[0022] Preferably, in the workflow of the multi-intersection traffic signal phase collaborative selection module, the outputs of the multi-intersection complex traffic dynamic graph representation module and the fine-grained traffic flow prediction module with fusion of uncertainty are first directly spliced; then, the spliced ​​feature vector is sent to the third multi-layer perceptron, and the input data is nonlinearly transformed and processed to better understand the complex traffic situation; the value of each action in the action set is calculated by the third multi-layer perceptron, so as to evaluate the pros and cons of each possible phase.

[0023] Preferably, the multi-intersection traffic light phase coordinated selection module adopts an ε-greedy algorithm, randomly selecting an action from the action set with a probability of ε, and selecting the action with the greatest value with a probability of 1-ε;

[0024] The specific process of selecting an action at a single intersection is as follows:

[0025]

[0026] In formula (5), represents the action set; Q represents the action value function, which contains the value of each action a in the action set; s is the current signal phase p i and the number of vehicles on the import lane v i represents the state of the intersection; β represents a number randomly sampled from the interval [0, 1]; ε represents a set fixed probability value, and its value is in the interval [0, 1].

[0027] Preferably, the prediction task integrating uncertainty adopts the quantile loss function as follows:

[0028]

[0029] In formula (6), θ represents the set of learnable parameters of the model, and the model is adjusted through the training process to better fit the data; f(x) represents the confidence interval of the prediction model output, y represents the true observation value; ρ represents the confidence level, which is given in the form of quantiles (ρ1, ρ2, ρ3);

[0030] The multi-intersection signal light control task adopts the following mean square error loss function:

[0031]

[0032] In formula (7), N represents the total number of sample data groups, and each data group {(s, a, r, s′)} consists of state s, action a, reward r, and next state s′; Q ω (s i , a i ) represents the action-value function, represents the temporal difference objective function, which means that the goal of temporal difference learning is to incrementally update the action-value function; in the temporal difference objective function, r represents the reward at the current moment, γ represents the discount factor, s′ represents the next state, and a′ represents the action that can obtain the maximum action value in the next state.

[0033] In a second aspect, an embodiment of the present invention further provides a traffic signal coordination method based on deep reinforcement learning and situation prediction. The method is implemented based on the above-described system and specifically includes:

[0034] The traffic system simulation software SUMO is configured through the simulation interaction environment module according to the actual topological structure of the road network and the corresponding vehicle routing data to build a simulation interaction environment similar to the actual traffic scene; in the built simulation interaction environment, the traffic operation information of the vehicles on the road network and the traffic intersections controlled by the signal lights are obtained by using the Traci interface and sumolib tool of the traffic system simulation software SUMO during the simulation interaction process, and the signal control scheme output by the multi-intersection signal collaborative optimization model of the fusion situation prediction is sent to the traffic light through the Traci interface, so as to adjust the traffic phase; the operation of the actual traffic environment is simulated by SUMO, and the traffic status information of each signal intersection in the road network is recorded at the same time, and this information is fed back to the multi-intersection signal collaborative optimization model of the fusion situation prediction for calculation of the next round of control scheme;

[0035] The multi-intersection signal collaborative optimization model that integrates situation prediction is based on current and future traffic conditions to achieve effective decision-making based on situation prediction;

[0036] The uncertainty-weighted multi-task training module introduces uncertainty to measure the importance of the prediction task and the multi-intersection signal light control task, thereby dynamically adjusting the weight of each task in the overall loss function. The more stable the task description, the higher the quality and the greater the contribution to the overall loss. Through the uncertainty weighting mechanism, the training process of the multi-intersection signal collaborative optimization model for fusion situation prediction is optimized to improve the robustness and generalization ability of the model.

[0037] The technical solution provided by the embodiment of the present invention has the following beneficial effects:

[0038] The embodiment of the present invention provides a traffic signal coordination method and system based on deep reinforcement learning and situation prediction, which provides reliable data support and decision support for the coordinated control of multi-intersection signals by predicting the range of changes in future traffic situations in advance. The method first uses the open source traffic system simulation software SUMO to build a simulated interactive traffic environment to realize the control of vehicles at signal intersections; then constructs a multi-intersection signal coordination optimization model that integrates situation prediction, and realizes effective decision-making based on the current and future traffic conditions; collects traffic status data during the interaction between the model and the simulation environment, and after pre-processing the collected data, constructs training samples and divides the data set; introduces uncertainty to measure the importance of prediction tasks and decision tasks, and uses a multi-task adaptive weighted loss function to train the model, while achieving reliable prediction and effective decision-making. The present invention defines a new signal light control method that integrates traffic uncertainty prediction, that is, predicting the range of future traffic fluctuations in advance to achieve reliable prediction, providing data support and important basis for effective control of signal lights to complete traffic diversion, which has broad application prospects and value in the field of traffic signal control.

[0039] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a schematic diagram of the structure of a traffic signal coordination system based on deep reinforcement learning and situation prediction according to an embodiment of the present invention;

[0042] Figure 2 2. Schematic diagram of a standard intersection topology controlled by a signal light according to an embodiment of the present invention;

[0043] Figure 3 1 is a schematic diagram of eight phases in a road traffic signal control system according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of a module for representing a complex traffic dynamic graph at multiple intersections according to an embodiment of the present invention;

[0045] Figure 5 Schematic diagram of a fine-grained traffic flow prediction module integrating uncertainty in an embodiment of the present invention;

[0046] Figure 6 Schematic diagram of a multi-intersection traffic light phase coordinated selection module in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. It should be noted that the embodiments of the present invention and the features in the embodiments can also be combined with each other in the absence of conflict.

[0048] It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures. In the description of the present invention, the terms "first," "second," "third," "fourth," etc. are used only to distinguish the description and are not to be understood as indicating or implying relative importance.

[0049] In response to the problem of coordinated control of signals at urban traffic intersections in the prior art, an embodiment of the present invention provides a traffic signal coordination method and system based on deep reinforcement learning and situation prediction. First, open source traffic system simulation software is used to build a simulated and interactive traffic environment to realize the control of traffic flow at signal intersections; then a multi-intersection signal coordinated optimization model integrating situation prediction is constructed. The model realizes effective decision-making integrating situation prediction based on current and future traffic conditions; then, traffic status data is collected during the interaction between the model and the simulation environment, and after pre-processing the collected data, training samples are constructed and data sets are divided; finally, uncertainty is introduced to measure the importance of prediction tasks and decision-making tasks, and a multi-task adaptive weighted loss function is used to train the model, while achieving reliable prediction and effective decision-making, thereby providing data support and decision-making support for the coordinated control of multi-intersection signals by predicting future traffic situations in advance.

[0050] like Figure 1As shown, the traffic signal coordination system based on deep reinforcement learning and situation prediction described in the embodiment of the present invention includes: a simulation interactive environment module, a multi-intersection signal coordination optimization model integrating situation prediction, and a multi-task training module based on uncertainty weighting.

[0051] The simulation interaction environment module is used to configure the traffic system simulation software SUMO according to the actual topological structure of the road network and the corresponding vehicle routing data, and to build a simulation interaction environment similar to the actual traffic scene.

[0052] In the constructed interactive simulation environment, the Traci interface and sumolib tools of the traffic system simulation software SUMO are used to accurately obtain traffic operation information of vehicles on the road network and traffic intersections controlled by traffic lights during the simulation. The model also outputs signal control plans to the traffic lights through the Traci interface, thereby adjusting the traffic phase. Subsequently, SUMO simulates the actual traffic environment and records traffic status information at each signalized intersection in the road network. This information is fed back to the model for the calculation of the next round of control plans. This process ensures the accuracy and practicality of the simulation environment, providing reliable data support for traffic management and optimization.

[0053] The traffic intersection controlled by the signal light includes an entrance section and an exit section. Figure 2 As shown in the figure, it consists of four entrance sections and four exit sections. Each section has three lanes: left turn, straight ahead, and right turn, for a total of 12 entrance lanes and 12 exit lanes. The right of way for vehicles on the lanes is controlled by traffic lights: a green light allows passage, and a red light prohibits passage.

[0054] In the phase, one or more traffic turns that simultaneously obtain the right of way without causing traffic conflicts constitute a phase p. Figure 3 As shown, the phase set P = {p1, p2, p P ,p b ,p c ,p d ,p e ,p f}.

[0055] The multi-intersection signal coordination optimization model, which integrates situation prediction, includes a module for representing complex traffic dynamic graphs at multiple intersections, a module for fine-grained traffic flow prediction that incorporates uncertainty, and a module for coordinated selection of traffic signal phases at multiple intersections. Each module of the model is described in detail below.

[0056] (1) Multi-intersection complex traffic dynamic graph representation module.

[0057] like Figure 4As shown, the multi-intersection complex traffic dynamic graph representation module consists of a first multi-layer perceptron and a graph attention mechanism sub-module, which is used to realize information interaction of multi-intersection traffic status through the adjacency relationship between intersections and effectively extract the status features of adjacent intersections.

[0058] The input of the multi-intersection complex traffic dynamic graph representation module is the N intersection states at time t and intersection adjacency matrix Among them, the intersection status of each intersection Including the current signal phase p of the intersection i and the number of vehicles on the import lane v i , expressed as Among them, || represents the serial operation; p i Indicates the current signal phase of the intersection, such as Figure 2 As shown, excluding the four right-turn lanes, the remaining eight import lanes are represented by a 0-1 vector of length 8, where 0 represents a red light and 1 represents a green light, and the phase is represented by p1 = (0, 1, 0, 0, 0, 1, 0, 0); v i The number of vehicles on the import lane is the number of vehicles on the import lane at the current moment, including queued vehicles and running vehicles (speed greater than 1m / s), specifically represented by a vector of length 12.

[0059] In addition, the new traffic flow in the import lanes is collected from the newly added traffic flow on the 12 import lanes at intersection i within the specified time slice τ, which is represented by a vector X with a length of 12 i The length of the vehicle queue on the import lane is the total number of vehicles with a speed less than 1m / s on the import lane, represented by a vector q of length 12 i Reward refers to the feedback information after executing an action in the traffic simulation software SUMO, which is used to evaluate the quality of the action. It is set as the inverse of the length of the vehicle queue on the import lane, expressed as r i =-q i .

[0060] The adjacency matrix consists of a 0-1 matrix Indicates that when intersection i and intersection j are adjacent, a ij =1, otherwise a ij = 0. The set of one-hop or multi-hop traffic intersections directly adjacent to intersection i is called the adjacent intersection set of intersection i. The set contains the index numbers corresponding to the adjacent intersections, which are expressed as

[0061] In order to obtain the status of each intersection The multi-intersection complex traffic dynamic graph representation module uses the first multi-layer perceptron MLP(·) to extract features and obtain the state features of each intersection. Complete the state characteristics H′ of each intersection t After extraction, use the state feature H′ t and the intersection adjacency matrix A as input, the graph attention mechanism is used in the graph attention mechanism submodule to aggregate the state features of each adjacent intersection, so as to obtain the intersection feature representation that integrates the states of adjacent intersections

[0062] The graph attention mechanism submodule first uses a single fully connected layer to obtain the similarity representation between the state features of each intersection:

[0063] e ij =LeakyReLU(ω T [Wh i ||Wh j ]) (1)

[0064] In formula (1), LeakyReLU(·) is a nonlinear activation function; || represents a series operation; · T Indicates transpose operation; h i and h j represents the intersection features after the adjacent intersection states of intersection i and intersection j are fused; ω and W represent the parameters of the fully connected layer, which are randomly initialized.

[0065] The Softmax function is further used to normalize the similarity to obtain the weight coefficient:

[0066]

[0067] In formula (2), exp(·) represents the exponential operation; represents the index of the neighboring intersection of intersection i; e ij Represents the similarity between the state features of intersection i and intersection j.

[0068] Use the weight coefficient to perform weighted summation on each adjacent intersection to obtain the eigenvector of intersection i:

[0069]

[0070] In formula (3), σ(·) represents the sigmoid activation function; W represents the parameters of the fully connected layer, which is randomly initialized; h j represents the intersection features after the adjacent intersection states of intersection j are fused; α ij Represents the normalized weight coefficient of the similarity between intersections i and j.

[0071] Use K groups of graph attention mechanism parameters to extract information of different feature dimensions:

[0072]

[0073] In formula (4), || represents the serial operation; σ(·) represents the sigmoid activation function; W (k) represents the parameters of the kth group of fully connected layers, which are randomly initialized; h j Represents the intersection characteristics after the adjacent intersection states of intersection j are fused; Represents the normalized weight coefficient of the similarity between intersections i and j in the kth group.

[0074] (2) A fine-grained traffic flow prediction module that integrates uncertainty.

[0075] like Figure 5 As shown in the figure, the uncertainty-integrated fine-grained traffic flow prediction module is a key component and includes an instance normalization layer, an encoder, a convolutional layer, a summation operator, and a second multi-layer perceptron. This module is used to achieve lane-level situational awareness by predicting future traffic flow on the road, effectively assisting in the traffic signal control decision-making process.

[0076] The input data of the fine-grained traffic flow prediction module integrating uncertainty is the historical traffic flow sequence of a certain intersection. First, the input historical traffic flow data undergoes instance normalization to better adapt to the data distribution at different intersections and time periods. The normalized data is then fed into an encoder for traffic flow prediction. The encoder, which can be composed of various existing traffic flow prediction models, utilizes two-dimensional convolution operations to predict multi-step future traffic flow trends. Next, the time dimension is compressed using a summation operator to determine the range of changes in new traffic flow in the import lanes within a predetermined timeframe. This yields a forecast for future traffic flow. Finally, a third multi-layer perceptron is used to extract features from the predicted results.

[0077] The uncertainty-integrated fine-grained traffic flow prediction module uses quantile regression to output confidence interval estimates. By using these estimates as input to the multi-intersection traffic signal phase coordination selection module, the module can understand the range of possible traffic flow variations, enabling a better understanding of traffic conditions and corresponding adjustments to signal timing, thereby achieving more intelligent and efficient traffic management.

[0078] For example, in Figure 2 The historical traffic flow sequence of each intersection in the input road network under the 12 import lanes environment shown where X i =(x1, x2, ..., x 12 ), which represents the traffic flow of each of the 12 import lanes in a specified time slice. First, after instance normalization, we get In order to better adapt to the data distribution of different intersections and different time periods. Then, the normalized sequence Predict future multi-step traffic through existing traffic prediction models and two-dimensional convolution operations Then, the time dimension is compressed by the sum operator and the feature extraction of the multi-layer perceptron is performed to obtain the change range of the new traffic flow in the import lane within a certain period of time in the future.

[0079] (3) Multi-intersection traffic light phase coordinated selection module.

[0080] like Figure 6 As shown, the Multi-Intersection Traffic Signal Phase Coordination Selection Module is composed of a third multi-layer perceptron to effectively determine the optimal traffic signal phase. First, the Multi-Intersection Traffic Signal Phase Coordination Selection Module receives a complex dynamic graph representation of traffic at multiple intersections, which helps the module understand the traffic flow between each intersection. Furthermore, the Multi-Intersection Traffic Signal Phase Coordination Selection Module incorporates the uncertainty of future traffic flow changes by integrating prediction data provided by the Uncertainty-Integrated Fine-Grained Traffic Flow Prediction Module, further improving the accuracy and robustness of its decisions.

[0081] In the workflow of the multi-intersection traffic signal phase collaborative selection module, the outputs of the multi-intersection complex traffic dynamic graph representation module and the uncertainty-integrated fine-grained traffic flow prediction module are first directly concatenated. This process is similar to fusing two data sources to obtain more comprehensive information. The concatenated feature vector is then fed into a third multilayer perceptron, which performs nonlinear transformations and processing on the input data to better understand complex traffic conditions. The third multilayer perceptron calculates the value of each action in the action set, thereby evaluating the pros and cons of each possible phase.

[0082] Finally, in order to determine the final action to be performed, the multi-intersection traffic light phase collaborative selection module adopts the ε-greedy algorithm, which randomly selects an action from the action set with a probability of ε and selects the action with the highest value with a probability of 1-ε. The action output by the model is the next passable phase of the signal intersection, and the action set is expressed as Typically, in practice, the value of ε decreases with the number of interaction steps. During the model initialization phase, it is not possible to accurately judge the quality of actions under different environmental conditions, so continuous exploration is performed in the initial phase. As the number of training times increases, the multi-intersection traffic light phase coordinated selection module has learned a better evaluation criterion, and the number of random explorations will be reduced, gradually reducing the value of ε. The specific process of selecting an action at a single intersection is as follows:

[0083]

[0084] In formula (5), represents the action set; Q represents the action value function, which contains the value of each action a in the action set; s is the current signal phase p i and the number of vehicles on the import lane v i represents the state of the intersection; β represents a number randomly sampled from the interval [0, 1]; ε represents a set fixed probability value, and its value is in the interval [0, 1].

[0085] The ε-greedy algorithm strikes a balance between exploration and exploitation, ensuring that the system can both explore new possibilities as much as possible when selecting actions and make reasonable decisions based on past experience. Therefore, the final execution action output by the ε-greedy algorithm can achieve good performance and stability in practical applications.

[0086] For example, in Figure 2 Under the 12 import lanes shown in the figure, we can get the complex traffic dynamic graph representation H of multiple intersections and the fine-grained traffic flow prediction representation with fusion uncertainty. After that, the two parts are concatenated to obtain Then use the multi-layer perceptron to get the value of each action in the action set Finally, the action to be executed is determined by the ε-greedy algorithm. Specifically, the action set is selected with a probability of ε. Randomly select an action from V, and with a probability of 1-ε, choose the action with the largest value in V. The action selection process is as follows:

[0087]

[0088] After the final action is selected, the action, that is, the signal control plan output by the model, is sent to the traffic light through the Traci interface provided by the simulation software SUMO, thereby adjusting the traffic phase.

[0089] Finally, the uncertainty-weighted multi-task training module introduces uncertainty to measure the importance of the uncertainty-integrated prediction task and the multi-intersection traffic light control task, thereby dynamically adjusting the weight of each task in the overall loss function. Uncertainty weighting considers the stability of the task; more stable tasks are considered to be of higher quality and should contribute more to the overall loss. This uncertainty-weighted mechanism optimizes the model training process and improves the model's robustness and generalization capabilities.

[0090] The prediction task of integrating uncertainty adopts the quantile loss function as follows:

[0091]

[0092] In formula (6), θ represents the set of learnable parameters of the model, which is used to adjust the model to better fit the data through the training process; f(x) represents the confidence interval of the prediction model output, y represents the true observation value; ρ represents the confidence level, which is given in the form of quantiles (ρ1, ρ2, ρ3), for example, ρ1 = 0.025 corresponds to the lower bound, ρ2 = 0.5 corresponds to the point estimate, and ρ3 = 0.975 corresponds to the upper bound.

[0093] The prediction model is trained using a quantile loss function, with the ultimate goal of ensuring that the upper and lower bounds of the prediction, as well as the point estimate, are as close as possible to the true value, resulting in more accurate and tight prediction intervals. In the traffic light control process, accurate prediction intervals can help the system better understand the variability and uncertainty of traffic flow, allowing for more precise adjustments to signal timing for smoother and safer traffic flow.

[0094] The multi-intersection signal light control task adopts the following mean square error loss function:

[0095]

[0096] In formula (7), N represents the total number of sample data groups, and each data group {(s, a, r, s′)} consists of state s, action a, reward r, and next state s′; Q ω (s i , a i ) represents the action-value function, It represents the temporal difference objective function, which means that the goal of temporal difference learning is to incrementally update the action value function; in the temporal difference objective function, r represents the reward at the current moment, γ represents the discount factor, s′ represents the next state, and a′ represents the action that can obtain the maximum action value in the next state. By making the action value function Q ω (s i , a i ) evaluates the temporal difference objective, gradually improving the accuracy of the action-value function and its ability to predict the environment. This incremental updating of the action-value function enables the agent to better adapt to environmental changes and make its decision-making more reliable and effective.

[0097] Finally, the entire model is trained through the multi-task loss function. The specific calculation process is as follows:

[0098]

[0099] In formula (8), σ1 and σ2 are the observation noise parameters of the model. In the process of multi-task optimization objectives, σ1 and σ2 play an important role and represent the degree of noise in the model output. In practical applications, observation noise is usually caused by various factors, such as sensor errors, interference during data acquisition, etc. During the optimization process, σ1 and σ2 will adaptively learn the loss function based on the training data. and This means that when the model's output for a task is affected by more noise, the weight of the loss function corresponding to that task will decrease to reduce the impact of noise on the overall optimization objective. Conversely, when the noise is smaller, the weight of the task will increase to better reflect the importance of the task in the overall optimization process.

[0100] It's worth noting that the last two objective terms act as regularizers for the noise term, meaning they effectively suppress excessive noise in the output data. This ensures that the model is more robust during optimization and better adapts to the various noise and uncertainty conditions in the real world. This adaptive approach to learning the weights for the loss function makes the model more robust, improving its performance and reliability in real-world applications.

[0101] In this embodiment, each module is implemented by a processor, and a memory is appropriately added when storage is required. The processor may be, but is not limited to, a microprocessor MPU, a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, etc. The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0103] Based on the same concept, an embodiment of the present invention further provides a traffic signal coordination method based on deep reinforcement learning and situation prediction. The method is implemented based on the above-mentioned traffic signal coordination system based on deep reinforcement learning and situation prediction, and specifically includes:

[0104] Through the simulation interaction environment module, the traffic system simulation software SUMO is configured according to the actual topological structure of the road network and the corresponding vehicle routing data to build a simulation interaction environment similar to the actual traffic scene. In this simulation interaction environment, the traffic system simulation software SUMO's Traci interface and sumolib tool are used to accurately obtain the traffic operation information of vehicles on the road network and the traffic intersections controlled by traffic lights during the simulation interaction process. At the same time, the signal control plan output by the model is sent to the traffic lights through the Traci interface to adjust the traffic phase. The operation of the actual traffic environment is then simulated through SUMO, and the traffic status information of each signal intersection in the road network is recorded. This information is fed back to the model for the calculation of the next round of control plans.

[0105] The multi-intersection signal collaborative optimization model that integrates situational predictions is based on current and future traffic conditions to achieve effective decision-making based on integrated situational predictions. Specifically, the multi-intersection complex traffic dynamic graph representation module uses a multi-layer perceptron and a graph attention mechanism to achieve information exchange of multi-intersection traffic status through the adjacency relationship between intersections, effectively extracting the state characteristics of adjacent intersections. The fine-grained traffic flow prediction module that integrates uncertainty achieves lane-level situational awareness by predicting future road traffic volume, thereby effectively assisting the control decision-making process of traffic lights. The multi-intersection traffic signal phase collaborative selection module receives the complex dynamic graph representation of multi-intersection traffic and integrates the uncertainty factors of future traffic flow changes through the prediction data provided by the fine-grained traffic flow prediction module that integrates uncertainty, further improving the accuracy and robustness of decision-making.

[0106] The uncertainty-weighted multi-task training module introduces uncertainty to measure the importance of the uncertainty-integrated prediction task and the multi-intersection traffic light control task, thereby dynamically adjusting the weight of each task in the overall loss function; the more stable the task, the higher its quality and the greater its contribution to the overall loss; through the uncertainty weighting mechanism, the model training process is optimized and the robustness and generalization ability of the model are improved.

[0107] It should also be noted that the traffic signal coordination system based on deep reinforcement learning and situation prediction described in this embodiment corresponds to the traffic signal coordination method based on deep reinforcement learning and situation prediction. The description and limitation of the method are also applicable to the system and will not be repeated here.

[0108] It can be seen from the above technical solutions that the traffic signal coordination method and system based on deep reinforcement learning and situation prediction described in the embodiment of the present invention provides reliable data support and decision support for the coordinated control of multi-intersection signals by predicting the range of changes in future traffic situations in advance. The method first uses the open source traffic system simulation software SUMO to build a simulated interactive traffic environment to realize the control of vehicles at signal intersections; then constructs a multi-intersection signal coordination optimization model that integrates situation prediction, and realizes effective decision-making based on the current and future traffic conditions; collects traffic status data during the interaction between the model and the simulation environment, and after pre-processing the collected data, constructs training samples and divides the data set; introduces uncertainty to measure the importance of prediction tasks and decision tasks, and uses a multi-task adaptive weighted loss function to train the model, while achieving reliable prediction and effective decision-making. The present invention defines a new signal light control method that integrates traffic uncertainty prediction, that is, predicting the range of future traffic fluctuations in advance to achieve reliable prediction, providing data support and important basis for effective control of signal lights to complete traffic diversion, which has broad application prospects and value in the field of traffic signal control.

[0109] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. It is not intended to limit the scope of the invention to be protected, but merely represents a preferred embodiment of the present invention. Those skilled in the art should understand that the scope of the invention involved in the present invention is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the inventive concept. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative work shall fall within the scope of protection of the present invention.

Claims

1. A traffic signal coordination system based on deep reinforcement learning and situation prediction, characterized by: The system includes: a simulation interactive environment module, a multi-intersection signal collaborative optimization model integrating situation prediction, and a multi-task training module based on uncertainty weighting; wherein, The simulation interaction environment module is used to configure the traffic system simulation software SUMO according to the actual topology of the road network and the corresponding vehicle routing data, and build a simulation interaction environment similar to the actual traffic scene; The multi-intersection signal collaborative optimization model integrating situation prediction is used to achieve effective decision-making based on current and future traffic conditions; The uncertainty-weighted multi-task training module is used to measure the importance of the uncertainty-integrated prediction task and the multi-intersection signal light control task by introducing uncertainty, thereby dynamically adjusting the weight of each task in the overall loss function; The multi-intersection signal collaborative optimization model integrating situation prediction includes a multi-intersection complex traffic dynamic graph representation module, a fine-grained traffic flow prediction module integrating uncertainty, and a multi-intersection traffic signal phase collaborative selection module; wherein, The multi-intersection complex traffic dynamic graph representation module is composed of a first multi-layer perceptron and a graph attention mechanism submodule, which is used to realize information exchange of multi-intersection traffic status through the adjacency relationship between intersections and extract the status features of adjacent intersections; The uncertainty-integrated fine-grained traffic flow prediction module includes an instance normalization layer, an encoder, a convolutional layer, a summation operator, and a second multi-layer perceptron, and is used to achieve lane-level situational awareness by predicting future traffic flow on the road, thereby assisting the control decision-making process of traffic lights; The multi-intersection traffic light phase collaborative selection module is composed of a third multi-layer perceptron, which is used to receive complex dynamic graph representations of multi-intersection traffic; it is also used to integrate the uncertainty factors of future traffic flow changes through the prediction data provided by the fine-grained traffic flow prediction module that integrates uncertainty, so as to further improve the accuracy and robustness of decision-making.

2. The traffic signal coordination system based on deep reinforcement learning and situation prediction according to claim 1 is characterized in that: The simulation interaction environment module is further used for: Using the Traci interface and sumolib tool of the traffic system simulation software SUMO, traffic operation information of vehicles on the road network and traffic intersections controlled by traffic lights are obtained during the simulation interaction process. At the same time, the signal control plan output by the multi-intersection signal collaborative optimization model that integrates situation prediction is sent to the traffic lights through the Traci interface to adjust the traffic phase. The operation of the actual traffic environment is simulated through SUMO, and the traffic status information of each signal intersection in the road network is recorded. This information is fed back to the multi-intersection signal collaborative optimization model that integrates situation prediction for calculation of the next round of control plans. The traffic phase is composed of one or more traffic turns that obtain the right of way at the same time without causing traffic conflicts.

3. The traffic signal coordination system based on deep reinforcement learning and situation prediction according to claim 1 is characterized in that: The first multi-layer perceptron MLP(·) is used to represent the states of N intersections at time t Perform feature extraction to obtain N intersection state features at time t The graph attention mechanism submodule is used to use the state feature H ′ t and the intersection adjacency matrix A as input, aggregate the state features of each adjacent intersection, and thus obtain the intersection feature representation that integrates the states of adjacent intersections 4. The traffic signal coordination system based on deep reinforcement learning and situation prediction according to claim 3 is characterized in that: The input of the multi-intersection complex traffic dynamic graph representation module is the N intersection states at time t and intersection adjacency matrix The intersection status of each intersection Including the current signal phase p of the intersection i and the number of vehicles on the import lane v i , expressed as The adjacency matrix consists of a 0-1 matrix Indicates that when intersection i and intersection j are adjacent, a ij =1, otherwise a ij =0.

5. The traffic signal coordination system based on deep reinforcement learning and situation prediction according to claim 4 is characterized in that: The input data of the fine-grained traffic flow prediction module integrating uncertainty is the historical traffic flow sequence of a certain intersection. The specific execution operations are as follows: The input historical traffic flow data is instance-normalized and fed into an encoder for traffic flow prediction. The encoder consists of a traffic flow prediction model that uses two-dimensional convolution operations to predict the changing trends of future multi-step traffic flow. The time dimension is then compressed using a sum operator to obtain the range of changes in the new traffic flow in the import lanes within a predetermined time period in the future, thereby obtaining the predicted results of future traffic flow. Finally, a third multi-layer perceptron is used to extract features from the predicted results.

6. The traffic signal coordination system based on deep reinforcement learning and situation prediction according to claim 5 is characterized in that: In the workflow of the multi-intersection traffic signal phase collaborative selection module, the outputs of the multi-intersection complex traffic dynamic graph representation module and the fine-grained traffic flow prediction module that integrates uncertainty are first directly spliced ​​together; then, the spliced ​​feature vector is sent to a third multi-layer perceptron, and the input data is nonlinearly transformed and processed to better understand the complex traffic situation; the value of each action in the action set is calculated by the third multi-layer perceptron, thereby evaluating the pros and cons of each possible phase.

7. The traffic signal coordination system based on deep reinforcement learning and situation prediction according to claim 6 is characterized in that: The multi-intersection traffic light phase coordinated selection module adopts the ε-greedy algorithm, which randomly selects an action from the action set with a probability of ε and selects the action with the greatest value with a probability of 1-ε; The specific process of selecting an action at a single intersection is as follows: In formula (5), represents the action set; Q represents the action value function, which contains the value of each action a in the action set; s is the current signal phase p i and the number of vehicles on the import lane v i represents the state of the intersection; β represents a number randomly sampled from the interval [0,1]; ε represents a set fixed probability value, and its value is in the interval [0,1].

8. A traffic signal coordination method based on deep reinforcement learning and situation prediction, characterized in that: The method is implemented based on the system according to any one of claims 1 to 7, and specifically includes: The traffic system simulation software SUMO is configured through the simulation interaction environment module according to the actual topological structure of the road network and the corresponding vehicle routing data to build a simulation interaction environment similar to the actual traffic scene; in the built simulation interaction environment, the traffic operation information of the vehicles on the road network and the traffic intersections controlled by the signal lights are obtained by using the Traci interface and sumolib tool of the traffic system simulation software SUMO during the simulation interaction process, and the signal control scheme output by the multi-intersection signal collaborative optimization model of the fusion situation prediction is sent to the traffic light through the Traci interface, so as to adjust the traffic phase; the operation of the actual traffic environment is simulated by SUMO, and the traffic status information of each signal intersection in the road network is recorded at the same time, and this information is fed back to the multi-intersection signal collaborative optimization model of the fusion situation prediction for calculation of the next round of control scheme; The multi-intersection signal collaborative optimization model that integrates situation prediction is based on current and future traffic conditions to achieve effective decision-making based on situation prediction; The uncertainty-weighted multi-task training module introduces uncertainty to measure the importance of the prediction task and the multi-intersection signal light control task, thereby dynamically adjusting the weight of each task in the overall loss function. The more stable the task description, the higher the quality and the greater the contribution to the overall loss. Through the uncertainty weighting mechanism, the training process of the multi-intersection signal collaborative optimization model for fusion situation prediction is optimized to improve the robustness and generalization ability of the model.

Images