Intelligent traffic scheduling method and system based on multi-modal fusion and space-time prediction

Abstract

The invention provides an intelligent traffic scheduling method and system based on multi-modal fusion and space-time prediction, and relates to the technical field of intelligent traffic scheduling, and the method comprises the steps: collecting traffic-related multi-modal data; performing preprocessing and data-level and feature-level fusion processing on the multi-modal data to generate a fusion feature matrix; constructing an STGCN-LSTM space-time prediction model according to the fusion feature matrix, and predicting the traffic flow and queuing length of each entrance lane of the current scene to obtain a prediction result; and dynamically optimizing a signal lamp timing scheme according to the prediction result and the real-time traffic state. By accurately sensing a real-time traffic state, predicting a traffic flow change trend in a short time in the future and dynamically adjusting an intersection signal lamp timing scheme, on-demand timing is realized, the average waiting time of vehicles is shortened, the queuing length at an intersection is reduced, the traffic efficiency at the intersection is improved, and meanwhile, the traffic demands of pedestrians and non-motor vehicles are considered; and the safety and smoothness of traffic operation are ensured.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic scheduling technology, and more particularly to an intelligent traffic scheduling method and system based on multimodal fusion and spatiotemporal prediction. Background Technology

[0002] With the rapid development of AI technology, its applications are becoming increasingly widespread, especially in intelligent transportation, where it plays a crucial role. At intersections of main urban roads, intersections near commercial areas, schools, and hospitals where traffic flow fluctuates significantly, and intersections exhibiting pronounced tidal traffic patterns during morning and evening rush hours, conventional fixed-duration traffic light allocation schemes are insufficient to meet the demands due to large fluctuations in traffic density at different times. This is particularly true during periods of high traffic density, which can lead to longer waiting times for vehicles and pedestrians, significantly impacting intersection efficiency.

[0003] Therefore, how to improve the traffic efficiency of intersections by adaptively scheduling the duration of traffic lights is the technical problem that this invention aims to solve. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides an intelligent traffic scheduling method and system based on multimodal fusion and spatiotemporal prediction. By accurately sensing real-time traffic conditions and predicting traffic flow trends in the near future, the system dynamically adjusts the signal timing scheme at intersections to achieve on-demand timing, reducing average vehicle waiting time, shortening intersection queue lengths, and improving intersection throughput. Simultaneously, it takes into account the needs of pedestrians and non-motorized vehicles, ensuring the safety and smoothness of traffic operations.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: This invention provides an intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction, comprising the following steps: S1: Collect traffic-related multimodal data, including basic traffic flow data, environmental data, time-series management data, and special event data; S2: The multimodal data is preprocessed and fused at the data level and feature level to generate a fused feature matrix, which includes real-time traffic flow status, environmental influencing factors, and time-series trend features; S3: Construct the STGCN-LSTM spatiotemporal prediction model based on the fused feature matrix to predict the traffic flow and queue length of each entrance lane in the current scenario, and obtain the prediction results; S4: Based on the predicted results and real-time traffic conditions, dynamically optimize the traffic light timing scheme.

[0006] Preferably, in step S2, the data after data-level fusion is processed by feature-level fusion based on the MAFNet modal adaptive fusion network to output a fusion feature matrix. The steps include: Traffic flow features, environmental impact factors, and temporal trend features are extracted from the data after data-level fusion, and the extracted features are mapped into tensors with consistent dimensions. The tensor is subjected to deep feature extraction to identify the current scene, and the weights of each modality feature are dynamically adjusted according to the current scene; Cross-modal feature interaction is achieved by constructing a modal interaction matrix, and all interaction features are weighted and fused based on the adjusted weights to obtain a fused feature matrix.

[0007] Preferably, in step S3, the prediction results are obtained, including: Based on the fused feature matrix, a time-series feature tensor is constructed using the sliding window method to encode the spatial topology of the inlet channel to obtain a spatial adjacency matrix. Based on the temporal feature tensor and spatial adjacency matrix, spatial correlation features are extracted using STGCN. Based on the spatial correlation features, temporal dependency features are captured using a bidirectional LSTM. Based on spatial correlation features and temporal dependence features, prediction heads for traffic flow and queue length are constructed respectively, and prediction result matrices are obtained.

[0008] Preferably, in step S3, when determining the current scene, time, weather, and event type are used as input features, and the scene category is output based on the random forest algorithm. A dedicated spatiotemporal prediction sub-model is trained for each scene, and the corresponding spatiotemporal prediction sub-model is matched according to the currently identified scene. The model parameters are updated periodically through an incremental learning algorithm.

[0009] Preferably, based on the prediction results and real-time traffic conditions, the traffic light timing scheme is dynamically optimized, including: Based on forecast data, real-time traffic data, and intersection constraint data, the current and future traffic operation status of the intersection is quantitatively assessed to obtain the traffic flow change trend; Based on traffic flow trends, a multi-objective reward function is constructed to minimize the average vehicle waiting time, maximize intersection efficiency, and minimize the average pedestrian waiting time. An initial timing scheme is generated based on a reinforcement learning algorithm. The deviation between the predicted data and the real-time data is calculated, and the initial timing scheme is optimized based on the deviation to obtain the final signal timing scheme. Based on the final signal timing scheme, a green wave band is constructed using the green wave coordination algorithm to optimize the signal timing of adjacent intersections.

[0010] As a preferred method, the traffic condition is quantitatively assessed by calculating a traffic state index, and the congestion level of each approach lane is scored. The formula is as follows: ; In the formula, This represents the traffic condition index for the i-th approach lane. Indicates the current queue length. This indicates the maximum queue length at the entrance lane. Indicates the current average vehicle speed. This indicates the free-flow vehicle speed of the aforementioned inlet lane. , These represent the weighting coefficients.

[0011] Preferably, the reinforcement learning algorithm is the MORL-NSGA-II algorithm, which generates an initial timing scheme, including: Model the multi-objective timing problem by defining the optimization objective, decision variables, and constraints; A multi-objective reward function is constructed based on the optimization objectives, and the priority of objectives is dynamically adjusted by incorporating traffic flow trends. Based on the multi-objective reward function, the agent learns the mapping between state, action, and multi-objective reward in the traffic environment and generates a candidate set of timing schemes. The candidate timing schemes are sorted by non-dominated order and crowding is calculated, and Pareto optimal solutions are obtained by genetic iterative optimization. From the Pareto optimal solution set, the optimal solution is matched according to the traffic flow trend to obtain the initial timing scheme.

[0012] As a preferred method, the deviation between the predicted data and the real-time data is calculated, and the initial timing scheme is optimized based on the deviation to obtain the final timing scheme, including: Calculate the single-entry lane deviation, which includes traffic flow deviation rate, queue length deviation rate, and traffic state index deviation. The single-entry lane deviations are aggregated into the overall intersection deviation to determine whether the adjustment type is local or global. The initial timing scheme is adjusted according to the type and degree of deviation, and the final timing scheme is obtained after legality verification.

[0013] As a preferred approach, for special scenarios, a robust optimization model is constructed, using interval uncertainty to describe the fluctuation range of key parameters. With the objectives of minimizing the average vehicle waiting time, maximizing traffic efficiency, and minimizing pedestrian waiting time in the worst case, a robust multi-objective function is constructed, and robust constraints are added to ensure traffic capacity in the worst-case scenario. A robust genetic algorithm is then used to solve for the optimal solution in the worst-case scenario. The robust multi-objective function formula is as follows: ; In the formula, C is the duration of the signal period. Let j be the duration of the green light for the j-th phase. Let i be the traffic flow of the i-th entrance lane. This indicates taking the worst-case value of the uncertainty parameter. These represent dynamic weights, Indicates the vehicle waiting time. Indicates traffic efficiency. Indicates the waiting time for pedestrians.

[0014] On the other hand, the present invention provides an intelligent traffic scheduling system based on multimodal fusion and spatiotemporal prediction, the system comprising: Data acquisition module: used to collect traffic-related multimodal data, including basic traffic flow data, environmental data, time series management data, and special event data; Multimodal fusion module: used to preprocess the multimodal data and perform data-level and feature-level fusion processing to generate a fusion feature matrix, which includes real-time traffic flow status, environmental impact factors, and time-series trend features; Predictive analysis module: used to construct STGCN-LSTM spatiotemporal prediction model based on the fused feature matrix, predict the traffic flow and queue length of each entrance lane in the current scenario, and obtain the prediction results; The predictive analysis module includes a scene recognition unit, which takes time, weather, and event type as input features, outputs scene categories based on the random forest algorithm, trains a dedicated spatiotemporal prediction sub-model for each scene, and matches the corresponding spatiotemporal prediction sub-model according to the currently recognized scene. Signal scheduling module: used to dynamically optimize the signal timing scheme based on the prediction results and real-time traffic conditions.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention accurately senses real-time traffic conditions and predicts traffic flow trends in the near future. It generates initial timing using MORL-NSGA-II and dynamically corrects it based on the deviation between actual and predicted values, achieving on-demand timing. Compared to traditional fixed timing, this significantly reduces average vehicle waiting time and greatly improves intersection capacity. Based on the MAXBAND algorithm, a green wave band is constructed for main roads. After optimizing the phase difference between adjacent intersections, regional collaborative scheduling dynamically diverts congested traffic, avoiding overload at single intersections, improving overall regional traffic efficiency, reducing the average number of stops per vehicle on main roads, and lowering commuting time in tidal traffic scenarios. The robust optimization model ensures that even in extreme scenarios such as heavy rain and accidents, the worst-case traffic efficiency remains ≥60% of the design value, and the queue length increase is controlled within 10%, preventing congestion spread and greatly optimizing the user experience. In short, this invention dynamically adjusts the traffic light timing scheme at intersections to reduce average vehicle waiting time, reduce queue length, and improve intersection efficiency, while also considering the needs of pedestrians and non-motorized vehicles, ensuring the safety and smoothness of traffic operations. Attached Figure Description

[0016] Figure 1 This is a flowchart of the intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction of the present invention; Figure 2 This is a flowchart of the dynamic optimization traffic light timing scheme of the present invention; Figure 3 This is a flowchart of the invention for generating an initial timing scheme based on a reinforcement learning algorithm; Figure 4 This is a block diagram of the intelligent traffic scheduling system based on multimodal fusion and spatiotemporal prediction of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0018] Please see Figure 1 As shown, this invention provides an intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction, comprising the following steps: S1: Collect traffic-related multimodal data, including basic traffic flow data, environmental data, time-series management data, and special event data; The system adopts an edge layer-region layer-cloud layer architecture. At each intersection, one edge computing node, one set of video cameras (covering the entire intersection in four directions), two microwave radars (deployed at the entrance lanes), several geomagnetic coil detectors (one per lane), and one set of meteorological sensors are deployed. A regional control center is set up every 3-5 adjacent intersections, deploying a regional collaborative scheduling server to achieve data sharing and collaborative decision-making between intersections. A central management platform is deployed, with a built-in sensor status monitoring module to detect the equipment's operating status in real time. When a sensor fails, it automatically switches to other sensors to fill the gap and simultaneously sends alarm information to notify maintenance personnel for repair. This enables the aggregation of traffic data across the entire area, model training and updates, and system operation and maintenance management.

[0019] Traffic flow basic data: Video cameras collect images of vehicles, pedestrians, and non-motorized vehicles at intersections; microwave radar detects vehicle speed, flow rate, and distance; geomagnetic coil detectors count the number of vehicles passing through and their dwell time; and data such as traffic flow (vehicles / 5 minutes), vehicle speed (km / h), queue length (meters), vehicle type (small cars, large cars), and pedestrian flow (people / 5 minutes) are obtained for each entrance and exit lane.

[0020] Environmental data: Rainfall, visibility, wind speed, and other data are collected by meteorological sensors deployed at intersections, and light intensity (which affects the image quality of the cameras) is collected by light sensors.

[0021] Time-series related data: Historical traffic flow data (traffic data during morning and evening rush hours, holidays, and special weather conditions in the past year) and basic intersection information (number of lanes, turning restrictions, and intersection spacing) are obtained by connecting to the traffic management platform interface.

[0022] Special event data: Information on events that temporarily affect traffic, such as traffic accidents, road construction, and large-scale events, is obtained through traffic police terminals and the 12345 hotline interface.

[0023] S2: The multimodal data is preprocessed and fused at the data level and feature level to generate a fused feature matrix, which includes real-time traffic flow status, environmental influencing factors, and time-series trend features; Specifically, step S2 includes: The multimodal data is preprocessed by data cleaning, alignment, and standardization to generate a high-quality traffic status dataset. Anomaly data caused by sensor malfunctions is removed by an outlier detection algorithm, and missing data is supplemented by linear interpolation and LSTM imputation. Data collected by different devices are synchronized in time based on timestamps, and sensor data at different locations are spatially aligned with intersection lanes and approach lanes based on spatial coordinates. Min-max standardization is performed on data of different magnitudes (such as vehicle speed, traffic flow, and rainfall) to map the data to the [0,1] interval, so as to avoid the impact of magnitude differences on model training.

[0024] The traffic state dataset undergoes data-level fusion processing to obtain corrected structured raw data. For video image data and radar data, the YOLOv8 target detection algorithm is used to detect vehicle / pedestrian target information from the video images, outputting pixel coordinates and type, which are then fused with the target physical coordinates (distance / angle) and speed data detected by radar. A Kalman filter algorithm is used to correct target trajectories, eliminating single-device errors, improving the accuracy of target detection and tracking, and addressing the problem of single-camera interference from lighting and occlusion. For geomagnetic and meteorological data, a weather coefficient 'a' (e.g., a rainstorm coefficient of 0.1) is introduced to correct geomagnetic vehicle traffic flow, using the following formula: In the formula, This indicates the corrected traffic flow. This indicates the flow rate of the geomagnetic vehicle.

[0025] The MAFNet modal adaptive fusion network is used to perform feature-level fusion processing on the data after data-level fusion (corrected structured original data), and output a fused feature matrix. The steps include: Traffic flow features, environmental impact factors, and temporal trend features are extracted from the data after data-level fusion, and the extracted features are mapped into tensors with consistent dimensions. Real-time traffic flow features: Extract 8 directions and 4 types of indicators (vehicle flow / vehicle speed / queue length / vehicle type ratio), the original dimension is 32, and it is converted into a tensor of shape [1,32]; Environmental impact factors: Rainfall / visibility / wind speed / light intensity / temperature were extracted, the original 5 dimensions were converted into tensors of shape [1,5]; Temporal trend features: Traffic flow time series data for the past 15 minutes (1 sample per minute) were extracted. The original dimension was 15×32. After extracting the temporal features through bidirectional LSTM, it was converted into a tensor of [1,16].

[0026] Z-score normalization is applied to each modal tensor to eliminate magnitude differences; 32-dimensional, 5-dimensional, and 16-dimensional tensors are uniformly mapped to a 64-dimensional intermediate feature space through three independent 1-layer fully connected layers (the output shape is [1,64]), thus solving the problem of inconsistent dimensions between different modalities.

[0027] The tensor is subjected to deep feature extraction to identify the current scene, and the weights of each modality feature are dynamically adjusted according to the current scene; Each modality is configured with one lightweight convolutional block (1D Conv+ReLU+batch normalization), with a kernel size of 3 and a stride of 1. The output is still a depth feature tensor in the range [1, 64] (denoted as ). The deep feature tensor of the three modalities is concatenated to [1, 192], and then input into two fully connected layers. The scene category probability is output through the Softmax function (e.g., morning rush hour 0.92, heavy rain 0.05, smooth traffic 0.03). Based on the recognition results of the current scene, the weights of each modal feature are dynamically adjusted in conjunction with a predefined scene-modal weight mapping table (e.g., morning rush hour → traffic flow weight 0.7, time series weight 0.25, environment weight 0.05; heavy rain → environment weight 0.5, traffic flow weight 0.4, time series weight 0.1), replacing the traditional fixed weight fusion.

[0028] Specifically, the initial base weights are obtained from the scene-modal weight mapping table. An attention fine-tuning term is introduced, and the basic weights are fine-tuned by calculating the information entropy of each modality feature. The formula is as follows: In the formula, This represents the weights after fine-tuning the i-th modal feature. This represents the initial basic weights of the i-th modal feature. This represents the fine-tuning coefficient. Let represent the tensor of the i-th modal feature; normalize the fine-tuned weights to obtain the final weights, ensuring that the sum of the weights is 1, as shown in the formula: ; In the formula, Tensor The final weight of (the t-th modality feature) is calculated using this formula, which is used to calculate the final weights of the three modes.

[0029] Cross-modal feature interaction is achieved by constructing a modal interaction matrix, and all interaction features are weighted and fused based on the adjusted weights to obtain a fused feature matrix.

[0030] Specifically, the modal interaction matrix is ​​constructed by calculating the cosine similarity between any two modal features, using the following formula: In the formula, Let represent the cosine similarity between the i-th modal feature and the j-th modal feature. Represents the i-th modal feature. This represents the j-th modal feature, which is calculated using this formula. Three cosine similarities.

[0031] Interaction features are generated based on cosine similarity, using the following formula: ; In the formula, This represents the interaction feature between the i-th modal feature and the j-th modal feature, calculated using this formula. Three interactive features.

[0032] The fused feature matrix is ​​obtained by weighted fusion of all interactive features, as shown in the formula: ; In the formula, F represents the fusion feature matrix, 0.1 is the fixed weight of the interaction feature, and then the 64-dimensional fusion feature matrix F is compressed into a 48-dimensional fusion feature vector and expanded into a batch matrix through a fully connected layer and the ReLU activation function (e.g., if 10 time steps of samples are processed, the output fusion feature matrix has a shape of [10,48].

[0033] S3: Construct the STGCN-LSTM spatiotemporal prediction model based on the fused feature matrix to predict the traffic flow and queue length of each entrance lane in the current scenario, and obtain the prediction results; The predicted results include: Based on the fused feature matrix, a time-series feature tensor is constructed using the sliding window method to encode the spatial topology of the inlet channel to obtain a spatial adjacency matrix. A sliding window method is used to divide the input sequence and the prediction target. The window length is T (T is set to 15, meaning the fused features from the first 15 time steps are used to predict the future), and the prediction step size is K (K can be set to 5 / 10 / 15 / 20 / 30 minutes, corresponding to 5 prediction scales). An input sequence [B,T,D] (B is the batch size, default 32, D is the feature dimension) and a label sequence [B,K,M×2] (M is the number of entrance lanes, e.g., 8 represents the number of directions, ×2 represents the corresponding traffic flow and queue length indicators) are generated. Entrance lanes are treated as graph nodes, and an adjacency matrix [M,M] is constructed with edges indicating whether there is a traffic flow influence relationship (e.g., adjacent entrance lanes in the same direction, upstream and downstream entrance lanes). The spatial association strength between nodes is labeled: direct influence is 1, indirect influence is 0.5, and no influence is 0.

[0034] Based on the temporal feature tensor and spatial adjacency matrix, spatial correlation features are extracted using STGCN. First, the graph structure is mapped by decomposing the temporal input sequence [B,T,D] into a feature matrix [B,D] with T time steps. A fully connected layer maps the D-dimensional features to M-dimensional node features (corresponding to the number of inlet channels), resulting in a graph node feature tensor of [B,M,T]. Then, spatiotemporal convolution operations are performed using STGCN. Each STGCN convolutional block contains a graph convolutional layer and a 1D temporal convolutional layer. In the graph convolutional layer, based on the spatial adjacency matrix, the neighborhood features of each node are aggregated using the spectral graph convolution formula X′=A^XW (A^ is the normalized adjacency matrix, W is the convolution weight) to capture spatial dependencies. In the temporal convolutional layer, a 1D convolutional kernel (size 3, stride 1) is used to extract local features in the temporal dimension. Batch normalization and ReLU activation functions are combined to alleviate gradient vanishing. After processing by three STGCN convolutional blocks, a temporal tensor [B,M,T,D1] (D1 is the spatial feature dimension, set to 32) with fused spatial features is output.

[0035] Based on the spatial correlation features, temporal dependency features are captured using a bidirectional LSTM. Based on the spatial features output by STGCN, LSTM is used to further mine the long-term periodicity (such as the daily morning rush hour pattern) and short-term volatility (such as sudden traffic inflows), adapting and fusing the temporal trend information in the feature matrix. [B,M,T,D1] is reshaped into [B×M,T,D1], meaning that temporal features are extracted separately for each approach lane, improving targeting. The bidirectional LSTM includes a forward LSTM (capturing temporal dependencies from the past to the present) and a backward LSTM (capturing potential trends from the present to the future), each with 64 hidden units, outputting bidirectional temporal features of [B×M,T,128]. Through global average pooling, the features from T time steps are compressed into one time step, resulting in a temporal feature vector of [B×M,128], which is then reshaped into [B,M,128], aligned with the spatial dimension of the approach lanes.

[0036] Based on spatial correlation features and temporal dependence features, prediction heads for traffic flow and queue length are constructed respectively, and prediction result matrices are obtained.

[0037] The spatial features output by STGCN and the temporal features output by bidirectional LSTM are concatenated and compressed into a final spatiotemporal feature of [B,M,64] through a fully connected layer. The traffic flow prediction head (two fully connected layers with ReLU activation function) outputs traffic flow prediction values ​​of [B,K,M], and the queue length prediction head (two fully connected layers with Dropout layer) outputs queue length prediction values ​​of [B,K,M]. The prediction confidence (e.g., 95% confidence interval) is also output. Outliers in the prediction values ​​(e.g., negative traffic flow or queue length exceeding the intersection's maximum capacity) are removed. Extreme prediction results are corrected using linear interpolation. Finally, the prediction results are merged into a prediction result matrix of [B,K,M×2], where each row corresponds to one time step and each pair of columns corresponds to the traffic flow and queue length of one approach lane.

[0038] In determining the current scene, time (hour, day of the week), weather (rainfall, visibility), and event type (construction, large-scale event) are used as input features. A random forest algorithm is used to output the current scene category (weekday morning rush hour, weekday evening rush hour, weekend, holiday, special weather). A dedicated spatiotemporal prediction sub-model is trained for each scene. The corresponding spatiotemporal prediction sub-model is matched based on the currently identified scene, and the model parameters are updated periodically through an incremental learning algorithm. By incorporating event feature input, an emergency prediction sub-model can be triggered when a sudden event is detected, enhancing the model's ability to identify and adapt to sudden events.

[0039] S4: Based on the predicted results and real-time traffic conditions, dynamically optimize the traffic light timing scheme. Please refer to [link / reference]. Figure 2 As shown, it includes: Based on forecast data, real-time traffic data, and intersection constraint data, the current and future traffic operation status of the intersection is quantitatively assessed to obtain the traffic flow change trend; The forecast data includes the predicted traffic flow and queue length of each approach lane for the next 5 / 10 / 15 / 30 minutes output by the spatiotemporal prediction model, as well as the prediction confidence level (e.g., 95% confidence interval); real-time traffic data includes the current real-time traffic flow, queue length, average vehicle speed, vehicle type ratio (large vehicles / small vehicles) of each approach lane collected by intersection sensors (radar, geomagnetic, cameras), as well as pedestrian crossing requests and non-motorized vehicle traffic flow; intersection constraint data includes basic intersection parameters (number of lanes, turning restrictions), timing rule constraints (signal cycle: the total time for the intersection signal lights to complete a set of full phase switching, ranging from 40 to 120 seconds; green light duration for each phase: the passage time for different turning traffic flows, such as eastbound straight, eastbound left turn, southbound straight, etc.; yellow light transition duration: fixed at 3 seconds; all-red light clearing duration: fixed at 1-2 seconds to avoid conflicts between different phase traffic flows).

[0040] Traffic conditions are quantitatively assessed by calculating a traffic state index, which scores the congestion level of each approach lane (0-10 points, 0 for smooth traffic and 10 for severe congestion). The formula is as follows: ; In the formula, This represents the traffic condition index for the i-th approach lane. This indicates the current queue length, and QLmax indicates the maximum queue length of the entrance lane. Indicates the current average vehicle speed. The free-flow vehicle speed of the approach lane is represented by α and β, which represent weighting coefficients. Simultaneously, combined with forecast data, the Predicted Traffic State Index (P-TSI) for each approach lane is calculated for the next 15 minutes to determine the traffic flow trend (increasing / stable / decreasing).

[0041] Based on traffic flow trends, a multi-objective reward function is constructed to minimize the average vehicle waiting time, maximize intersection efficiency, and minimize the average pedestrian waiting time. An initial timing scheme is generated based on a reinforcement learning algorithm. Please see Figure 3 As shown, the reinforcement learning algorithm is the MORL-NSGA-II algorithm. The initial timing scheme is generated using the MORL-NSGA-II algorithm, including: Model the multi-objective timing problem by defining the optimization objective, decision variables, and constraints; The optimization objectives are to minimize the average vehicle waiting time f1, maximize the intersection's traffic efficiency f2 (intersection traffic efficiency = actual traffic volume / designed traffic volume, which is converted to f2′ = 1 - f2 in engineering), and minimize the average pedestrian waiting time. The decision variables are timing parameters, namely the signal cycle C (40~120 seconds, discretized, with a step size of 5 seconds) and the green light duration for each phase. (Straight ahead ≥ 15 seconds, left turn ≥ 10 seconds, discretized, step size 1 second), pedestrian crossing green light duration Gp (≥ 15 seconds, ≤ 40 seconds, discretized, step size 1 second). Constraints are engineering constraints (all green light durations ≥ minimum threshold, cycle ≤ 120 seconds, no phase switching conflicts) and traffic flow trend constraints (if the traffic flow is predicted to increase in the next 15 minutes, and the P-TSI increase is ≥ 20%, then the timing scheme must ensure that the green light duration of the core approach lane is ≥ 1.1 times the current value).

[0042] A multi-objective reward function is constructed based on the optimization objectives, and the priority of objectives is dynamically adjusted by incorporating traffic flow trends. The formula for average vehicle waiting time is: The closer R1 is to 0, the shorter the vehicle waiting time. The formula for the efficiency of traffic flow at intersections is: ; The formula for average pedestrian waiting time is: ; In the reward evaluation stage, a trend weighting factor is added to adjust the priority of the objectives: when traffic flow increases (congestion worsens), if R1 < −0.8 (vehicle waiting time is extremely long), the scheme is directly marked as low priority; when traffic flow is stable, |R1| + |R3| ≤ 1.2 (human and vehicle waiting time is balanced); when traffic flow decreases, if R3 < −0.4 (pedestrian waiting time is relatively long), the scheme is directly marked as low priority; the final output is a 3-dimensional reward vector R = [R1, R2, R3], which is unweighted and retains the independence of multiple objectives.

[0043] Based on the multi-objective reward function, the agent learns the mapping between state, action, and multi-objective reward in the traffic environment and generates a candidate set of timing schemes. state space 98 dimensions, of which It includes the P-TSI and traffic flow change rate of each approach lane in the next 5 / 10 / 15 minutes. The action space A=[G,G1,G2,…,G8,Gp], with a total of 10 discrete actions (8 approach lane green light durations + cycles + pedestrian green light). A digital twin environment for the intersection is built based on VISSIM / SUMO. Real-time traffic flow and predicted traffic flow trends are input. The MORL agent (using a multi-objective DQN framework) interacts with the environment for 1000 rounds. In each round, it outputs one timing scheme and records the corresponding A (timing parameters) and [f1,f2′,f3] (actual values ​​of the 3 objectives) for the scheme. Finally, a candidate set containing 1000 schemes is generated.

[0044] The candidate timing schemes are sorted by non-dominated order and crowding is calculated, and Pareto optimal solutions are obtained by genetic iterative optimization. For each solution x in the candidate set, determine if there exists a solution y such that all objectives of y are better than x. If no such y exists, x is a non-dominated solution (Pareto optimal solution) and is assigned to the first level. The remaining solutions are sorted repeatedly to obtain the second and third levels. For the non-dominated solutions in the first level, calculate the crowding degree. A higher crowding degree indicates a sparser distribution of solutions in the objective space. Retain the non-dominated solutions in the first level. Select the top 80% of solutions by crowding degree from highest to lowest. Randomly select two solutions and cross their timing parameters. Make minor adjustments to the cross-selected solutions to ensure no constraint violations. Repeat the "sorting-screening-generative" process for 50 generations until the solution set stabilizes (new non-dominated solutions < 5%), ultimately obtaining a Pareto optimal solution set containing 10-20 solutions.

[0045] From the Pareto optimal solution set, the optimal solution is matched according to the traffic flow trend to obtain the initial timing scheme. When the traffic flow is increasing, the scheme with the smallest f1 is selected; when the traffic flow is stable, the scheme with the most balanced f1 / f3 is selected; when the traffic flow is decreasing, the scheme with the smallest f3 is selected. The final selected scheme is the initial timing scheme for this scenario.

[0046] Calculate the deviation between the predicted data and the real-time data, optimize the initial timing scheme based on the deviation, and obtain the final timing scheme, including: Calculate the deviation for a single approach lane, the deviation including the traffic flow deviation rate. Queue length deviation rate Traffic State Index Deviation (The deviation is the difference between the actual value and the predicted value, and the deviation rate is the difference between the actual value and the predicted value divided by the predicted value.) The single-entry lane deviations are aggregated into the overall intersection deviation to determine whether the adjustment type is local or global. Local adjustment trigger: If a certain import lane or or This triggers the phase fine-tuning corresponding to that inlet channel; Global adjustment trigger: Local adjustment is triggered if more than 3 approach lanes are affected, or if the average traffic flow deviation rate at the intersection is exceeded. This will trigger a global timing scheme optimization.

[0047] The initial timing scheme is adjusted according to the type and degree of deviation, and the final timing scheme is obtained after legality verification.

[0048] Specifically, for local adjustment triggers, the green light duration is fine-tuned based on a fuzzy control algorithm to reduce the traffic flow deviation rate. Queue length deviation rate The mapping is set as a fuzzy set (negative large / negative medium / negative small / zero / positive small / positive medium / positive large), fuzzy rules are formulated, and the centroid method is used to convert the fuzzy output into a specific adjustment value. After adjustment, the minimum green light duration constraint must be met.

[0049] For global adjustment triggers, a rolling optimization window is constructed starting from the current time. The function is reconstructed with the goal of minimizing the deviation-weighted waiting time. The cycle of the initial timing and the green light duration are iteratively optimized using a genetic algorithm to generate the corrected global scheme.

[0050] For special scenarios, a robust optimization model is constructed, and the fluctuation range of key parameters is described by interval uncertainty. With the objectives of minimizing the average vehicle waiting time, maximizing traffic efficiency, and minimizing pedestrian waiting time in the worst case, a robust multi-objective function is constructed, and robust constraints are added to ensure traffic capacity in the worst case. A robust genetic algorithm is used to solve the optimal solution in the worst case. The robust multi-objective function formula is as follows: ; In the formula, C is the duration of the signal period. Let j be the duration of the green light for the j-th phase. Let i be the traffic flow of the i-th entrance lane. This represents the worst-case scenario for the uncertain parameter (traffic flow) (i.e., the traffic flow scenario that results in the longest waiting time and the lowest efficiency). These represent dynamic weights, Indicates the vehicle waiting time. Indicates traffic efficiency. Indicates the waiting time for pedestrians.

[0051] Specifically, first, identify special scenarios (such as priority for emergency vehicles, dense pedestrian crossings, sudden congestion / accidents), which can be triggered by data anomalies, such as the deviation rate of traffic flow in a single approach lane. Queue length deviation rate or the average deviation rate of intersections Or it may be triggered by an event alarm, such as traffic accident or road construction information pushed by the traffic police terminal, or by a weather sensor detecting heavy rain / snow (visibility < 50 meters); or it may be triggered by a rule.

[0052] For traffic flow fluctuations in specific scenarios, interval uncertainty is used to describe the fluctuation range of key parameters, replacing single forecast values: Traffic flow fluctuation range: , This represents the predicted traffic flow at the i-th entrance, as output by the spatiotemporal prediction model. For robustness coefficients, a value of 0.3 is used for sudden accidents and 0.2 is used for heavy rain. Queue length fluctuation range: , This represents the predicted queue length of the i-th entrance channel output by the spatiotemporal prediction model. Use 0.4, and 0.5 for extreme scenarios; Minimum threshold for traffic efficiency: , The design aims to achieve 60% of the traffic efficiency, ensuring that complete congestion does not occur in extreme scenarios.

[0053] In addition to basic timing constraints (such as In addition to 40≤C≤120), robust constraints are added to ensure passage capacity in the worst-case scenario: Queue length constraint: , This represents the maximum tolerable queue length for the entrance lane. Traffic efficiency constraints: , The designed traffic efficiency is 60%; Pedestrian safety constraints: Pedestrian green light duration Gp ≥ 15 seconds, and pedestrian waiting time in the worst-case scenario. Second.

[0054] The timing parameters (cycle C, green light duration) Discretize the timeframe (cycle step 5 seconds, green light duration step 1 second), and use a robust genetic algorithm (R-GA) to solve for the worst-case optimal solution, i.e., the cycle C and the green light duration for each phase. Encode the time-sharing scheme as a binary chromosome; for each chromosome (time-sharing scheme), substitute the worst-case value of the uncertainty parameter (such as the upper limit of traffic flow and the upper limit of queue length) and calculate the objective function value (waiting time + efficiency loss in the worst case); after 50 iterations through selection (roulette wheel method, retaining chromosomes with high fitness), crossover (single-point crossover, combining time-sharing parameters of different schemes), and mutation (low mutation rate of 0.02, to avoid destroying the optimal solution), output the chromosome with the highest fitness (i.e., the time-sharing scheme that is still optimal in the worst case).

[0055] Based on the final signal timing scheme, a green wave band is constructed using the MAXBAND green wave coordination algorithm to optimize the signal timing of adjacent intersections.

[0056] Specifically, select 3-5 adjacent intersections on the main road (distance ≤ 1 km), prioritizing road segments with strong traffic flow correlation and significant tidal characteristics. Combine the main road speed limit and the actual average traffic speed, and take the minimum of the two as the green wave speed vg. Based on the final timing cycle of the participating intersections, calculate the least common multiple of the cycles. If the cycle difference is too large, the cycle of a single intersection can be fine-tuned (within ±10 seconds) to ensure the feasibility of coordination. Based on the green wave speed and intersection spacing, calculate the travel time of vehicles from upstream to downstream, which is the initial phase difference. Use the MAXBAND algorithm to optimize the phase difference of adjacent intersections with the goal of maximizing the green wave bandwidth. That is, with the main road traffic flow direction as the core, construct a time-distance green wave band model, using the final timing cycle C and green light duration of each intersection. The initial phase difference is taken as input, and the output is the optimal phase difference for each intersection (i.e., the offset of the green light start time relative to the upstream intersection), so that as many vehicles as possible can pass through all intersections within the green wave bandwidth. The larger the bandwidth, the better the green wave effect. The timing scheme of adjacent intersections is adjusted based on the output optimal phase difference, and the green wave effect is verified. If it does not meet expectations, dynamic correction is performed.

[0057] On the other hand, please see Figure 4 As shown, this invention provides an intelligent traffic scheduling system based on multimodal fusion and spatiotemporal prediction, the system comprising: Data acquisition module: used to collect traffic-related multimodal data, including basic traffic flow data, environmental data, time series management data, and special event data; Multimodal fusion module: used to preprocess the multimodal data and perform data-level and feature-level fusion processing to generate a fusion feature matrix, which includes real-time traffic flow status, environmental impact factors, and time-series trend features; Predictive analysis module: used to construct STGCN-LSTM spatiotemporal prediction model based on the fused feature matrix, predict the traffic flow and queue length of each entrance lane in the current scenario, and obtain the prediction results; The predictive analysis module includes a scene recognition unit, which takes time, weather, and event type as input features, outputs scene categories based on the random forest algorithm, trains a dedicated spatiotemporal prediction sub-model for each scene, and matches the corresponding spatiotemporal prediction sub-model according to the currently recognized scene. Signal scheduling module: used to dynamically optimize the signal timing scheme based on the prediction results and real-time traffic conditions.

[0058] The system also includes an application management module, which includes a visualization platform, an early warning unit, and an information push unit. The visualization platform is used to display the traffic flow status (volume, queue length, waiting time), traffic light timing scheme, and prediction results of each intersection in real time, and supports management personnel to manually intervene in the timing scheme (switch to manual mode in case of emergency). The early warning unit is used to send early warning information to management personnel via SMS or platform pop-up when it is predicted that a serious traffic jam (queue length ≥ 200 meters) or equipment failure will occur at a certain intersection. The information push unit is used to push real-time traffic conditions, intersection waiting time predictions, and optimal route suggestions to the public through navigation apps (such as Gaode and Baidu Maps) to guide the public to travel during off-peak hours.

[0059] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction, characterized in that, Includes the following steps: S1: Collect traffic-related multimodal data, including basic traffic flow data, environmental data, time-series management data, and special event data; S2: The multimodal data is preprocessed and fused at the data level and feature level to generate a fused feature matrix, which includes real-time traffic flow status, environmental influencing factors, and time-series trend features; S3: Construct the STGCN-LSTM spatiotemporal prediction model based on the fused feature matrix to predict the traffic flow and queue length of each entrance lane in the current scenario, and obtain the prediction results; S4: Based on the predicted results and real-time traffic conditions, dynamically optimize the traffic light timing scheme.

2. The intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction according to claim 1, characterized in that, In step S2, the data after data-level fusion is processed by feature-level fusion based on the MAFNet modal adaptive fusion network to output a fused feature matrix. The steps include: Traffic flow features, environmental impact factors, and temporal trend features are extracted from the data after data-level fusion, and the extracted features are mapped into tensors with consistent dimensions. The tensor is subjected to deep feature extraction to identify the current scene, and the weights of each modality feature are dynamically adjusted according to the current scene; Cross-modal feature interaction is achieved by constructing a modal interaction matrix, and all interaction features are weighted and fused based on the adjusted weights to obtain a fused feature matrix.

3. The intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction according to claim 1, characterized in that, In step S3, the prediction results are obtained, including: Based on the fused feature matrix, a time-series feature tensor is constructed using the sliding window method to encode the spatial topology of the inlet channel to obtain a spatial adjacency matrix. Based on the temporal feature tensor and spatial adjacency matrix, spatial correlation features are extracted using STGCN. Based on the spatial correlation features, temporal dependency features are captured using a bidirectional LSTM. Based on spatial correlation features and temporal dependence features, prediction heads for traffic flow and queue length are constructed respectively, and prediction result matrices are obtained.

4. The intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction according to claim 1, characterized in that, In step S3, when determining the current scene, time, weather, and event type are used as input features. The scene category is output based on the random forest algorithm, and a dedicated spatiotemporal prediction sub-model is trained for each scene. The corresponding spatiotemporal prediction sub-model is matched according to the currently identified scene, and the model parameters are updated periodically through an incremental learning algorithm.

5. The intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction according to claim 1, characterized in that, Based on the predicted results and real-time traffic conditions, the traffic light timing scheme is dynamically optimized, including: Based on forecast data, real-time traffic data, and intersection constraint data, the current and future traffic operation status of the intersection is quantitatively assessed to obtain the traffic flow change trend; Based on traffic flow trends, a multi-objective reward function is constructed to minimize the average vehicle waiting time, maximize intersection efficiency, and minimize the average pedestrian waiting time. An initial timing scheme is generated based on a reinforcement learning algorithm. The deviation between the predicted data and the real-time data is calculated, and the initial timing scheme is optimized based on the deviation to obtain the final signal timing scheme. Based on the final signal timing scheme, a green wave band is constructed using the green wave coordination algorithm to optimize the signal timing of adjacent intersections.

6. The intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction according to claim 5, characterized in that, Traffic conditions are quantitatively assessed by calculating a traffic state index, which scores the congestion level of each approach lane. The formula is as follows: ； In the formula, This represents the traffic condition index for the i-th approach lane. Indicates the current queue length. This indicates the maximum queue length at the entrance lane. Indicates the current average vehicle speed. This indicates the free-flow vehicle speed at the entrance lane. , These represent the weighting coefficients.

7. The intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction according to claim 5, characterized in that, The reinforcement learning algorithm is the MORL-NSGA-II algorithm. The initial timing scheme is generated using the MORL-NSGA-II algorithm, including: Model the multi-objective timing problem by defining the optimization objective, decision variables, and constraints; A multi-objective reward function is constructed based on the optimization objectives, and the priority of objectives is dynamically adjusted by incorporating traffic flow trends. Based on the multi-objective reward function, the agent learns the mapping between state, action, and multi-objective reward in the traffic environment and generates a candidate set of timing schemes; The candidate timing schemes are sorted by non-dominated order and crowding is calculated, and Pareto optimal solutions are obtained by genetic iterative optimization. From the Pareto optimal solution set, the optimal solution is matched according to the traffic flow trend to obtain the initial timing scheme.

8. The intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction according to claim 5, characterized in that, Calculate the deviation between the predicted data and the real-time data, optimize the initial timing scheme based on the deviation, and obtain the final timing scheme, including: Calculate the single-entry lane deviation, which includes traffic flow deviation rate, queue length deviation rate, and traffic state index deviation. The single-entry lane deviations are aggregated into the overall intersection deviation to determine whether the adjustment type is local or global. The initial timing scheme is adjusted according to the type and degree of deviation, and the final timing scheme is obtained after legality verification.

9. The intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction according to claim 8, characterized in that, For special scenarios, a robust optimization model is constructed, and the fluctuation range of key parameters is described by interval uncertainty. With the objectives of minimizing the average vehicle waiting time, maximizing traffic efficiency, and minimizing pedestrian waiting time in the worst case, a robust multi-objective function is constructed, and robust constraints are added to ensure traffic capacity in the worst case. A robust genetic algorithm is used to solve the optimal solution in the worst case. The robust multi-objective function formula is as follows: ； In the formula, C is the duration of the signal period. Let j be the duration of the green light for the j-th phase. Let i be the traffic flow of the i-th entrance lane. This indicates taking the worst-case value of the uncertainty parameter. These represent dynamic weights, Indicates the vehicle waiting time. Indicates traffic efficiency. Indicates the waiting time for pedestrians.

10. An intelligent traffic scheduling system based on multimodal fusion and spatiotemporal prediction, applied to the intelligent traffic scheduling method based on multimodal fusion and spatiotemporal prediction as described in any one of claims 1-9, characterized in that, The system includes: Data acquisition module: used to collect traffic-related multimodal data, including basic traffic flow data, environmental data, time series management data, and special event data; Multimodal fusion module: used to preprocess the multimodal data and perform data-level and feature-level fusion processing to generate a fusion feature matrix, which includes real-time traffic flow status, environmental impact factors, and time-series trend features; Predictive analysis module: used to construct STGCN-LSTM spatiotemporal prediction model based on the fused feature matrix, predict the traffic flow and queue length of each entrance lane in the current scenario, and obtain the prediction results; The predictive analysis module includes a scene recognition unit, which takes time, weather, and event type as input features, outputs scene categories based on the random forest algorithm, trains a dedicated spatiotemporal prediction sub-model for each scene, and matches the corresponding spatiotemporal prediction sub-model according to the currently recognized scene. Signal scheduling module: used to dynamically optimize the signal timing scheme based on the prediction results and real-time traffic conditions.

Images