Multi-modal space-time traffic state prediction method based on semantic modeling

Abstract

The invention relates to a multi-modal space-time traffic state prediction method based on semantic modeling, and belongs to the field of intelligent traffic. The method comprises the following steps: S1, acquiring multi-source data, and preprocessing the multi-source data; s2, converting the preprocessed data into a space-time sequence, and carrying out category balance processing; s3, constructing an S3MFNet model, wherein the S3MFNet model comprises a spatial feature encoder, a time feature processor and a spatial-temporal feature fusion module; the time feature processor is used for mapping low-dimensional time features to high-dimensional space-time representation; the spatio-temporal feature fusion module realizes efficient fusion and accurate prediction of spatio-temporal features through a cross-modal interaction mechanism and time sequence downsampling convolution; s4, training the MFNet model in the step S3 by adopting a composite loss function so as to optimize the semantic precision and the space-time continuity of traffic state prediction at the same time; and S5, inputting a real-time traffic data flow into the trained MFNet model in the S3, and predicting a probability distribution diagram of a future traffic state.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and relates to a multimodal spatiotemporal traffic state prediction method based on semantic modeling. Background Technology

[0002] The development of traffic condition prediction clearly demonstrates the evolving technological paradigms that have continuously evolved to address the inherent complexity of traffic systems. Early in its development, the field primarily relied on traditional statistical models, such as ARIMA. While these models possessed a solid mathematical foundation, their assumptions of linearity and stationarity made it difficult to capture the highly nonlinear and dynamic nature of real traffic data. Subsequently, machine learning methods (such as support vector machines and gradient boosting trees) improved the ability to handle nonlinear relationships to some extent, but their shallow architecture and reliance on manual feature engineering became increasingly apparent in large-scale spatiotemporal data scenarios. Current research has fully shifted to the deep learning paradigm, particularly graph neural networks (GNNs), which can naturally represent road network topology. GNNs effectively analyze the cascading effects of congestion in space through graph convolutions, while recurrent neural networks (RNNs / LSTMs / GRUs) focus on characterizing long-term dependencies in the time dimension. The combination of these two approaches, resulting in spatiotemporal graph neural networks, has become the mainstream architecture for handling the spatiotemporal dynamics of traffic flow. Despite this, existing models still face multiple challenges: the "black box" nature of deep learning leads to a lack of interpretability in model decisions; its reliance on massive amounts of labeled data results in a significant decrease in generalization ability when data is sparse or encounters rare and unexpected events (such as accidents or extreme weather); furthermore, effectively integrating multi-source heterogeneous data (such as weather, POI, and event information) and the high computational costs brought about by complex models such as Transformers are also problems that urgently need to be solved. In the future, this field is developing in a direction that combines knowledge-guided learning with large-scale model technology, balances prediction accuracy and computational efficiency, and enhances practicality in open and complex environments.

[0003] Current traffic condition prediction technologies suffer from several shortcomings: First, most models heavily rely on complete, high-quality historical data. However, in real-world road networks, data gaps due to sensor malfunctions or communication interruptions are common. Traditional methods often treat data completion and condition prediction as two separate steps, failing to fully utilize the road network topology for effective spatial interpolation. Furthermore, the completion process can easily introduce biases, affecting the accuracy of subsequent predictions. Second, while graph neural network (GNN)-based models can capture the spatial structure of road networks, their adjacency matrices are typically predefined and static, making it difficult to dynamically reflect changes in the spatial correlation of traffic flow over time, such as the propagation effect of sudden congestion within the road network. Additionally, existing models have limited ability to model long-term dependencies in the time dimension. While recurrent neural networks (RNNs) and their variants can handle time series, they suffer from error accumulation problems, especially in long-term predictions, where small errors in the early stages can be amplified as the prediction step size increases. Another significant drawback is that most deep learning models are purely data-driven, lacking the embedding of traffic flow physics principles (such as conservation equations and car-following models). This can lead to predictions that deviate from actual traffic dynamics, especially in scenarios with sparse data or extreme events (such as accidents or severe weather), where generalization ability is significantly insufficient. Finally, existing methods have weak capabilities in fusing multi-source heterogeneous information (such as traffic flow, vehicle trajectories, weather, and construction events), typically relying on only single-modal data and ignoring the dynamic impact of external environmental factors on traffic conditions, thus limiting further improvements in prediction accuracy. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a multimodal spatiotemporal traffic state prediction method based on semantic modeling, aiming to address the major public governance challenge of urban traffic congestion, focusing on improving the accuracy and reliability of traffic state prediction to support the decision optimization of intelligent traffic management systems. Addressing the significant shortcomings of existing methods in spatiotemporal dynamic interaction modeling, multimodal information fusion, class imbalance handling, and long-term dependency capture, this invention proposes a multimodal spatiotemporal traffic state prediction network based on semantic modeling (S3MFNet). By integrating high-resolution multi-source data (such as traffic flow, vehicle trajectory, meteorological and event information), constructing structured spatiotemporal samples, and designing core modules such as multi-scale feature extraction, gating residual mechanism, cross-modal attention fusion, and physical constraint loss function, S3MFNet is committed to simultaneously enhancing the model's ability to represent traffic flow evolution patterns, its robustness to sudden events, and its maintenance of physical consistency, ultimately achieving advanced prediction and adaptive regulation of road network operation status, providing technical support for the sustainable development of urban transportation.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A multimodal spatiotemporal traffic state prediction method based on semantic modeling specifically includes the following steps: S1: Acquire multi-source data, including traffic flow, vehicle trajectory, weather and event information, and preprocess it; S2: The preprocessed data is transformed into a spatiotemporal sequence and class balancing is performed. Finally, the balanced dataset is divided into training set, validation set and test set according to the principle of non-overlapping time. S3: Construct a multimodal spatiotemporal traffic state prediction network model based on semantic modeling, abbreviated as S3MFNet model, including a spatial feature encoder, a temporal feature processor, and a spatiotemporal feature fusion module; the temporal feature processor is used to map low-dimensional temporal features to high-dimensional spatiotemporal representation; the spatiotemporal feature fusion module achieves efficient fusion and accurate prediction of spatiotemporal features through cross-modal interaction mechanism and temporal downsampling convolution; S4: The S3MFNet model is trained using a composite loss function to simultaneously optimize the semantic accuracy and spatiotemporal continuity of traffic state prediction. S5: Input the real-time traffic data stream into the trained S3MFNet model to predict a detailed probability distribution map of future traffic conditions.

[0006] Furthermore, in step S1, a distributed asynchronous acquisition framework is used to acquire multi-source data; the preprocessing includes: spatiotemporal registration, i.e., using an improved RANSAC algorithm to align geographic coordinate systems; image enhancement; and storage architecture, i.e., constructing time-series data cubes at hourly granularity.

[0007] Furthermore, step S2 specifically includes: transforming the preprocessed data into a spatiotemporal sequence, specifically using a sliding window strategy that follows the time order to generate input-output pairs; for key categories in the traffic semantic map, using class weight adjustment based on logarithmic inverse frequency or dynamic oversampling techniques to enhance the model's learning ability for minority class samples; and dividing the dataset into training set, validation set, and test set according to the principle of non-overlapping time.

[0008] Furthermore, in step S3, the spatial feature encoder includes a semantic embedding layer, a multi-scale 3D convolution module, and a gated residual block; The semantic embedding layer is used to map discrete traffic state labels to a continuous semantic vector space; The multi-scale 3D convolution module extracts features in the spatiotemporal joint domain through three-dimensional convolution kernels to simultaneously model the spatial locality and temporal continuity of traffic states. The gated residual block adjusts the feature reuse ratio through a dynamic gating function and gradient optimization mechanism.

[0009] Furthermore, in step S3, the processing flow of the temporal feature processor includes high-order representation learning and dynamic dimension alignment. The higher-order representation learning is achieved through nonlinear dimensionality increase and feature decoupling. The nonlinear dimensionality increase maps the original temporal features to a higher-dimensional latent space through a two-layer fully connected network to learn the higher-order patterns of traffic state evolution. The hidden layer of the feature decoupling uses ReLU activation to achieve feature sparsity, so that the contributions of different time factors (hours, days of the week, etc.) can be separated. The dynamic dimension alignment achieves feature map resolution matching through trilinear interpolation; the interpolation weights are controlled by learnable parameters.

[0010] Furthermore, in step S3, the spatiotemporal feature fusion module specifically includes: establishing a dynamic correlation between spatial features and temporal features, and driving the generation of prediction results with physical constraints; In the cross-modal interaction mechanism, the model uses cross attention to achieve bidirectional alignment of spatial and temporal features; specifically, the feature map output by the spatial branch generates a query vector through linear transformation, while the temporal features generate key and value vectors; attention weights are calculated through normalized dot products. The temporal downsampling convolution controls the prediction stride by adjusting the size of the convolution kernel in the temporal dimension.

[0011] Furthermore, in step S4, the composite loss function Including the weighted cross-entropy loss and the spatiotemporal smoothing regularization term, the expression is:

[0012] in, For category weight coefficients, For category The total number of pixels; These are the weighting coefficients; Let cross-entropy be the loss function. For gradient operators, For the model's predicted output, This represents the length of the time window.

[0013] Furthermore, in step S4, the spatiotemporal smoothing regularization term includes: The time-based second-order difference constraint is expressed as follows:

[0014] in, Indicates time smoothing loss, In the future t The complete output of each time step Represents the square of the Frobenius norm; Spatial gradient constraint, expressed as:

[0015] in, Indicates spatial smoothing loss, This represents the gradient operator in the horizontal direction of space. This represents the gradient operator in the vertical direction of space.

[0016] Furthermore, 9. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 1 is characterized in that, in step S4, the S3MFNet model is trained by using the Adam optimizer and adopting a cosine annealing strategy to dynamically adjust the learning rate, and performing multiple rounds of training until the loss function converges.

[0017] The beneficial effects of this invention are as follows: This invention designs a novel spatiotemporal residual network model, S3MFNet, which integrates innovative modules such as multi-scale spatiotemporal feature extraction, gated residual blocks, cross-modal fusion mechanisms, and physical constraint optimization. Firstly, in the temporal dimension, traffic flow features are decomposed into three parts: nearest neighbor dependency (short-term disturbances), daily periodicity, and weekly trend. Parallel residual convolutional branches extract corresponding features, and dynamic weighted adaptive fusion is used. This multi-branch design helps to simultaneously capture both local fluctuations and long-term evolution patterns of traffic flow. Secondly, the residual blocks in STResNet incorporate a gating mechanism to filter and strengthen key information flows, effectively improving the training stability and information representation capabilities of deep networks. Thirdly, the model comprehensively introduces multimodal data (such as real-time road network images, weather, traffic events, etc.) during the feature fusion stage, and achieves information interaction between different modalities through the design of cross-channel or attention fusion units. Overall, S3MFNet, through the combination of multi-scale spatiotemporal fusion and physical priors, exhibits stronger expressive power and generalization in complex traffic scenarios.

[0018] Furthermore, this invention emphasizes a high degree of compatibility between data design and model structure. To support the training and evaluation of S3MFNet, this invention constructs a new high-resolution urban traffic image dataset (CCTID 2025). This dataset records the real-time distribution of large-scale traffic flows in image form, containing fine-grained spatial and temporal information. The high spatial resolution and multi-channel features of CCTID data provide rich input for the model, enabling S3MFNet to capture more refined traffic flow patterns. Research shows that fusing multi-source high-resolution data can significantly improve the accuracy of traffic prediction. S3MFNet trained using CCTID achieved significant performance improvements in experiments, validating the importance of co-designing data and models.

[0019] In summary, the S3MFNet model employed in this invention significantly improves prediction accuracy and enhances the ability to identify key regions. By introducing a logarithmically inverse frequency-weighted loss function and dynamic spatial enhancement strategies (such as morphological dilation of restricted areas), this invention effectively addresses the problem of insufficient learning of minority class samples and significantly enhances robustness to extreme events and class imbalances. Furthermore, by incorporating spatiotemporal second-order difference constraints into the loss function design, this invention ensures that the prediction results conform to physical laws and exhibit better spatiotemporal continuity. The dynamic time window adaptive strategy employed in this invention adjusts the length of the input sequence according to the real-time traffic flow change rate, intelligently optimizing computational resources during peak and off-peak periods, further reducing the average prediction error, improving inference speed, and achieving a balance between accuracy and efficiency, demonstrating potential for real-time applications.

[0020] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0021] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the overall architecture of the multimodal spatiotemporal traffic state prediction method based on semantic modeling of this invention. Figure 2 Here is a diagram of the S3MFNet model framework; Figure 3 Loss curve for training the S3MFNet model; Figure 4 This is a showcase of some of the achievements of the traffic congestion monitoring platform; Figure 5 A radar chart comparing multiple models; Figure 6 To verify the effectiveness of each module in the S3MFNet model. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] Please see Figure 1 This invention provides a multimodal spatiotemporal traffic state prediction method based on semantic modeling. It constructs a logically rigorous and highly operable five-step process covering the entire lifecycle of a traffic state prediction system, from data preparation to actual deployment. Each step is detailed in paragraphs, outlining its objectives, core tasks, and technical implementation, ensuring the systematic nature and depth of the solution. Figure 1 This is a flowchart illustrating the overall process framework of the method of the present invention. To achieve the above objectives, the present invention proposes the following technical solutions: Step 1: Data Acquisition and Preprocessing Data preprocessing is the foundational stage of the entire system, with the core objective of transforming raw, multi-source, heterogeneous traffic data into high-quality, standardized input. This step encompasses the acquisition, cleaning, and fusion of multimodal data, specifically including data acquisition from traffic sensors, vehicle trajectories, meteorological information, and emergency reports, and correction for common issues such as data loss due to sensor malfunctions, GPS drift errors, or inconsistencies in timestamps from different sources. Key technologies involve interpolation-based missing value imputation, statistical model-based outlier detection and removal, and the use of georeferencing algorithms (such as an improved RANSAC) to unify various data types into the same spatiotemporal coordinate system, ensuring spatial offset errors are controlled within ±0.5 meters. Furthermore, preliminary feature engineering, such as time period encoding (hours, days of the week, holidays) and spatial topology extraction, is also completed at this stage, providing a clean and aligned multidimensional data foundation for subsequent model construction. The rigorous execution of this step directly determines the signal-to-noise ratio and reliability of the model input, and is the primary prerequisite for ensuring prediction accuracy.

[0024] Detailed steps for data acquisition and preprocessing: (1) Data source integration: It integrates traffic status data from Baidu Maps, topological data from OpenStreetMap, and multi-dimensional environmental information such as weather and POIs (Points of Interest). The data covers 278 main roads in the nine main urban districts, ensuring spatial integrity.

[0025] (2) Data collection strategy: A distributed asynchronous data acquisition framework was adopted, calling the platform's authorized interface at a frequency of 6 time windows per hour (10-minute intervals) to achieve continuous spatiotemporal coverage. The data acquisition period was from March to November 2024, resulting in 38,000 traffic situation images.

[0026] (3) Pre-processing line: Spatiotemporal registration: The improved RANSAC algorithm is used for geographic coordinate system alignment, and the offset error of road intersections is controlled within ±0.5 meters.

[0027] Image enhancement: Integrating HSV color space conversion and morphological filtering (such as dilation and erosion operations) improves image quality while preserving road structure information. For example, traffic conditions are divided into red restricted areas, yellow congested areas, and green unobstructed areas through color segmentation.

[0028] Storage architecture: Time-series data cubes are constructed at the hourly granularity, supporting efficient querying and access. After preprocessing, the dataset possesses high resolution (spatial coverage of 3800 square kilometers) and multimodal characteristics (RGB images + traffic vector data), providing rich input for the model.

[0029] Step 2: Dataset Construction Building upon data preprocessing, this step aims to construct a structured dataset that combines spatiotemporal continuity with statistical representativeness. Its core task is to transform the cleaned data into a sequence of samples that the model can directly learn from, focusing on addressing the inherent spatiotemporal dependencies and class imbalances in traffic data. By employing a sliding window strategy that strictly adheres to temporal order, input-output pairs are generated. For example, traffic state images and their corresponding temporal feature matrices over six consecutive time steps (60 minutes) are used as input, with the state over the next three time steps (30 minutes) as the prediction target. The dataset is divided into training, validation, and test sets based on the principle of non-overlapping time intervals to ensure the fairness of model evaluation and prevent future information leakage. Addressing the severe imbalance where key categories (such as red areas representing prohibited traffic) account for less than 6% of the traffic semantic map, class weight adjustment based on logarithmic inverse frequency or dynamic oversampling techniques are introduced to enhance the model's learning ability for minority class samples. The final dataset is a high-dimensional data cube containing spatiotemporal sequences, metadata, and balanced labels, providing a solid foundation for training complex prediction models.

[0030] Step 3: S3MFNet Model Construction The S3MFNet model aims to design a predictive network capable of deeply fusing spatiotemporal features and multimodal information. This step uses the semantic modeling-based multimodal spatiotemporal traffic state prediction network (S3MFNet) as its architectural blueprint, focusing on integrating three major mechanisms: multi-scale feature extraction, dynamic information flow control, and physical law embedding. At the feature extraction level, a three-dimensional convolutional kernel is used to capture local short-term fluctuations (such as sudden congestion) and macroscopic long-term trends (such as morning and evening rush hour patterns) in parallel in the joint spatiotemporal domain. To address the gradient decay problem in deep network training, gated residual blocks are introduced. Through a learnable gating function, the feature reuse ratio of different paths is adaptively adjusted to enhance the network's information representation capability. Simultaneously, a cross-modal attention fusion module is designed to enable bidirectional interaction and alignment between spatial visual features (such as traffic flow images) and temporal context features (such as weather and events). To ensure that the prediction results conform to actual traffic dynamics, physical constraints, such as spatiotemporal second-order difference constraints, are embedded in the model training objective to suppress non-physical abrupt predictions. The final output of this model is a detailed probability distribution map of future traffic states, rather than just a single-value prediction.

[0031] like Figure 2 As shown, the various parts of the S3MFNet model are described in detail below: 1. Spatial Feature Encoder 1) Semantic embedding layer This layer aims to map discrete traffic state labels to a continuous semantic vector space, addressing the high-dimensional sparsity problem of traditional one-hot encoding. Given an input semantic graph... The embedding process is defined as follows: (1) in, Indicates learnable parameters, c It represents a discrete integer value, indicating the traffic state category number of a location at a specific time. T , H , W The three dimensions of the input data semantic graph X are time step, height, and width.

[0032] Mapping discrete semantic labels to a continuous vector space allows for the learning of parameters. The gradient update formula is: (2) in, Represents the total loss function. This represents the embedded feature tensor obtained after passing through the embedding layer at time step t; T Indicates the time step; This indicates an indicator function, which is a conditional function that equals 1 when the condition is true and 0 when the condition is false. This represents the set of original input semantic labels at all locations and time step t. This represents the category index (k=0, 1, 2).

[0033] 2) Multi-scale 3D convolution This module extracts features in the spatiotemporal joint domain using 3D convolutional kernels. Its core idea is to simultaneously model the spatial locality and temporal continuity of traffic states. Given an input feature tensor... The 3D convolution operation is defined as follows: (3) in, This represents the value at a specific position in the output feature tensor. The learnable weight parameters represent the 3D convolution kernel. This represents the value at a specific position in the input feature tensor. , , , These represent the number of channels in the input features, the radius of the 3D convolution kernel in the time, height, and width directions, respectively. They determine the receptive field size of the convolution kernel (i.e., the "detector"). Corresponding to a 3×3×3 convolution kernel; For channel dimension, These are learnable weights. They are activated using the ReLU function. Introducing nonlinearity: (4) in, This represents the final output feature after nonlinear activation. This indicates the bias term.

[0034] 3) Gated residual block The gated residual block is the core component proposed in this invention. It addresses the gradient decay and feature degradation problems in deep spatiotemporal modeling of traditional residual networks by dynamically adjusting the feature reuse ratio. Its core principle is a dynamic gating function and a gradient optimization mechanism. Its computational process is formally defined as follows: (5) Gating function It consists of a 1×1×1 convolution and a Sigmoid activation, where ⊙ represents element-wise multiplication.

[0035] (6) in, The weight parameters represent the gate function. This represents the bias parameter of the gating function.

[0036] The introduction of gating weights makes the gradient backpropagation path differentiable and adjustable: (7) 2. Time Feature Processor This module is designed to map low-dimensional temporal features to high-dimensional spatiotemporal representations, solving the challenge of multimodal feature scale matching. Its processing flow comprises two core stages: high-order representation learning and dynamic dimensionality alignment. High-order representation learning is achieved through nonlinear dimensionality upscaling and feature decoupling. Nonlinear dimensionality upscaling utilizes a two-layer fully connected network to transform the original temporal features... Mapping to a 512-dimensional latent space, higher-order patterns of traffic state evolution (such as peak-hour cycles and holiday effects) are learned. The hidden layer for feature decoupling employs ReLU activation to achieve feature sparsity, allowing the contributions of different time factors (hours, days of the week, etc.) to be separated. The higher-order representation learning expression is: (8) in, These represent the weight matrices of the first fully connected layer and the second fully connected layer, respectively. This represents the high-order time feature representation obtained after nonlinear transformation through two layers of network. , These represent the bias vectors of the first fully connected layer and the second fully connected layer, respectively.

[0037] Dynamic dimension alignment achieves feature map resolution matching through trilinear interpolation: (9) Interpolation weights From learnable parameters control: (10) in, Indicates the interpolation weights. The spatiotemporal distance between feature locations.

[0038] 3. Spatiotemporal Feature Fusion Module This module achieves efficient fusion and accurate prediction of spatiotemporal features through a collaborative design of cross-modal interaction mechanisms and temporal downsampling convolutions. Its core innovation lies in establishing a dynamic correlation between spatial and temporal features, and driving the generation of prediction results with physical constraints. In the cross-modal interaction mechanism, the model employs cross-attention to achieve bidirectional alignment of spatial and temporal features. Specifically, the feature map output from the spatial branch generates a query vector through linear transformation, while the temporal features generate key and value vectors. Attention weights are calculated using normalized dot products, with the following formula: (11) in, This represents attention weights or alignment scores. This represents a query vector derived from spatial features. Represents the key vector. The scaling factor is the query vector Q. s and bond vector K t Dimensions.

[0039] This mechanism enables each spatial location to adaptively focus on key information at different time steps. For example, when a spatial feature detects a red light at an intersection, the model strengthens its prediction of the region's continued state in future time steps. Through a grouped attention (8 heads) design, the computational complexity is reduced from... Down to This approach improves efficiency while maintaining accuracy. The entire decoding process achieves joint optimization of feature fusion and prediction generation through end-to-end training. Local details of spatial features and global patterns of temporal features interact deeply in the cross-attention layer, and then resolution reduction is achieved through temporal convolution.

[0040] 1) Cross-modal interaction mechanism In this invention, the proposed cross-modal interaction mechanism aims to enhance the model's predictive ability for traffic conditions by efficiently fusing spatial and temporal features. Through this mechanism, spatial and temporal features can influence and adjust each other across multiple dimensions, thereby capturing the interactive relationships between spatiotemporal information. Specifically, the model first measures the importance of spatial and temporal features by calculating attention weights.

[0041] (12) in, This represents the fused spatiotemporal features output after cross-modal interaction at the i-th layer (or i-th stage). This represents the spatial features input to the i-th layer interaction mechanism. This represents the temporal characteristics of the input to the i-th layer interaction mechanism.

[0042] 2) Temporal downsampling convolution To control the prediction stride of the model, this invention designs a temporal downsampling convolution, which controls the prediction stride by adjusting the size of the convolution kernel in the temporal dimension. Specifically, the temporal dimension of the convolution kernel determines the range of information the model focuses on at each time step. By designing an appropriate temporal convolution kernel, the downsampling process of the input temporal data can be effectively controlled, thereby optimizing the prediction accuracy of the model at different time steps. This scheme further simplifies the computational complexity of the model by converting the input temporal length into a lower-dimensional representation, and helps the model focus on important time step information. This method ensures efficient computation without sacrificing sensitivity to key temporal information. Through this temporal downsampling technique, the model can more accurately predict future traffic conditions while avoiding noise introduced by too many time steps, thus improving the stability and reliability of the prediction. The prediction stride is controlled by designing the temporal dimension of the convolution kernel. (13) in, Indicates the prediction step size. Indicates the input length. This indicates the number of zero values ​​padded at both ends of the time dimension of the input sequence. Indicates the kernel size. This indicates the sliding step size.

[0043] This plan is set up Input timing length Convert to .

[0044] 4. Loss Function Design This model employs a composite loss function design, aiming to simultaneously optimize the semantic accuracy and spatiotemporal continuity of traffic state prediction. The loss function consists of a weighted cross-entropy loss and a spatiotemporal smoothing regularization term, and its design motivation stems from the two core challenges of traffic prediction tasks: (1) Class Imbalance Problem: In the traffic semantic map, background areas (class 0) account for more than 80%, while key states (such as red restricted areas, class 2) account for less than 6%. Directly using standard cross-entropy will cause the model to be biased towards the majority class prediction. To address this, class weight coefficients are introduced. ,in For category The total number of pixels. This weight amplifies the gradient contribution of the minority class through backpropagation, such as the weight of the red region. The accuracy of the predictions in the background area is 3.5 times that of the background area, forcing the model to focus on the prediction accuracy of key areas.

[0045] (2) Spatiotemporal continuity constraint: The evolution of traffic conditions has physical inertia. For example, the transition from a smooth flow (green) to a congested state (red) requires a gradual transition to a more congested state (yellow), rather than an abrupt leap. Congested areas usually spread along the direction of traffic flow. Therefore, a spatiotemporal smoothing regularization term is designed, which includes two constraints: 1) Time-based second-order difference constraint: (14) The acceleration abrupt change in the penalty prediction result avoids non-physical jumps (such as the second frame being all green and the third frame suddenly becoming all red).

[0046] 2) Spatial gradient constraints: (15) The loss function consists of two parts: (16) Among them, the weighting coefficient , This represents the time gradient operator.

[0047] Step 4: Model Training and Evaluation The training and evaluation phases are crucial for verifying the effectiveness and robustness of the model. Systematic iterative optimization ensures the model meets practical standards. This step first configures the training environment and hyperparameters, for example, using the Adam optimizer and a cosine annealing strategy to dynamically adjust the learning rate. Multiple rounds of training are then performed on a GPU computing platform until the loss function converges. Figure 3 As shown, the core of model optimization is a composite loss function that combines a weighted cross-entropy loss to address class imbalance with a spatiotemporal regularization term to enhance prediction smoothness. The evaluation system is designed with multi-dimensional metrics, including Accuracy and F1-score to measure overall classification accuracy, recall (Recall@Red) for key regions (such as no-go zones), MAE and RMSE to measure prediction error, and SSIM to measure structural similarity. Furthermore, detailed ablation experiments are conducted to analyze the contribution of each core module (such as gating mechanisms and attention fusion) to the final performance, and the model's sensitivity to different input lengths and prediction step sizes is tested to comprehensively evaluate its generalization ability in complex real-world scenarios.

[0048] Step 5: Predicted Output and System Integration The final step aims to transform the trained prediction model into a workable intelligent service within a real-world traffic management system, achieving a leap from theoretical research to practical application. This stage first deploys the model to a production environment, enabling it to receive real-time data streams (such as current road network status images and real-time event information) and generate online probability maps of traffic conditions for a future period. To achieve dynamic adaptation, the system can integrate a feedback mechanism to automatically adjust the size of the historical window input to the model based on the real-time traffic flow change rate. For example, it can automatically extend the historical observation window during peak periods of dramatic traffic flow changes to improve prediction stability. Figure 4 As shown, the prediction results will be integrated into the traffic control center's management platform in a visual format (such as a heat map) to provide decision-making support for control measures such as traffic light timing optimization, variable message sign content dissemination, and dynamic route guidance. To ensure the long-term effectiveness of the system, a continuous monitoring and model update mechanism needs to be established, periodically fine-tuning the model parameters with new data to adapt to the long-term evolution of the urban road network.

[0049] Example 1: Performance Comparison Experiment of S3MFNet Overall Model This embodiment aims to verify the comprehensive performance advantages of the S3MFNet model compared to existing mainstream prediction methods, and is the core experiment to prove its effectiveness. The implementation goal is to compare S3MFNet with a series of representative baseline models under a unified CCTID 2025 dataset and evaluation system, demonstrating its superiority in prediction accuracy, key region identification, and spatiotemporal consistency. Specifically, we selected eight types of models covering different technical approaches for comparison, including temporal models (such as LSTM, GRU), spatial models (3D-CNN), spatiotemporal fusion models (such as ConvLSTM, STGCN, ASTGCN), Transformer architecture (TrafficTransformer), and the latest graph model (GraphWaveNet). The experimental setup strictly followed the chronological division of the training set (March-July 2024), validation set (August-September), and test set (October-November). The input was a traffic semantic graph of the past 6 frames (60 minutes), predicting the state of the next 3 frames (30 minutes). All experiments were conducted on the same hardware platform and with the same hyperparameters. The evaluation adopts a multi-dimensional indicator system, including semantic accuracy (Accuracy, F1-score, Recall@Red), prediction error (MAE, RMSE), temporal consistency (TCI), and structural similarity (SSIM).

[0050] Table 1 Overall performance of each model on the test set Model MAE↓ RMSE↓ Accuracy↑ F1↑ Recall@Red↑ SSIM↑ LSTM 0.142±0.009 0.241±0.013 0.823 0.649 0.312 0.742 GRU 0.135±0.008 0.233±0.011 0.831 0.661 0.328 0.753 3D-CNN 0.126±0.007 0.219±0.010 0.847 0.687 0.357 0.769 ConvLSTM 0.118±0.006 0.207±0.009 0.862 0.705 0.381 0.784 STGCN 0.109±0.005 0.198±0.008 0.878 0.723 0.402 0.803 ASTGCN 0.103±0.004 0.185±0.007 0.891 0.741 0.423 0.816 TrafficTransformer 0.097±0.003 0.176±0.006 0.925 0.758 0.447 0.829 GraphWaveNet 0.093±0.003 0.168±0.005 0.911 0.772 0.468 0.838 S3MFNet (This invention) 0.082±0.002 0.149±0.004 0.934 0.813 0.532 0.862 Experimental results and analysis show that S3MFNet achieves best results across almost all metrics. Quantitative data in this experiment clearly demonstrates an overall accuracy of 93.4% and a weighted F1 score of 0.813. The most significant improvement is in recall (Recall@Red) for critical "no-go zones" (red), where S3MFNet achieves 0.532, a 13.7% improvement compared to the second-best model, GraphWaveNet's 0.468, proving its strong ability to capture sparse critical events. Simultaneously, its Temporal Consistency Index (TCI) is the lowest (0.041), indicating the smoothest prediction results and effectively mitigating the state abrupt change problem. These data collectively demonstrate that S3MFNet achieves leading overall performance through its multi-scale spatiotemporal fusion and gated residual design. Figure 5 The predictive results are visually demonstrated. This shows that the model can not only make accurate macro-level predictions, but also has a keen ability to perceive fragmented, small-scale key congestion areas, providing a reliable basis for precise traffic intervention.

[0051] Example 2: Ablation Experiment of Core Module This embodiment is used to decompose and verify the necessity and contribution of each innovative module in S3MFNet. The implementation goal is to quantitatively analyze the specific impact of each component on model performance by gradually adding core modules—gated residual blocks, cross-modal attention mechanisms, and physical constraint loss functions—and to prove the effectiveness of their collaborative work. Specifically, a simplified model with the aforementioned innovative points removed is first constructed as a baseline. Then, each module is added incrementally to form a complete model, and the performance of each variant is evaluated on a fixed test set. In the experimental results and analysis section, the ablation experiment data in the paper shows that the addition of gated residual blocks improves mIoU by 8.2%, the core of which lies in the dynamic gating function. The feature reuse ratio was effectively adjusted. While the cross-modal attention mechanism did not significantly improve overall accuracy, it significantly improved the IoU of the "forbidden region" by 6.0%, with its attention weights calculated as follows: This achieves effective alignment of spatial and temporal information. The introduction of the physical constraint loss function significantly reduces MAE (11.0%), with its key spatiotemporal second-order difference constraint term being... The unreasonable proportion of drastic state transitions was reduced from 12.7% to 4.3%. The complete model achieved optimal performance, and the contribution of each module was obtained through Shapley value decomposition, verifying the complementarity and necessity of multi-module collaborative design. Figure 6 The performance of various model variants was visually compared using radar charts. This shows that the prediction results are not only more accurate, but also more consistent with the actual propagation and dissipation patterns of traffic flow, increasing the confidence of traffic management departments in the model's output.

[0052] Example 3: Spatiotemporal Scale Sensitivity and Dynamic Adaptability Experiment This embodiment examines the resilience and adaptability of S3MFNet in different application scenarios and verifies the effectiveness of its dynamic optimization strategy. The implementation goal is to analyze the variation of model performance with input history length (T) and prediction step size (Δt), and to test whether the proposed dynamic time window strategy can improve performance and efficiency in practical applications. Specifically, the scheme includes systematically changing T and Δt to construct different prediction scenarios and implementing a dynamic time window strategy based on the rate of change of traffic. The performance of fixed and dynamic strategies was compared. Experimental results and analysis show that when the input T=6 and the prediction Δt=3, the model achieves the best balance between accuracy and stability, at which point the convolutional kernel can effectively cover a 90-minute local spatiotemporal neighborhood. However, when the prediction step size increases to Δt=6, the congestion propagation error rate rises to 21.6%, highlighting the challenge of long-term prediction. The dynamic time window strategy can intelligently adjust the input length according to traffic conditions. The results show that this strategy reduces the model's average MAE by 6.7% (from 0.082 to 0.077), while improving the inference speed by approximately 22% (from 18.2 frames / second to 22.3 frames / second), demonstrating that S3MFNet has higher practical value in dynamically changing real-world environments. The adaptive adjustment mechanism of this strategy can be intuitively represented by the following formula, which is dynamically calculated based on real-time data.

[0053] Table 2. Model performance comparison under different input / output time frame combinations

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal spatiotemporal traffic state prediction method based on semantic modeling, characterized in that, The method specifically includes the following steps: S1: Acquire multi-source data, including traffic flow, vehicle trajectory, weather and event information, and preprocess it; S2: The preprocessed data is transformed into a spatiotemporal sequence and class balancing is performed. Finally, the balanced dataset is divided into training set, validation set and test set according to the principle of non-overlapping time. S3: Construct a multimodal spatiotemporal traffic state prediction network model based on semantic modeling, abbreviated as S3MFNet model, including a spatial feature encoder, a temporal feature processor, and a spatiotemporal feature fusion module; the temporal feature processor is used to map low-dimensional temporal features to high-dimensional spatiotemporal representation; the spatiotemporal feature fusion module achieves efficient fusion and accurate prediction of spatiotemporal features through cross-modal interaction mechanism and temporal downsampling convolution; S4: The S3MFNet model is trained using a composite loss function to simultaneously optimize the semantic accuracy and spatiotemporal continuity of traffic state prediction. S5: Input the real-time traffic data stream into the trained S3MFNet model to predict the probability distribution of future traffic conditions.

2. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 1, characterized in that, In step S1, a distributed asynchronous acquisition framework is used to acquire multi-source data; the preprocessing includes: spatiotemporal registration, i.e., using an improved RANSAC algorithm to align geographic coordinate systems; image enhancement; and storage architecture, i.e., constructing time-series data cubes at hourly granularity.

3. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 1, characterized in that, Step S2 specifically includes: transforming the preprocessed data into a spatiotemporal sequence, specifically using a sliding window strategy that follows the time order to generate input-output pairs; for key categories in the traffic semantic map, using class weight adjustment based on logarithmic inverse frequency or dynamic oversampling techniques to enhance the model's learning ability for minority class samples; and dividing the dataset into training set, validation set, and test set according to the principle of non-overlapping time.

4. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 1, characterized in that, In step S3, the spatial feature encoder includes a semantic embedding layer, a multi-scale 3D convolutional module, and a gated residual block; The semantic embedding layer is used to map discrete traffic state labels to a continuous semantic vector space; The multi-scale 3D convolution module extracts features in the spatiotemporal joint domain through three-dimensional convolution kernels to simultaneously model the spatial locality and temporal continuity of traffic states. The gated residual block adjusts the feature reuse ratio through a dynamic gating function and gradient optimization mechanism.

5. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 1, characterized in that, In step S3, the processing flow of the temporal feature processor includes high-order representation learning and dynamic dimension alignment; The higher-order representation learning is achieved through nonlinear dimensionality increase and feature decoupling. The nonlinear dimensionality increase maps the original temporal features to a higher-dimensional latent space through a two-layer fully connected network to learn the higher-order patterns of traffic state evolution. The hidden layer of the feature decoupling uses ReLU activation to achieve feature sparsity, so that the contributions of different time factors can be separated. The dynamic dimension alignment achieves feature map resolution matching through trilinear interpolation; the interpolation weights are controlled by learnable parameters.

6. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 1, characterized in that, In step S3, the spatiotemporal feature fusion module specifically includes: establishing a dynamic correlation between spatial features and temporal features, and driving the generation of prediction results with physical constraints; In the cross-modal interaction mechanism, the model uses cross attention to achieve bidirectional alignment of spatial and temporal features; specifically, the feature map output by the spatial branch generates a query vector through linear transformation, while the temporal features generate key and value vectors; attention weights are calculated through normalized dot products. The temporal downsampling convolution controls the prediction stride by adjusting the size of the convolution kernel in the temporal dimension.

7. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 1, characterized in that, In step S4, the composite loss function Including the weighted cross-entropy loss and the spatiotemporal smoothing regularization term, the expression is: in, For category weight coefficients, For category The total number of pixels; These are the weighting coefficients; Let cross-entropy be the loss function. For gradient operators, For the model's predicted output, This represents the length of the time window.

8. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 7, characterized in that, In step S4, the spatiotemporal smoothing regularization term includes: The time-based second-order difference constraint is expressed as follows: in, Indicates time smoothing loss, In the future t The complete output of each time step Denotes the square of the Frobenius norm; Spatial gradient constraint, expressed as: in, Indicates spatial smoothing loss, This represents the gradient operator in the horizontal direction of space. This represents the gradient operator in the vertical direction of space.

9. The multimodal spatiotemporal traffic state prediction method based on semantic modeling according to claim 1, characterized in that, In step S4, the S3MFNet model is trained by using the Adam optimizer and a cosine annealing strategy to dynamically adjust the learning rate, and performing multiple rounds of training until the loss function converges.

Images