A traffic prediction method based on multi-scale sparse and structured low-rank space-time network

Abstract

The application discloses a traffic prediction method based on a multi-scale sparse and structured low-rank space-time network, first designs a multi-scale sparse channel attention module, suppresses redundant interaction through a sparse selection strategy, and enhances the expression ability of key nodes. Subsequently, a structured low-rank self-attention module is constructed, the overall spatiality is approximately modeled by a small number of representative nodes, and the calculation bottleneck caused by the quadratic complexity is relieved. Finally, a time-dependent attention module is designed, the dot product is replaced by cosine similarity, each attention head is given a learnable parameter, different time patterns are adapted, and a local time convolution residual strategy is combined to enhance the perception ability of short-term mutation time. The application effectively copes with the two challenges of space-time redundancy and low computational efficiency in the traffic prediction method.

Description

Technical Field

[0001] This invention pertains to the field of traffic prediction technology, specifically relating to a traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks. Background Technology

[0002] Spatiotemporal forecasting aims to simultaneously utilize information from both temporal and spatial dimensions to characterize and predict the complex changes of variables in a future spatiotemporal state, and is widely applied in fields such as climate change and energy demand. Among numerous spatiotemporal forecasting tasks, traffic forecasting has received widespread attention due to its significant practical value; accurate traffic forecasting plays a crucial role in route planning and urban resource optimization. However, traffic systems possess the dual characteristics of complex temporal and spatial dependencies, and road segments are interconnected through topological dependencies. These features make traffic forecasting a highly challenging spatiotemporal forecasting problem.

[0003] Traditional methods primarily rely on statistics, such as Vector Autoregression (VAR) and ARIMA models. However, these methods are based on the assumptions of linearity and stationarity, making them unable to capture nonlinear features. Furthermore, these methods suffer from parameter inflation in large-scale road network scenarios. Subsequent machine learning methods, such as K-Nearest Neighbors (KNN) and Support Vector Machine Regression (SVR), while possessing some nonlinear modeling capabilities, heavily rely on manually constructed features and lack the ability to model road network topology and long-term time dependencies.

[0004] With the development of deep learning technology, researchers have developed more powerful prediction models. Graph Neural Networks (GNNs) have been widely introduced into traffic prediction. They explicitly model the spatial dependencies between nodes through adjacency matrices, performing feature propagation and information aggregation on the graph structure to achieve predictive results. In addition, the Transformer method, which has made progress in computer vision and natural language processing, has also been used in traffic prediction to enhance the ability to model long-distance dependencies.

[0005] Traffic prediction, as one of the most typical spatiotemporal prediction tasks, has undergone iterative evolution in prediction technology from machine learning to deep learning. Although early machine learning methods such as SVR improved the ability to model nonlinearities, these methods relied on manual feature extraction, and the researcher's prior knowledge could have a certain impact on the prediction results, thus limiting their performance.

[0006] Later, deep learning technology emerged. Early methods, such as Long Short-Term Memory (LSTM) networks, utilized gating mechanisms to capture short-term fluctuations and long-term evolution patterns in data. DCRNN used diffusing convolutions to characterize spatial dependencies in roads, then embedded them into recurrent neural networks (RNNs), combining them with GRU units for temporal dynamic modeling. STGCN used graph convolutions and temporal convolutions to characterize the topology of traffic networks and capture local temporal dependencies. Building on this, STSGCN abandoned the separate modeling of time and space, instead treating nodes at different time steps as nodes within a unified spatiotemporal graph, achieving joint modeling of time and space through synchronous convolution mechanisms. STGNCDE treated the prediction process as a continuous temporal dynamic system, using neural control differential equations to extract features during temporal evolution and combining them with GNNs to model spatial dependencies. GWNET combined graph convolutions with dilated temporal convolutions, solving the gradient vanishing problem inherent in RNNs and improving parallel computation efficiency. It further introduced a learnable adjacency matrix to automatically discover dependencies between nodes, and then expanded the temporal receptive field through dilated convolutions to improve prediction performance. STIDGCN designs a framework that interacts across two dimensions of time and space, supplemented by a data-driven dynamic graph structure to achieve spatiotemporal joint modeling. This model utilizes a traffic pattern library to capture spatial heterogeneity and a dynamic adjacency matrix to capture the relationships between nodes. LightST is a traffic prediction framework based on knowledge distillation, where a lightweight MLP learns spatiotemporal dependencies under the guidance of a pre-trained GNN.

[0007] With the successful application of Transformer models in sequence modeling, researchers have gradually introduced self-attention modules into traffic prediction tasks to enhance the spatiotemporal feature representation capabilities through global dependency modeling. GMAN employs multi-head self-attention to simultaneously capture spatial and temporal dependencies; spatial attention models the global correlation between nodes, while temporal attention characterizes long-range temporal dependencies. STAEformer enhances the Transformer's ability to model spatial structure and temporal patterns by introducing adaptive spatiotemporal embedding. PDFormer uses node relationships to generate two mask matrices to constrain the attention range and reduce interference from irrelevant nodes. STDN provides a trend-seasonal decomposition architecture that decouples the input sequence into two components: trend and season. GRU is used as the encoder, and Transformer as the decoder for modeling and prediction, respectively. Meanwhile, AutoSTF integrates GNN and Transformer into a unified search space, employing automated neural architecture search technology to adaptively search for the optimal prediction architecture in both spatiotemporal dimensions, and performs final training after determining the optimal structure. In addition, researchers explored the application of time-series prediction methods in the field of traffic prediction. For example, TimeMixer is a model based on multi-scale decomposition and mixing that does not rely on self-attention modules and uses lightweight mixing operations to achieve efficient modeling.

[0008] Currently, existing technologies mainly have the following drawbacks:

[0009] First, real-world traffic networks typically exhibit strong regional clustering characteristics. The traffic status of a particular road segment is usually influenced by its neighboring areas or functionally similar road segments, while the influence of distant nodes may be relatively weak. However, most existing Transformer-based models use fully connected self-attention modules to model the interactions of all nodes in pairs across the spatial dimension. While this dense modeling approach enhances the model's global perception capability, it inevitably introduces a large number of redundant connections, some of which are only partially relevant or even irrelevant. These redundant connections may reduce the model's ability to learn spatial features and also increase noise interference, negatively impacting the model's discriminative ability. Therefore, in real-world traffic scenarios, minimizing the interactions of redundant nodes while maintaining spatial modeling capabilities becomes a pressing issue.

[0010] Secondly, although self-attention modules are relatively effective at modeling long-distance dependencies, they still have efficiency limitations in traffic prediction. Standard self-attention modules require constructing a correlation matrix, and their time and space complexity increases quadratically. As the size of the traffic network increases, the computational complexity and memory overhead increase significantly.

[0011] Meanwhile, from a temporal modeling perspective, traditional scaled dot product attention is quite sensitive to feature magnitudes and relies on a fixed scaling factor, making it difficult to adapt to heterogeneous temporal patterns in sequences. When faced with alternating morning and evening rush hours, as well as sudden traffic congestion, relying solely on a global temporal attention module may not be sensitive enough to short-term fluctuations and may fail to fully capture their dynamic characteristics. Summary of the Invention

[0012] Despite significant progress in existing traffic prediction methods, they still face two major challenges: spatiotemporal redundancy and low computational efficiency. To address these issues, this invention proposes a traffic prediction method based on Multi-Scale Sparse and Structured Low-Rank Spatiotemporal Networks (MSSL-ST). First, a multi-scale sparse channel attention module is designed, which suppresses redundant interactions and enhances the expressive power of key nodes through a sparse selection strategy. Then, a structured low-rank self-attention module is constructed, using a small number of representative nodes to approximate the overall spatiality, alleviating the computational bottleneck caused by quadratic complexity. Finally, a time-dependent attention module is designed, replacing the dot product with cosine similarity and assigning learnable parameters to each attention head to adapt to different temporal patterns. This is further enhanced by combining a local temporal convolutional residual strategy to improve the ability to perceive short-term abrupt changes.

[0013] The specific plan is as follows:

[0014] A traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks includes the following steps:

[0015] Step 1: Historical data is collected from traffic sensors to form an input sequence. The input data features include state information such as traffic flow and speed. This data is then fed into an embedding layer to extract feature information. The embedding layer includes numerical embedding, time period embedding, location embedding, and spatial feature embedding. First, the raw data is mapped to a high-dimensional space through a linear layer. Then, the sequential relationship of the time series is modeled through time-location encoding. Simultaneously, daily and weekly periodic information is embedded into the feature representation by combining contextual information. Furthermore, a spatial structure embedding is constructed using a graph Laplacian matrix to reflect the topological relationships between nodes. Through these methods, a unified spatiotemporal embedding representation can be obtained.

[0016] Step 2: The spatiotemporal features after the embedding layer are input into the spatiotemporal encoder, which includes multi-scale sparse channel attention, structured low-rank self-attention and temporal dependency attention modules to model the temporal and spatial dependencies in the data.

[0017] Step 3: The multi-scale sparse attention module filters out nodes with strong correlations, reducing redundant computation of weakly correlated or even uncorrelated nodes; the structured low-rank self-attention module uses the Nystrom approximation principle to replace the global model with a small number of landmarks, thus reducing computational complexity; the time-dependent attention module captures the dynamic changes of traffic flow over time, assigns learnable parameters to each attention head, and also incorporates a temporal convolutional residual module, which is beneficial to the model's ability to judge abnormal events.

[0018] Step 4: Concatenate and fuse the output features of the three attention modules to generate a new spatiotemporal feature representation through linear mapping. Then, stack the spatiotemporal encoders in multiple layers, preserving the original feature information through residual connections during each encoding layer.

[0019] Step 5: Input the spatiotemporal features obtained through the multi-layer encoder into the prediction layer, perform end-to-end mapping from historical sequences to future sequences, and form the prediction result.

[0020] Step 6: Evaluate the model performance using quantitative metrics. Here, three evaluation metrics are used: mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square error (RMSE). The gradient descent method is used to update the model parameters, so that the model can continuously optimize its predictive performance.

[0021] The beneficial effects of this invention are as follows: MSSL-ST achieves optimal prediction performance on multiple real traffic datasets, significantly reduces MAE, MAPE and RMSE indices, and demonstrates strong robustness and generalization ability to different data distributions. Attached Figure Description

[0022] Figure 1 This is a framework diagram of the MSSL-ST invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the technical solutions of this invention will be described in detail below with reference to specific embodiments.

[0024] Step 1: Historical data is collected from traffic sensors to form an input sequence. The input data features include state information such as traffic flow and speed. This data is then fed into an embedding layer to extract feature information. The embedding layer includes numerical embedding, time period embedding, location embedding, and spatial feature embedding. First, the raw data is mapped to a high-dimensional space through a linear layer. Then, the sequential relationship of the time series is modeled through time-location encoding. Simultaneously, daily and weekly periodic information is embedded into the feature representation by combining contextual information. Furthermore, a spatial structure embedding is constructed using a graph Laplacian matrix to reflect the topological relationships between nodes. Through these methods, a unified spatiotemporal embedding representation can be obtained.

[0025] Step 2: The spatiotemporal features after the embedding layer are input into the spatiotemporal encoder, which contains three modules: multi-scale sparse channel attention, structured low-rank self-attention, and temporal dependent attention, to model the temporal and spatial dependencies in the data.

[0026] Step 3: The multi-scale sparse attention module filters out nodes with strong correlations, reducing redundant computation of weakly correlated or even uncorrelated nodes; the structured low-rank self-attention module uses the Nystrom approximation principle to replace the global model with a small number of road signs, modeling the overall space and reducing computational complexity; the time-dependent attention module captures the dynamic changes of traffic flow in time, assigns learnable parameters to each attention head, and also combines a temporal convolutional residual module, which is beneficial to the model's ability to judge abnormal events.

[0027] Step 4: Concatenate and fuse the output features of the three attention branches to generate a new spatiotemporal feature representation through linear mapping. Then, stack the spatiotemporal encoder in multiple layers, preserving the original feature information through residual connections during each encoding layer.

[0028] Step 5: Input the spatiotemporal features obtained through the multi-layer encoder into the prediction layer, perform end-to-end mapping from historical sequences to future sequences, and form the prediction result.

[0029] Step 6: Evaluate the model performance using quantitative metrics. Here, three evaluation metrics are used: mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square error (RMSE). The gradient descent method is used to update the model parameters, so that the model can continuously optimize its predictive performance.

[0030] The data embedding layer transforms the original low-dimensional data into a high-dimensional representation tensor, containing temporal periodic information, spatial structure information, and sequence position information; assuming the input tensor is... Where B is the batch size, N is the number of nodes, T is the historical time step, and F represents the original feature dimension;

[0031] First, input tensors The original input is transformed through a linear layer. Where D is the embedding dimension; then, two types of time period embeddings, daily and weekly, are introduced, denoted as , respectively. and The diurnal embedding maps the relative position of each time step within a day to discrete indices and transforms them into a high-dimensional vector representation using a learnable embedding matrix. Weekly embedding maps the week category to which a time step belongs to into an index form, which is also a way to generate the corresponding vector representation through embedding lookup. Next, absolute position coding is introduced. By combining sine and cosine functions of different frequencies, a unique position coding vector is generated for each time step. ;

[0032] In addition, spatial features need to be extracted; for a given adjacency matrix A, its normalized graph Laplacian matrix is ​​calculated, and the top k smallest nontrivial eigenvectors are selected through eigenvalue decomposition to generate the spatial graph Laplacian embedding. ;

[0033] Finally, summing all the above embeddings yields the output of the embedding layer:

[0034]

[0035] Multi-scale sparse channel attention: This module employs a multi-scale sparse module in spatial interaction, removing a large number of irrelevant node relationships and retaining those with strong dependencies. Then, it fuses these relationships using multiple sets of different sparsity ratios to highlight key nodes. Finally, it performs dimensionality transformation through linear projection. The specific operations are as follows:

[0036] 1. Node meshing and reorganization: The input tensor is First, reorganize the node dimensions into a two-dimensional grid structure, and rearrange the N nodes into one. The grid follows the selection of Q. When the mesh size exceeds the number of nodes, zero-padding is performed along the node dimensions. After this reorganization process, a two-dimensional feature map representation is obtained. .

[0037] 2. Data Encoding: Perform contextual encoding along the feature dimensions, first using... Convolution is used to perform linear projection, generating the corresponding query, key, and value matrix:

[0038]

[0039] in It is a learnable matrix. Then we introduce a... Depthwise convolutions enhance local structure awareness without significantly increasing computational cost:

[0040]

[0041] in express convolution.

[0042] 3. Multi-head segmentation and spatial flattening: Next, the feature dimension is divided into H subspaces according to the multi-head mechanism, and the dimension of each subspace is... After multi-head partitioning and spatial flattening, the tensor is reshaped into... ,in This indicates the number of grid positions after flattening.

[0043] 4. Construct a similarity matrix between each query Q and key K: retain only the top k most relevant scores in each row of the similarity matrix, and elements with low mask weights; this invention includes a mechanism to automatically adjust the sparsity. It is a dynamically changing value within a ratio range; then, multiple branches with different ratios are constructed in parallel, i.e., multiple sets of sparse attention matrices are obtained under different sparsity settings, which are used to respectively... Perform weighted aggregation:

[0044]

[0045] The Softmax operation is used to normalize the attention scores into a probability distribution. The sparsification operator is defined as follows:

[0046]

[0047] in Represents the first element in the similarity matrix. Line number Column elements, Represents the similarity matrix of the th element. The k-th maximum value in the row is found, and the probability of the remaining elements smaller than the k-th maximum value is set to 0.

[0048] 5. Multi-branch fusion: Linear weighted fusion of multi-path sparse aggregation results is performed using learnable weight coefficients. Let... This represents the aggregation result at the r-th sparsity ratio. For the corresponding learnable fusion weights. First, calculate the weighted combination of the multi-branch sparse representations, then use... Convolutional projection, followed by dimensionality reduction using linear layers to produce the final output:

[0049]

[0050] in, It is a set of sparsity ratios. and They represent Convolution and linear projection.

[0051] Structured Low-Rank Self-Attention: This module reduces the computational complexity from quadratic to linear by selecting a small number of representative landmarks to perform a low-rank approximation of the original attention matrix. It constructs landmark features using a piecewise averaging strategy, then calculates three small-scale similarity matrices based on the original features and the landmark features, and efficiently obtains the pseudo-inverse using a high-order matrix iteration method, ultimately achieving lightweight self-attention aggregation. The specific operations are as follows:

[0052] 1. Construction of Qs, Ks, and Vs matrices: At each time step t, linear projection is used to obtain the corresponding query, key, and value matrices:

[0053]

[0054] in It is a learnable matrix. This represents a slice of the input tensor at time step t.

[0055] 2. Construction of landmark points: The standard self-attention matrix is ​​defined as follows: The approximation principle involves selecting M representative landmarks and decomposing the original matrix S into three smaller matrices, which are then multiplied to construct a low-rank approximation. The dimensions of the three smaller matrices are... The approximate format is

[0056]

[0057] in , Represents the feature matrix of road signs. It is the Moore-Penrose pseudo-inverse operation. It is the scaling factor. This represents the transpose operation; as mentioned above, the low-rank approximation constructs landmark points. Specifically, the sequence of length N is divided into M segments, each segment having a length of... Then, the landmark features are obtained by averaging within each segment:

[0058]

[0059] Where i represents the index of the original word in the input sequence, and j represents the index of the landmark.

[0060] 3. Construction of the three similarity matrices: Based on the landmark features, the following similarity matrices are constructed:

[0061]

[0062] in Let Q-landmarkK, landmarkQ-landmarkK, and landmarkQ-K matrices represent these respectively. Next, the softmax function is used to normalize each row.

[0063]

[0064] 4. Higher-order iterative pseudo-inverse approximation: For a matrix Construct the following iterative sequence:

[0065]

[0066] in Represents the identity matrix. It is the matrix after the k-th iteration, and its initialization is defined as follows: , These are the coefficients. To ensure third-order convergence, the parameter configuration used is... Its approximation error satisfies After several iterations, a stable approximation can be obtained. Therefore, the self-attention output format is: ,Right now

[0067]

[0068] 5. Local Information Enhancement: To compensate for the loss of local spatial information in global attention, depthwise separable convolutions are introduced as residual connections. These convolutions slide only along the node dimensions, and each attention head processes information independently.

[0069]

[0070] in Represent a Depth convolution, It is a configurable hyperparameter.

[0071] Temporally Dependent Attention: This module models temporal dependencies in data. Addressing the sensitivity of traditional dot-product attention to data scale, this module first performs L2 normalization on the query and key, then calculates attention weights using cosine similarity. To accommodate the varying attention sharpness requirements of different temporal patterns, the module introduces a learnable parameter to adaptively adjust the concentration of attention distribution for each attention head. After obtaining the global attention output, the module further models the temporal dimension locally using depthwise convolution and sums the results as residuals. The specific operations are as follows:

[0072] 1.Q t K t Vt Matrix generation: First, obtain the query, key, and value matrix of node N through linear projection:

[0073]

[0074] in It is a learnable weight matrix. Indicates the output channel dimension. Then... Divided into multiple heads The dimension of each head is set to .

[0075] 2. Normalization: Unlike traditional temporal self-attention modules, traffic data often exhibits amplitude variations, which can interfere with pattern recognition. The normalization process may alter the original data distribution, making dot-product attention sensitive to scale differences. Therefore, cosine similarity is used to focus on changing trends and relative patterns, and the query and key matrices are normalized to generate new representations.

[0076]

[0077] in Indicates along the last dimension Calculated Norm.

[0078] 3. Calculate cosine attention: A learnable parameter Θ is used to adaptively adjust the sharpness of the attention distribution, enabling different temporal patterns to learn different attention granularities. Attention weights are calculated using cosine similarity.

[0079]

[0080] in, It is a learnable parameter. It involves element-wise multiplication, and the softmax function normalizes along the time dimension T.

[0081] 4. Multi-head attention concatenation and aggregation: Aggregate the corresponding value matrix and concatenate the outputs of all attention heads:

[0082]

[0083] Where h is the number of headers, and Concat represents the concatenation operation.

[0084] 5. Local Dynamic Enhancement: To prevent the global temporal attention from being insufficiently sensitive to short-term changes such as sudden road congestion, a local dynamic enhancement module is introduced. The final output generated by the time-dependent attention block is represented as follows:

[0085]

[0086] in It is the GELU activation function. express Depthwise convolution.

[0087] Spatiotemporal Fusion Layer

[0088] In the above process, three attention modules were used. Next, a fusion strategy was employed to integrate the outputs of the three attention branches into a unified multi-head attention module, jointly capturing spatial and temporal information. The fused representation is as follows:

[0089]

[0090] in It is a learnable output projection matrix. Let represent the outputs of multi-scale sparse channel attention, structured low-rank self-attention, and time-dependent attention, respectively. The number of heads in each of the three attention modules is denoted as . The output dimension of each attention branch is set to... These features are then concatenated to form a D-dimensional feature representation. Finally, a fully connected feedforward network is used after the multi-head attention block to obtain the final output. ,in It is a fully connected feedforward network.

[0091] Prediction layer

[0092] After each spatiotemporal encoder layer, its output First, project the convolutional projection onto the dimension. Then, the projected features from different layers are accumulated across layers:

[0093]

[0094] Where L represents the number of encoder layers. After the skip connections, two consecutive 1×1 convolutional layers are used for prediction. The first convolution removes the temporal dimension from the input window. Transform to prediction window Meanwhile, the second convolution is responsible for connecting the skip connections along the channel dimension. Projecting onto the output variable dimension enables multi-step prediction and completes an end-to-end mapping from historical sequences to future sequences:

[0095]

[0096] The performance of the MSSL-ST invention was evaluated on three real public transportation datasets: PEMS04, PEMS08, and PEMS_BAY. These datasets were obtained from the California Department of Transportation Performance Measurement System (PeMS) and were sampled at 5-minute intervals, containing 12 time points per hour.

[0097] This invention selected 10 baseline models and conducted experiments on the PEMS04, PEMS08, and PEMS_BAY datasets. The comparison models were divided into the following categories: VAR, SVR, and LSTM are based on traditional statistics, machine learning, and recurrent neural networks, respectively; DCRNN, STGCN, GWNET, STSGCN, STGNCDE, STIDGCN, and LightST are based on GNNs; GMAN, STAFormer, and PDFormer are based on attention; STDN and AutoSTF are hybrid methods; and TimeMixer is based on a multilayer perceptron.

[0098] The dataset is divided into training, validation, and test sets in a 6:2:2 ratio. Historical traffic flow data from the previous hour is used to predict traffic flow or speed for the next hour. Therefore, both the input and output sequences are set to a length of 12. Before training, all input data are standardized using Z-score normalization.

[0099] All experiments were conducted on a Tesla P100-SXM2 GPU, implemented using PyTorch 1.12.1 and Python 3.9.12. The encoder layers for PEMS04, PEMS08, and PEMS_BAY were 6, 4, and 4, respectively, with batch sizes of 16, 16, and 8, and a hidden dimension of 64 for all models. The AdamW optimizer was used to train the models with a learning rate of 0.001 and 100 training epochs. The prediction performance is shown in the table below:

[0100] Table predicts performance effect

[0101]

[0102] As shown in the table above, the MSSL-ST proposed in this invention achieves near-optimal performance on all three datasets, demonstrating a significant performance advantage. Compared to the suboptimal model, MSSL-ST reduces MAE, MAPE, and RMSE by 2.36%, 0.22%, and 0.35% on PEMS08, and by 1.22%, 2.66%, and 0.80% on PEMS_BAY, respectively. Furthermore, on PEMS04, MAE and MAPE are reduced by 0.71% and 2.69%, respectively.

[0103] Traffic data typically exhibits both complex spatial topology and dynamic temporal evolution patterns, with different datasets showing significant differences in network size, correlation strength, and fluctuation characteristics. For the PEMS04 dataset, which has a relatively complex spatial structure and uneven distribution of correlations among nodes, MSSL-ST adaptively selects highly correlated nodes and effectively suppresses redundant connections. Therefore, on this dataset, MAPE further reduces the error by 2.69% compared to suboptimal methods, demonstrating the advantages of sparse selection mechanisms in complex topological environments. Meanwhile, the low-rank approximation reduces computational redundancy while maintaining the ability to model global dependencies, thereby enhancing generalization ability and achieving stable error reduction across multiple datasets.

[0104] Furthermore, for the PEMS08 dataset, which exhibits more pronounced periodic patterns and sudden fluctuations, the local dynamic enhancement mechanism in the TDA module strengthens the modeling ability for short-term changes. Therefore, MSSL-ST achieves a 2.36% reduction in MAE and a 0.35% reduction in RMSE compared to the suboptimal method, indicating improvements in both overall error control and extremum suppression. Moreover, through optimization of the overall spatiotemporal modeling framework, MSSL-ST achieves a 2.66% relative improvement in MAPE on the PEMS_BAY dataset. This model enhances the stability of feature representations while maintaining global dependency modeling capabilities, reducing redundant information propagation and local noise interference, and achieving more stable and accurate predictions in low-flow intervals. These results demonstrate that the proposed MSSL-ST maintains strong robustness and generalization ability under different data distribution conditions.

[0105] The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks, comprising the following steps: Step 1: Historical data is collected by traffic sensors to form an input sequence. The input data features include the status information of traffic data. Then, this data is sent to the embedding layer to extract feature information. The embedding layer includes numerical embedding, time period embedding, location embedding and spatial feature embedding. First, the raw data is mapped to a high-dimensional space through a linear layer. Then, the sequential relationship of the time series is modeled through time and location encoding. At the same time, combined with context information, daily and weekly period information is embedded into the feature representation. Furthermore, spatial structure embedding is constructed using the Graph Laplace matrix to reflect the topological relationships between nodes; Step 2: The spatiotemporal features after the embedding layer are input into the spatiotemporal encoder, which includes multi-scale sparse channel attention, structured low-rank self-attention and temporal dependency attention modules to model the temporal and spatial dependencies in the data. Step 3: The multi-scale sparse attention module filters out nodes with strong correlations, reducing redundant computation of weakly correlated or even uncorrelated nodes; the structured low-rank self-attention module uses the Nystrom approximation principle to replace the global model with a small number of road signs; the time-dependent attention module captures the dynamic changes of traffic flow in time, assigns learnable parameters to each attention head, and also combines the temporal convolutional residual module to enhance the ability to perceive short-term abrupt events. Step 4: Concatenate and fuse the output features of the three attention modules to generate a new spatiotemporal feature representation through linear mapping; then stack the spatiotemporal encoder in multiple layers, and preserve the original feature information through residual connections during each layer encoding process; Step 5: Input the spatiotemporal features obtained through the multi-layer encoder into the prediction layer to perform end-to-end mapping from historical sequences to future sequences, thereby forming the prediction result; Step 6: Evaluate the model performance using quantitative indicators, including three evaluation metrics: mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square error (RMSE). Then, use the gradient descent method to update the model parameters, so that the model can continuously optimize its predictive performance.

2. The traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks according to claim 1, characterized in that: The data embedding layer transforms the original low-dimensional data into a high-dimensional representation tensor, containing temporal periodic information, spatial structure information, and sequence position information; assuming the input tensor is... Where B is the batch size, N is the number of nodes, T is the historical time step, and F represents the original feature dimension; First, input tensors The original input is transformed through a linear layer. Where D is the embedding dimension; then, two types of time period embeddings, daily and weekly, are introduced, denoted as , respectively. and The diurnal embedding maps the relative position of each time step within a day to discrete indices and transforms them into a high-dimensional vector representation using a learnable embedding matrix. Weekly embedding maps the week category to which a time step belongs to into an index form, which is also a way to generate the corresponding vector representation through embedding lookup. Next, absolute position coding is introduced. By combining sine and cosine functions of different frequencies, a unique position coding vector is generated for each time step. ; In addition, spatial features need to be extracted; for a given adjacency matrix A, its normalized graph Laplacian matrix is ​​calculated, and the top k smallest nontrivial eigenvectors are selected through eigenvalue decomposition to generate the spatial graph Laplacian embedding. ; Finally, summing all the above embeddings yields the output of the embedding layer: 。 3. The traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks according to claim 2, characterized in that: The multi-scale sparse attention module employs a multi-scale sparse module in spatial interaction, removing a large number of irrelevant node relationships and retaining those with strong dependencies. Then, it fuses multiple sets of nodes with different sparsity ratios to highlight key nodes. Finally, it performs dimensionality transformation through linear projection. The specific operations are as follows: (1) Node meshing and reorganization: The input tensor is First, reorganize the node dimensions into a two-dimensional grid structure, and rearrange the N nodes into one. The grid follows the selection of Q. ; When the mesh size exceeds the number of nodes, zero-padding is performed along the node dimensions; after this reorganization process, a two-dimensional feature map representation is obtained. ; (2) Encoding the data: Perform context encoding along the feature dimensions, first using Convolution is used to perform linear projection, generating the corresponding query, key, and value matrix: in It is a learnable matrix; then a... Depthwise convolutions enhance local structure awareness without significantly increasing computational cost: in express convolution; (3) Multi-head segmentation and spatial flattening: Next, the feature dimensions are divided into H subspaces according to the multi-head mechanism, and the dimension of each subspace is... After multi-head partitioning and spatial flattening, the tensor is reshaped into... ,in Indicates the number of grid positions after flattening; (4) Construct a similarity matrix between each query Q and key K: retain only the top k most relevant scores in each row of the similarity matrix, and elements with low mask weights; set up a parameter that can automatically adjust the sparsity. It is a value that changes dynamically within a range of ratios; Multiple branches with different ratios were then constructed in parallel, resulting in multiple sets of sparse attention matrices under different sparsity settings, which were used to respectively... Perform weighted aggregation: The Softmax operation is used to normalize the attention scores into a probability distribution. The sparsification operator is defined as follows: in Represents the first element in the similarity matrix. Line number Column elements, Represents the similarity matrix of the th element. The k-th maximum value in the row is found, and the probability of the remaining elements smaller than the k-th maximum value is set to 0. (5) Multi-branch fusion: Linear weighted fusion of multi-path sparse aggregation results is performed using learnable weight coefficients; let This represents the aggregation result at the r-th sparsity ratio. For the corresponding learnable fusion weights; First, calculate the weighted combination of the multi-branch sparse representation, then use... Convolutional projection, followed by dimensionality reduction using linear layers to produce the final output: in, It is a set of sparsity ratios. and They represent Convolution and linear projection.

4. The traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks according to claim 3, characterized in that: The structured low-rank self-attention module reduces the computational complexity from quadratic to linear by selecting a small number of representative landmarks to perform a low-rank approximation of the original attention matrix. It constructs landmark features using a piecewise averaging strategy, then calculates three small-scale similarity matrices based on the original features and the landmark features, and efficiently obtains the pseudo-inverse using a high-order matrix iteration method, ultimately achieving lightweight self-attention aggregation. The specific operations are as follows: (1) Q s K s V s Matrix construction: At each time step t, use linear projection to obtain the corresponding query, key, and value matrix: in It is a learnable matrix. This represents a slice of the input tensor at time step t; (2) Construction of landmark points: The standard self-attention matrix is ​​defined as follows: The approximation principle involves selecting M representative landmarks and decomposing the original matrix S into three smaller matrices, which are then multiplied to construct a low-rank approximation. The dimensions of the three smaller matrices are... The approximate format is in , Represents the feature matrix of road signs. It is the Moore-Penrose pseudo-inverse operation. It is the scaling factor. Indicates the transpose operation; As mentioned above, the low-rank approximation requires constructing landmarks; specifically, it divides a sequence of length N into M segments, each segment having a length of... Then, the landmark features are obtained by averaging within each segment: Where i represents the index of the original word in the input sequence, and j represents the index of the landmark; (3) Construction of the three similarity matrices: Based on the landmark features, the following similarity matrices are constructed: in Let Q-landmarkK, landmarkQ-landmarkK, and landmarkQ-K be the landmark matrices, respectively; then, the softmax function is used to normalize each row: (4) Higher-order iterative pseudo-inverse approximation: for matrix Construct the following iterative sequence: in Represents the identity matrix. It is the matrix after the k-th iteration, and its initialization is defined as follows: , These are the coefficients; to ensure third-order convergence, the parameter configuration used is as follows: Its approximation error satisfies After several iterations, a stable approximation is obtained. ; Therefore, the self-attention output format is ,Right now (5) Local information enhancement: To compensate for the loss of local spatial information in global attention, depthwise separable convolutions are introduced as residual connections; these convolutions slide only along the node dimension, and each attention head processes information independently. in Represent a Depth convolution, It is a configurable hyperparameter.

5. The traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks according to claim 4, characterized in that: The time-dependent attention module is used to model time dependencies in the data. To address the sensitivity of traditional dot-product attention to data scale, L2 normalization is first applied to the query and the A key, followed by cosine similarity calculation of attention weights. To adapt to the varying attention sharpness requirements of different time patterns, a learnable parameter is introduced to adaptively adjust the concentration of attention distribution for each attention head. After obtaining the global attention output, a local model of the time dimension is further performed using depthwise convolution, and the results are superimposed as residuals. The specific operations are as follows: (1) Q t K t V t Matrix generation: First, obtain the query, key, and value matrix of node N through linear projection: in It is a learnable weight matrix. Indicate the output channel dimension; then... Divided into multiple heads The dimension of each head is set to ; (2) Normalization: Cosine similarity is used to focus on trends and relative patterns, and the query and key matrices are normalized to generate new representations. in Indicates along the last dimension Calculated Norm; (3) Calculate cosine attention: A learnable parameter Θ is used to adaptively adjust the sharpness of the attention distribution, enabling different temporal patterns to learn different attention granularities; attention weights are calculated using cosine similarity with an adaptive scaling parameter: in, It is a learnable parameter. It involves element-wise multiplication, and the softmax function normalizes along the time dimension T. (4) Multi-head attention concatenation and aggregation: Aggregate the corresponding value matrix and concatenate the outputs of all attention heads: Where h is the number of headers, and Concat represents the concatenation operation; (5) Local dynamic enhancement: A local dynamic enhancement module is introduced; the final output generated by the time-dependent attention block is represented as: in It is the GELU activation function. express Depthwise convolution.

6. The traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks according to claim 5, characterized in that: A fusion strategy is employed to integrate the outputs of the three attention modules into a unified multi-head attention module, jointly capturing spatial and temporal information; the fused representation is as follows: in It is a learnable output projection matrix. Let represent the outputs of multi-scale sparse channel attention, structured low-rank self-attention, and time-dependent attention, respectively; and denote the number of heads of the three attention modules as . The output dimension of each attention branch is set to... The features are then concatenated to form a D-dimensional feature representation; finally, a fully connected feedforward network is used after the multi-head attention block to obtain the final output. ,in It is a fully connected feedforward network.

7. The traffic prediction method based on multi-scale sparse and structured low-rank spatiotemporal networks according to claim 6, characterized in that: After each spatiotemporal encoder layer, its output First, project the convolutional projection onto the dimension. Then, the projected features from different layers are accumulated across layers: Where L represents the number of encoder layers; after the skip connections, two consecutive 1×1 convolutional layers are used for prediction; the first convolution removes the temporal dimension from the input window. Transform to prediction window Meanwhile, the second convolution is responsible for connecting the skip connections along the channel dimension. Projecting onto the output variable dimension enables multi-step prediction and completes an end-to-end mapping from historical sequences to future sequences: 。

Images