A traffic flow prediction method based on a sparse attention mechanism

Abstract

The application belongs to the technical field of traffic flow prediction, and particularly relates to a traffic flow prediction method based on a sparse attention mechanism; the method comprises the following steps: obtaining historical traffic flow data and preprocessing the same to obtain standard traffic data; processing the standard traffic network data by using a feature embedding module to obtain traffic embedding features; inputting the traffic embedding features into a sparse attention module for processing to obtain traffic attention features; processing the traffic attention features by using a representation fine-tuning module to obtain accurate traffic features; inputting the accurate traffic flow features into a regression layer for processing to obtain a traffic flow prediction result; and the application can effectively capture the potential spatiotemporal correlation of non-adjacent nodes and relieve the performance degradation caused by the homogenization of node features in a deep attention network, thereby improving the accuracy of traffic flow prediction.

Description

Technical Field

[0001] This invention belongs to the field of traffic flow prediction technology, specifically relating to a traffic flow prediction method based on a sparse attention mechanism. Background Technology

[0002] With the acceleration of urbanization and the continuous increase in traffic flow, traffic flow prediction has become an important research topic in traffic management and control. Accurate traffic flow prediction not only helps improve the efficiency of the transportation system but also effectively alleviates traffic congestion, improves the scheduling efficiency of public transportation, and even reduces the risk of traffic accidents. Currently, there are various research methods for traffic flow prediction, mainly including statistical time series methods, machine learning-based prediction models, and deep learning methods that have received widespread attention in recent years.

[0003] Traditional traffic flow prediction methods, such as autoregressive integral moving average models and support vector machines, can provide reliable predictions to a certain extent. However, because these methods largely rely on historical traffic flow data and lack in-depth analysis of the complex features within the data, they often fail to fully capture the nonlinear characteristics of traffic flow changes, resulting in significant limitations in prediction accuracy. In recent years, deep learning-based traffic flow prediction methods have gradually become mainstream, with methods such as long short-term memory networks and convolutional neural networks achieving good results in many studies. However, although deep learning methods can capture temporal relationships in time series data well, they still have insufficient ability to focus on important information, especially in complex traffic systems where traffic flow changes at different times and locations often have complex interdependencies.

[0004] Therefore, how to further improve the ability of traditional deep learning models to extract key features from traffic flow data, especially their adaptability to different spatiotemporal features, is a key problem that current traffic flow prediction technology urgently needs to solve. Attention mechanisms, as a technique that enables models to focus on important parts of the data, have been widely used in natural language processing, image recognition, and other fields in recent years, achieving significant results. By introducing attention mechanisms, the model's ability to focus on traffic flow information at key moments and locations can be effectively improved, thereby achieving more accurate predictions.

[0005] Existing attention mechanisms and deep neural networks suffer from the following drawbacks: Current global attention mechanisms easily introduce interactions between numerous irrelevant nodes, increasing redundant computation, and noise is difficult to suppress effectively, preventing the model from focusing on capturing important traffic patterns. Although some methods have attempted to introduce spatiotemporal attention mechanisms, they have not completely solved the problem of ineffective interactions between low-relevance nodes. As the number of layers in a deep neural network increases, node features gradually converge, causing an "oversmoothing" phenomenon. This means that deep models cannot effectively distinguish the features of different nodes, thus affecting the ability to capture spatiotemporal features and leading to a decline in prediction performance.

[0006] In summary, there is an urgent need for a new traffic flow prediction method that can effectively capture the potential spatiotemporal correlations of non-adjacent nodes and alleviate the performance degradation caused by the homogenization of node features in deep attention networks, thereby improving the accuracy of traffic flow prediction. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a traffic flow prediction method based on a sparse attention mechanism. The method includes: acquiring historical traffic flow data and preprocessing it, inputting the preprocessed data into a trained traffic flow prediction model, and obtaining traffic flow prediction results.

[0008] The training process for a traffic flow prediction model includes:

[0009] S1: Acquire historical traffic flow data and preprocess it to obtain standard traffic data;

[0010] S2: The standard traffic network data is processed using a feature embedding module to obtain traffic embedding features;

[0011] S3: Input the traffic embedding features into the sparse attention module for processing to obtain traffic attention features;

[0012] S4: The traffic attention features are processed using the representation fine-tuning module to obtain accurate traffic features;

[0013] S5: Input the precise traffic flow characteristics into the regression layer for processing to obtain the traffic flow prediction results;

[0014] S6: Calculate the total model loss and adjust the model parameters based on the total model loss to obtain the trained traffic flow prediction model.

[0015] Preferably, the process of processing historical traffic flow data includes:

[0016] Historical traffic flow data is divided using a sliding window with a window size of 12.

[0017] The segmented historical traffic flow data is standardized to obtain standard traffic data.

[0018] Preferably, the process of processing standard traffic network data using a feature embedding module includes:

[0019] A fully connected layer is used to map standard traffic network data to a high-dimensional feature space to obtain high-dimensional features;

[0020] The time slices of each sample data are mapped according to the weekly and daily cycles to obtain the weekly and daily time embedding features.

[0021] By concatenating the high-dimensional features, the weekly time-period embedding features, and the daily time-period embedding features, a complete embedding feature is obtained.

[0022] The SENet mechanism is used to process the complete embedding features to obtain traffic embedding features.

[0023] Preferably, the process of obtaining traffic attention features includes:

[0024] The traffic embedding features are patched to obtain the input features; different linear projections are then applied to the input features to obtain the query matrix, key matrix, and value matrix.

[0025] Multiply the query matrix by the transpose of the key matrix to obtain the sparse association matrix between nodes;

[0026] The key matrix and value matrix are processed based on the sparse correlation matrix to obtain a new key matrix and a new value matrix;

[0027] Attention is calculated based on the query matrix, new key matrix, and new value matrix to obtain initial attention features;

[0028] The initial attention features are residually connected to the input features and then subjected to layer normalization to obtain traffic attention features.

[0029] Furthermore, the formulas for obtaining the new key matrix and the new value matrix are expressed as follows:

[0030] I r =Topkindex(A r )

[0031] K g =gather(K r I r ),V g =gather(V r I r )

[0032] Among them, I r Denotes the index matrix, Kg V represents the new bond matrix. g Let A represent the new value matrix. r This represents a sparse correlation matrix, where Topkindex() represents selecting the K most relevant tokens, K r V represents the original bond matrix. r Represents the original value matrix, and gather() represents the expression based on the index matrix I. r It aggregates and extracts the K most relevant token information for each token in terms of key or value.

[0033] Preferably, the process of processing traffic attention features using the characterization fine-tuning module includes:

[0034] Learnable adaptive node embeddings are introduced and concatenated with traffic attention features to obtain concatenated features;

[0035] A multilayer perceptron module is used to optimize the stitched features to obtain accurate traffic features.

[0036] Furthermore, a multilayer perceptron module is used to optimize the representation of the stitched features as follows:

[0037] Z2=MLP2(σ(MLP1(Z1)))+Z1

[0038] Where Z2 represents precise traffic features, MLP2 represents the second layer multilayer perceptron, MLP1 represents the first layer multilayer perceptron, σ represents the activation function, and Z1 represents the stitched features.

[0039] Preferably, the regression layer processes the precise traffic flow characteristics as follows:

[0040] y = W·reshape(Z2) + b

[0041] Where y represents the traffic flow prediction result, W represents the weight matrix, Z2 represents the precise traffic features, b represents the bias, and reshape() represents the shape adjustment operation.

[0042] The beneficial effects of this invention are as follows:

[0043] This invention addresses the complex spatiotemporal dependencies in traffic flow prediction by employing a sparse spatiotemporal attention mechanism. This mechanism effectively avoids redundant information and computational waste associated with global attention mechanisms by dynamically selecting the most relevant interaction information between nodes. Unlike traditional global attention mechanisms, the sparse attention strategy of this invention improves computational efficiency, thereby enhancing the accuracy and robustness of traffic flow prediction.

[0044] This invention introduces a representation fine-tuning module, which combines adaptive node embedding and residual gating strategies to alleviate the problem of over-smoothing features in deep networks. Traditional deep networks tend to lead to excessive homogenization of node features, affecting the effective transmission of information and pattern capture. The representation fine-tuning module effectively maintains the diversity of node features by dynamically adjusting node embeddings, thereby improving the model's adaptability and predictive performance in complex spatiotemporal traffic scenarios. Attached Figure Description

[0045] Figure 1 This is a diagram showing the overall structure of the traffic flow prediction model in this invention;

[0046] Figure 2 This is a schematic diagram of the sparse attention module structure in this invention;

[0047] Figure 3 This is a schematic diagram of the structure of the fine-tuning module in this invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] This invention proposes a traffic flow prediction method based on a sparse attention mechanism, such as... Figure 1 As shown, the method includes the following:

[0050] Historical traffic flow data is acquired and preprocessed. The preprocessed data is then input into a trained traffic flow prediction model to obtain traffic flow prediction results.

[0051] The training process for a traffic flow prediction model includes:

[0052] S1: Obtain historical traffic flow data and preprocess it to obtain standard traffic data.

[0053] Preferably, public datasets such as PeMS04, PeMS07, PeMS08, and PeMS-BAY can be used for training, and the training set can be divided into training set and test set according to the corresponding protocol.

[0054] The historical traffic flow data in the training set is preprocessed as follows: a sliding window method is used for partitioning, with a historical window size of 12 and a prediction window size of 12. Simultaneously, the traffic flow characteristics are standardized using standard deviation to obtain standardized traffic data, as shown in the following formula:

[0055]

[0056] Where μ is the mean of the feature and σ is the standard deviation.

[0057] S2: The standard traffic network data is processed using the feature embedding module to obtain traffic embedding features.

[0058] The process of processing standard traffic network data using a feature embedding module includes:

[0059] A fully connected layer is used to map standard traffic network data to a high-dimensional feature space, resulting in high-dimensional feature E. f .

[0060] For each sample data time slice, map it according to the weekly and daily cycles respectively, and then concatenate them to obtain the weekly time embedding feature E. w and daily periodic time embedding features E d .

[0061] By concatenating the high-dimensional feature, the weekly time-period embedding feature, and the daily time-period embedding feature, the complete embedding feature E is obtained, denoted as:

[0062] E = E f ||E w ||E d

[0063] To further improve the effectiveness of the embedding representation, the SENet mechanism is used for dynamic weight learning, outputting traffic embedding features X. in SENet generates weight coefficients for each channel through global average pooling and fully connected layers, thereby adaptively adjusting the influence of different features, suppressing noise information, and enhancing the representation of important features.

[0064] S3: Input the traffic embedding features into the sparse attention module for processing to obtain traffic attention features.

[0065] like Figure 2 As shown, the process of inputting traffic embedding features into the sparse attention module for processing includes:

[0066] Traffic embedding features X in The Patch operation aggregates information originally scattered across different time steps and nodes into several Tokens. Each Token not only contains the static feature information of the node but also incorporates the dynamic changes within the historical time window, resulting in the input feature X. t ∈R Nt×C The input features are mapped to the query matrix Q through linear projection. r ∈R Nt×d Key matrix Kr ∈R Nt×d Value matrix V r ∈R Nt×d , where d is the projection dimension, N represents the number of traffic nodes, and t represents the number of time slices.

[0067] Multiplying the query matrix by the transpose of the key matrix yields the sparse association matrix between nodes. Specifically:

[0068] Unlike conventional methods that directly use global self-attention mechanisms between nodes, this approach considers the local correlations between road network nodes and treats the features of a node's historical time window as a token, which is then used in Q... r and K r Dynamically derive the sparse correlation matrix A between tokens by matrix multiplication between their transposes. r :

[0069] A r =Q r (K r ) T

[0070] Sparse correlation matrix A r To measure the relevance of nodes within a historical time window, and to reduce low-relevance interactions, attention is calculated for each token by selecting the K most relevant nodes based on the relevance matrix. Specifically: First, the key matrix and value matrix are processed according to the sparse relevance matrix to obtain a new key matrix and a new value matrix, as expressed by the following formula:

[0071] I r =Topkindex(A r )

[0072] K g =gather(K r I r ),V g =gather(V r I r )

[0073] Among them, I r Denotes the index matrix, K g V represents the new bond matrix. g Let A represent the new value matrix. r This represents a sparse correlation matrix, where Topkindex() represents selecting the K most relevant tokens, K r V represents the original bond matrix. r This represents the original value matrix, and gather() represents the expression based on I. rThe index information recorded in the database aggregates and extracts the K most relevant token information for each token on the key or value.

[0074] Then, attention is calculated based on the query matrix, new key matrix, and new value matrix to obtain the initial attention features:

[0075] O = Attention(Q) r K g V g )

[0076] The initial attention features are residually concatenated with the traffic embedding features, followed by layer normalization to preserve the original information and enhance training stability, resulting in the traffic attention features, represented as:

[0077] h = LayerNorm(O+X) in )

[0078] S4: The traffic attention features are processed using the representation fine-tuning module to obtain accurate traffic features.

[0079] like Figure 3 As shown, the process of processing traffic attention features using the representation fine-tuning module includes:

[0080] A learnable adaptive node embedding p is introduced and concatenated with traffic attention features to obtain the concatenated feature Z1, which is represented as:

[0081] Z1=h||p

[0082] In this way, the model can adaptively adjust the node representation based on the embedding characteristics of each layer, avoiding over-smoothing caused by deep networks.

[0083] After incorporating adaptive embedding, a Multilayer Perceptron (MLP) module is used to further optimize the features. Specifically, to effectively preserve the original information and alleviate the gradient vanishing problem in deep networks, residual connections are added to the MLP. The embedded features processed by the MLP are added to the original features to obtain a new feature representation, i.e., the accurate traffic features, expressed by the formula:

[0084] Z2=MLP2(σ(MLP1(Z1)))+Z1

[0085] Where Z2 represents precise traffic features, MLP2 represents the second-layer multilayer perceptron, MLP1 represents the first-layer multilayer perceptron, and σ represents the activation function.

[0086] S5: Input the precise traffic flow characteristics into the regression layer for processing to obtain the traffic flow prediction results.

[0087] After optimization by the multilayer perceptron, the model finally outputs the prediction result through the regression layer. The specific formula is as follows:

[0088] y = W·reshape(Z2) + b

[0089] Where y represents the traffic flow prediction result, W represents the weight matrix, Z2 represents the precise traffic features, b represents the bias, and reshape() represents the shape adjustment operation, which is used to adjust the shape of the features to fit the regression output layer.

[0090] S6: Calculate the total model loss and adjust the model parameters based on the total model loss to obtain the trained traffic flow prediction model.

[0091] In some preferred embodiments of the present invention, HuberLoss is used as the loss function to train the traffic flow prediction model, and then the PeMS dataset is used to verify the prediction performance of the model. Mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean square error (RMSE) are used as evaluation metrics; lower values ​​for all three indicate higher model accuracy.

[0092] In some preferred embodiments of this invention, the invention employs the Python programming language, enabling it to run on mainstream computer platforms. This embodiment uses Ubuntu 20.04 as the operating system, requires at least 16GB of RAM, at least 32GB of hard disk space, and an NVIDIA A100 GPU with 40GB of video memory. The invention is implemented using the PyTorch 1.13.1 framework, employing the Adam optimization algorithm to update model parameters. The initial learning rate is set to 0.001, and a MultiStepLR decay strategy is used, decaying at a rate of 0.1. The batch size during training is set to 16, and during testing, it is set to 64.

[0093] The training datasets used were PeMS04, PeMS07, PeMS08, and PeMS-BAY. These datasets cover subsets of different time periods and regions, including data reflecting busy road sections during morning and evening rush hours, as well as subsets showcasing traffic flow characteristics during off-peak hours and in different geographical locations. This diverse data source ensures the model's adaptability and predictive performance under different traffic scenarios. One pass of the gradient descent algorithm on all training data is called one epoch. Each epoch updates the model's parameters, with a maximum of 300 epochs. The learning rate is updated every 20 epochs. During the 300 epochs of training, model parameters are saved whenever better results are achieved.

[0094] After obtaining the trained traffic flow prediction model, historical traffic flow data is acquired and preprocessed. The preprocessed data is then input into the trained traffic flow prediction model to obtain the traffic flow prediction results.

[0095] In summary, this invention maps raw traffic flow data to a high-dimensional feature space through a fully connected layer and constructs temporal embeddings by combining periodic features. These features are then concatenated to form a complete embedding representation, enriching spatiotemporal information. SENet is used to dynamically learn feature weights, improving the expression of important features and suppressing noise. To capture the spatiotemporal correlations between non-adjacent nodes, this invention introduces a sparse spatiotemporal attention mechanism. This mechanism avoids redundant computation by dynamically selecting key node interactions, while effectively capturing cross-regional spatiotemporal dependencies. To alleviate the problem of homogenization of node features in deep networks, this invention designs a representation fine-tuning module. Through adaptive node embedding and residual gating strategies, node features are dynamically optimized to ensure that the features of each node in the deep network remain independent, thereby improving prediction performance.

[0096] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A traffic flow prediction method based on a sparse attention mechanism, characterized in that, include: Historical traffic flow data is acquired and preprocessed. The preprocessed data is then input into a trained traffic flow prediction model to obtain traffic flow prediction results. The training process for a traffic flow prediction model includes: S1: Acquire historical traffic flow data and preprocess it to obtain standard traffic network data; S2: The standard traffic network data is processed using a feature embedding module to obtain traffic embedding features; S3: Input the traffic embedding features into the sparse attention module for processing to obtain traffic attention features; the process of obtaining traffic attention features includes: The traffic embedding features are patched to obtain the input features; different linear projections are then applied to the input features to obtain the query matrix, key matrix, and value matrix. Multiply the query matrix by the transpose of the key matrix to obtain the sparse association matrix between nodes; The key matrix and value matrix are processed based on the sparse correlation matrix to obtain a new key matrix and a new value matrix; the formulas for obtaining the new key matrix and the new value matrix are expressed as follows: ； ； in, Represents an index matrix. Represents the new bond matrix, Represents the new value matrix, Represents the sparse correlation matrix. This means selecting the K most relevant tokens. Represents the original bond matrix, Represents the original value matrix, Indicates based on the index matrix Aggregate and extract the information of the K most relevant tokens for each token in terms of key or value; Attention is calculated based on the query matrix, new key matrix, and new value matrix to obtain initial attention features; The initial attention features are residually connected to the input features and then subjected to layer normalization to obtain traffic attention features. S4: The traffic attention features are processed using a representation fine-tuning module to obtain accurate traffic flow features; the process of processing traffic attention features using the representation fine-tuning module includes: Learnable adaptive node embeddings are introduced and concatenated with traffic attention features to obtain concatenated features; A multilayer perceptron module is used to optimize the stitched features to obtain accurate traffic flow features; the optimized stitched features using the multilayer perceptron module are represented as follows: ； in, Indicates precise traffic flow characteristics. This represents the second layer of the multilayer perceptron. This represents the first layer of a multilayer perceptron. This represents the activation function. Indicates splicing characteristics; S5: Input the precise traffic flow characteristics into the regression layer for processing to obtain the traffic flow prediction results; S6: Calculate the total model loss and adjust the model parameters based on the total model loss to obtain the trained traffic flow prediction model.

2. The traffic flow prediction method based on sparse attention mechanism according to claim 1, characterized in that, The process of preprocessing historical traffic flow data includes: Historical traffic flow data is divided using a sliding window with a window size of 12. The segmented historical traffic flow data is standardized to obtain standard traffic network data.

3. The traffic flow prediction method based on sparse attention mechanism according to claim 1, characterized in that, The process of processing standard traffic network data using a feature embedding module includes: A fully connected layer is used to map standard traffic network data to a high-dimensional feature space to obtain high-dimensional features; The time slices of each sample data are mapped according to the weekly and daily cycles to obtain the weekly and daily time embedding features. By concatenating the high-dimensional features, the weekly time-period embedding features, and the daily time-period embedding features, a complete embedding feature is obtained. The SENet mechanism is used to process the complete embedding features to obtain traffic embedding features.

4. The traffic flow prediction method based on sparse attention mechanism according to claim 1, characterized in that, The regression layer processes precise traffic flow characteristics as follows: ； in, This indicates the traffic flow forecast results. Represents the weight matrix. Indicates precise traffic flow characteristics. Indicates bias. This indicates a shape adjustment operation.

Images