Traffic flow prediction method based on convolution and self-attention

By combining convolution and self-attention in the CTformer model, the problem of capturing long-term and short-term dependencies in traffic flow prediction is solved, achieving more efficient prediction results.

CN119005495APending Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411013055.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing traffic flow prediction models based on CNN and Transformer have limitations in capturing short-term and long-term dependencies. CNN has difficulty learning long-term dependencies, while Transformer requires a large amount of data and computing resources.

Method used

We employ the CTformer convolutional self-attention model, which combines convolutional operations to capture short-term dependencies in shallow layers and self-attention to capture long-term dependencies in deep layers. We use a temporal convolutional network, a location encoder, convolutional modules, fully connected layers, and a multi-head self-attention module to predict traffic flow.

Benefits of technology

It achieves a balance between computational efficiency and accuracy, effectively combining the advantages of convolution and self-attention, and improving the accuracy of traffic flow prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005495A_ABST
    Figure CN119005495A_ABST
Patent Text Reader

Abstract

The invention discloses a traffic flow prediction method based on convolution and self-attention. The method comprises the following steps: S1, data acquisition; s2, preprocessing the acquired data; s3, establishing a convolutional self-attention CTform model and training the convolutional self-attention CTform model; and S4, predicting the traffic flow. According to the traffic flow prediction method based on convolution and self-attention, short-term dependence in traffic flow data is captured in the shallow layer of the model by using convolution operation, long-term dependence in the traffic flow data is captured in the deep layer of the model by using self-attention, the advantages of the two architectures are effectively combined, and the prediction accuracy is improved. And the calculation efficiency and the precision are better balanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a traffic flow prediction method based on convolution and self-attention. Background Art

[0002] Traffic flow prediction is a fundamental problem in intelligent transportation systems and has attracted widespread attention in recent years. Early work is usually based on linear time series analysis methods. Representative traditional methods include autoregressive integrated moving average (ARIMA), filtering-based methods, support vector machines, and vector autoregression (VAR). These models have the characteristics of interpretability, but have difficulty in dealing with traffic data with complex time series dependency characteristics.

[0003] Models based on deep learning have shown their superiority in learning latent feature representations. Models based on recurrent neural networks (RNNs) summarize past states and current information for prediction, where states are recursively updated with new inputs at each time step. Models based on (CNNs) repeatedly use concatenated convolutions and activation operations to mine deep features of time series data, using multiple convolution kernels to extract features from multiple perspectives. Models based on Transformers use variants of the self-attention mechanism to extract temporal dependencies, showing strong capabilities.

[0004] Among the above methods, CNN-based and Transformer-based models have achieved great success, but they each have their own problems. For CNN-based models, they introduce inductive bias through convolution operations, which can better capture short-term dependencies, but due to the limited receptive field, it is difficult to learn long-term dependencies. For Transformer-based models, they can effectively capture long-term dependencies through the self-attention mechanism, but require a lot of data and computation to train the model. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a traffic flow prediction method based on convolution and self-attention, comprising the following steps: S1. data collection; S2. preprocessing the collected data; S3. establishing and training a convolutional self-attention CTformer model; S4. predicting traffic flow.

[0006] Furthermore, the step S1 specifically includes the following sub-steps: S11. Setting a long time interval and a short time interval; S12. Collecting traffic flow data in real time according to the short time interval; S13. Aggregating the collected traffic flow data according to the long time interval.

[0007] Furthermore, the S2 step specifically includes the following sub-steps: S21. Standardize the traffic flow data; S22. Introduce a sliding window, set the input sequence length seq_len, the start mark length label_len, the predicted sequence length pred_len and the sliding window size seq_len according to the total time step length of the data, and use the value of the next pred_len time step as the label; S23. Divide the processed data into a training set, a validation set and a test set in a ratio of 6:2:2.

[0008] Furthermore, the convolutional self-attention CTformer model in the S3 step is specifically divided into: a temporal convolutional network layer, a position encoder, a convolutional module, a fully connected layer, a self-attention module and a multi-head self-attention module.

[0009] Furthermore, the step S3 specifically includes the following sub-steps: S31. Take the training set samples, and respectively convert the encoder input sequence X and the decoder input sequence X into the convolution module and the position encoder. de Convert to a high-dimensional representation h (0) and h de (0) ; S32. Characterize h (0) Input convolution module learns short-term dependencies to obtain feature h (1) Then, the feature h (1) Input the self-attention module to learn long-term dependencies and obtain feature h (2) ; S33. Characterize h de (0) Input convolution module to obtain the sequence feature h to be predicted de (1) , the feature h de (1) As the query of the multi-head self-attention module, the feature h (2) As key and value get feature h de (2) ; S34. The feature h de (2) Input the fully connected layer to obtain the prediction result of the sample; S35. Compare the prediction result with the real data and verify it through the validation set data to determine whether the model is overfitting.

[0010] Furthermore, the encoder input sequence X in step S31 is specifically: X={x1,x2,...,x seq_len}; The decoder input sequence X de Specifically: X de ={X token ,X0}={x seq_len+1 ,...,x seq_len+token_len ,0,...,0}, where Xtoken It is the label item of time series data.

[0011] Furthermore, the step S35 specifically includes the following sub-steps: when the model is not overfitted, repeating steps S31-S34; when the model is overfitted or the number of repeated executions reaches a threshold, ending the training and saving the current model.

[0012] Furthermore, it also includes a traffic flow prediction system for implementing the method, including a traffic flow data acquisition module, a data preprocessing module, a flow prediction module, and a result display module; the traffic flow data acquisition module is used to collect traffic flow data in real time; the data preprocessing module is used to preprocess the collected traffic flow data; the flow prediction module is used to predict and analyze the traffic flow situation based on the preprocessed data; the result display module is used to output the prediction results and send them to the user.

[0013] The present invention provides a traffic flow prediction method based on convolution and self-attention, which has the following beneficial effects:

[0014] The present invention proposes a traffic flow prediction method based on convolution and self-attention. Convolution operations are used in the shallow layer of the model to capture short-term dependencies in traffic flow data, and self-attention is used in the deep layer of the model to capture long-term dependencies in traffic flow data, effectively combining the advantages of the two architectures and achieving a better balance between computational efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0016] Figure 1 A flow chart of the method provided by the present invention;

[0017] Figure 2 A schematic diagram of the model framework provided by the present invention;

[0018] Figure 3 A schematic diagram of embedding input time series data provided by the present invention. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0020] The following is a detailed description of the implementation method of the present invention in conjunction with the accompanying drawings. Only some embodiments are described, not all embodiments. For the purpose of clarity, representations and descriptions that are not related to the present invention are omitted in the drawings and descriptions.

[0021] In order to have a clearer understanding of the technical features, purposes and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below. Obviously, the implementation cases described are part of the embodiments of the present invention, not all of the embodiments, and cannot be understood as limiting the scope of the implementation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0022] like Figure 1 As shown, the present invention provides a traffic flow prediction method based on convolution and self-attention, comprising the following steps: S1. data collection; S2. preprocessing the collected data; S3. establishing and training a convolutional self-attention CTformer model; S4. predicting traffic flow.

[0023] Step S1 specifically includes the following sub-steps: S11. Setting a long time interval and a short time interval; S12. Collecting traffic flow data in real time according to the short time interval; S13. Aggregating the collected traffic flow data according to the long time interval.

[0024] Step S2 specifically includes the following sub-steps: S21. Standardize the traffic flow data; S22. Introduce a sliding window, set the input sequence length seq_len, the start mark length label_len, the prediction sequence length pred_len and the sliding window size seq_len according to the total time step length of the data, and use the value of the next pred_len time step as the label; S23. Divide the processed data into training set, validation set and test set in a ratio of 6:2:2.

[0025] The convolutional self-attention CTformer model in step S3 is specifically divided into: temporal convolutional network layer, position encoder, convolution module, fully connected layer, self-attention module and multi-head self-attention module.

[0026] Step S3 specifically includes the following sub-steps: S31. Take the training set samples, and use the convolution module and the position encoder to input the sequence X and the decoder to input the sequence X respectively. de Convert to a high-dimensional representation h (0) and h de (0) ; S32. Characterize h (0) Input convolution module learns short-term dependencies to obtain feature h (1) Then, the feature h (1)Input the self-attention module to learn long-term dependencies and obtain feature h (2) ; S33. Characterize h de (0) Input convolution module to obtain the sequence feature h to be predicted de (1) , the feature h de (1) As the query of the multi-head self-attention module, the feature h (2) As key and value get feature h de (2) ; S34. The feature h de (2) Input the fully connected layer to obtain the prediction result of the sample; S35. Compare the prediction result with the real data and verify it through the validation set data to determine whether the model is overfitting.

[0027] The encoder input sequence X in step S31 is specifically: X={x1,x2,...,x seq_len}; The decoder input sequence X de Specifically: X de ={X token ,X0}={x seq_len+1 ,...,x seq_len+token_len ,0,...,0}, where X token is the label item of the time series data; 0,...,0 is the input sequence of the predicted sequence length. Since prediction is required in the subsequent process, the initial value of the input sequence is set to 0.

[0028] Step S35 specifically includes the following sub-steps: when the model is not overfitted, repeat steps S31-S34; when the model is overfitted or the number of repeated executions reaches a threshold, end the training and save the current model.

[0029] It also includes a traffic flow prediction system for implementing the method, including a traffic flow data acquisition module, a data preprocessing module, a flow prediction module, and a result display module; the traffic flow data acquisition module is used to collect traffic flow data in real time; the data preprocessing module is used to preprocess the collected traffic flow data; the flow prediction module is used to predict and analyze the traffic flow situation based on the preprocessed data; the result display module is used to output the prediction results and send them to the user.

[0030] Embodiment: A traffic flow prediction method based on convolution and self-attention comprises the following steps:

[0031] Step 1: Data collection, collect traffic flow data using a traffic measurement system.

[0032] Traffic performance measurement system was used to collect traffic flow data in real time every 30 seconds, and the obtained traffic flow data were aggregated into 5-minute intervals.

[0033] Step 2: Data preprocessing: standardize and segment the traffic flow data.

[0034] First, the traffic flow data is z-score standardized, and the standard normalization function is: x = (x-μ) / σ; where x represents the traffic flow data sample, μ represents the sample mean, and σ represents the sample standard deviation.

[0035] Then, a sliding window is introduced to process the standardized traffic flow data. According to the total time step length of the data, the input sequence length seq_len, the start mark length label_len and the predicted sequence length pred_len are set. The sliding window size is set to seq_len, and the value of the next pred_len time step is used as the label. The step size of the sliding window is set to 1.

[0036] The processed data is divided into training set, validation set and test set in the ratio of 6:2:2.

[0037] Step 3: Model training. Based on the convolution and self-attention mechanism, the present invention proposes a convolutional self-attention network CTformer, which follows the encoder-decoder architecture and includes a convolutional layer, a position encoder, a convolutional module, a multi-head self-attention module, and a fully connected layer. The training data is input into the CTformer model for training.

[0038] Take the training sample and record the encoder input sequence as X = {x1, x2, ..., x seq_len}, the decoder input sequence is X de ={X token ,X0}={x seq_len+1 ,...,x seq_len+token_len ,0,...,0}. X and X are respectively transformed into de Convert to a high-dimensional representation h (0) and h de (0) .like Figure 2 As shown in Figure 1, the representation consists of three parts: original sequence projection, local timestamp, and global timestamp. In order to align the dimensions, a one-dimensional convolution filter is used, the convolution kernel size is set to 3, the step size is set to 1, and the original sequence is projected into a high-dimensional vector. The vector dimension is d model .

[0039] The local timestamp is calculated by the position encoder: in , pos represents the relative position in the current sample, L x Represents the total length of the time series. Each global timestamp is represented by a learnable timestamp embedding.

[0040] In the encoding part, such as Figure 3 As shown, first h (0) The input convolution module captures short-term dependencies and is calculated as follows: (1) =h (0) +Conv(h (0) ). Then the feature h (1) The input self-attention module captures long-term dependencies. Self-attention is a special implementation of the attention mechanism, where the query, key, and value are the same sequence of symbolic representations. Multi-head self-attention is the most widely adopted self-attention in practice, which can jointly attend to information from different representation subspaces.

[0041] The basic operation of multi-head self-attention is scaled dot product attention. The formula for scaled dot product attention is as follows: where Q, K, V and d model are the query, key, value and their dimensions respectively. In self-attention, all queries, keys and values ​​are the same representation sequence, i.e., Q = K = V. Multi-head self-attention first linearly projects the query, key and value into different representation subspaces, and then performs the attention function in parallel.

[0042] Finally, the outputs are concatenated and further projected to get the final output as shown below: Where W O is the output projection matrix after multi-head attention, W j is the output projection matrix of the jth head. The output of the self-attention module is output as the encoding result: h (2) =h (1) +MHA(h (1) ,h (1) ,h (1) ).

[0043] In the decoding part, first Input convolution layer to obtain the sequence features to be predicted The calculation is as follows:

[0044] Then As a query, the feature h obtained in the decoding part (2) As keys and values, we input them into the multi-head attention layer and get the output h (2) :

[0045] The decoder output Input the fully connected layer to get the prediction result of the sample.

[0046] Train the model, compare the predicted results with the true labels, and use the validation set data to verify whether the model is overfitting. Repeat the above steps until the specified round is reached or the model is found to be overfitting and stop training. Save the current model as the best model.

[0047] Step 4: Traffic flow prediction: input the traffic flow data to be tested into the trained model to make predictions for subsequent time steps.

[0048] The present invention proposes a traffic flow prediction method based on convolution and self-attention. Convolution operations are used in the shallow layer of the model to capture short-term dependencies in traffic flow data, and self-attention is used in the deep layer of the model to capture long-term dependencies in traffic flow data, effectively combining the advantages of the two architectures and achieving a better balance between computational efficiency and accuracy.

[0049] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A traffic flow prediction method based on convolution and self-attention, characterized in that: The following steps are involved: S1. Data collection; S2. Preprocess the collected data; S3. Build and train the convolutional self-attention CTformer model; S4. Predict traffic flow.

2. The traffic flow prediction method based on convolution and self-attention according to claim 1, characterized in that: The S1 step specifically includes the following sub-steps: S11. Set long time interval and short time interval; S12. Collect traffic flow data in real time at short time intervals; S13. Aggregate the collected traffic flow data at long time intervals.

3. The traffic flow prediction method based on convolution and self-attention according to claim 1, characterized in that: The S2 step specifically includes the following sub-steps: S21. Standardize traffic flow data; S22. Introduce a sliding window, set the input sequence length seq_len, the start mark length label_len, the predicted sequence length pred_len and the sliding window size seq_len according to the total time step length of the data, and use the value of the next pred_len time step as the label; S23. Divide the processed data into training set, validation set and test set in the ratio of 6:2:

2.

4. The traffic flow prediction method based on convolution and self-attention according to claim 1, characterized in that: The convolutional self-attention CTformer model in the S3 step is specifically divided into: a temporal convolutional network layer, a position encoder, a convolution module, a fully connected layer, a self-attention module and a multi-head self-attention module.

5. The traffic flow prediction method based on convolution and self-attention according to claim 3 or 4, characterized in that: The S3 step specifically includes the following sub-steps: S31. Take the training set samples, and use the convolution module and the position encoder to input the sequence X and the decoder to input the sequence X respectively. de Convert to a high-dimensional representation h (0) and h de (0) ; S32. Characterize h (0) Input convolution module learns short-term dependencies to obtain feature h (1) Then, the feature h (1) Input the self-attention module to learn long-term dependencies and obtain feature h (2) ; S33. Characterize h de (0) Input convolution module to obtain the sequence feature h to be predicted de (1) , the feature h de (1) As the query of the multi-head self-attention module, the feature h (2) As key and value get feature h de (2) ; S34. Set the feature h de (2) Input the fully connected layer to get the prediction result of the sample; S35. Compare the predicted results with the real data and verify them through the validation set data to determine whether the model is overfitting.

6. The traffic flow prediction method based on convolution and self-attention according to claim 5, characterized in that: The encoder input sequence X in step S31 is specifically: X={x1,x2,...,x seq_len }; The decoder input sequence X de Specifically: X de ={X token ,X0}={x seq_len+1 ,...,x seq_len+token_len ,0,...,0}, where X token It is the label item of time series data.

7. The traffic flow prediction method based on convolution and self-attention according to claim 5, characterized in that: The step S35 specifically includes the following sub-steps: when the model is not overfitted, repeating steps S31-S34; when the model is overfitted or the number of repeated executions reaches a threshold, ending the training and saving the current model.

8. The traffic flow prediction method based on convolution and self-attention according to claim 1, further comprising a traffic flow prediction system for implementing the method, characterized in that: It includes traffic flow data collection module, data preprocessing module, flow prediction module and result display module; The traffic flow data collection module is used to collect traffic flow data in real time; The data preprocessing module is used to preprocess the collected traffic flow data; The traffic prediction module is used to predict and analyze the traffic flow situation based on the preprocessed data; The result display module is used to output the prediction results and send them to the user.