Traffic Flow Prediction Method Based on Long-Short-Term Transformer Model
Through the method based on the long-term and short-term Transformer model, combined with long-term and short-term attention mechanisms, the long-term and short-term dependencies are explicitly modeled, and the problem of difficult to capture the medium-term and short-term dependencies of existing traffic flow prediction is solved, and more accurate traffic flow prediction is achieved.
Patent Information
- Application Number
- CN202211076654.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-05
- Publication Date
- 2025-08-05
- Estimated Expiration
- 2042-09-05
AI Technical Summary
Existing deep learning-based traffic flow prediction methods are difficult to capture both long-term and short-term dependencies, resulting in inaccurate predictions, especially the existing Transformer model is insensitive to local information.
Using a method based on the long-term and short-term Transformer model, multi-angle features are extracted through position coding and time coding, combined with long-term and short-term attention mechanisms, long-term and short-term dependencies are explicitly modeled, and adaptive block loss function is used for training to capture long-term and short-term features.
It improves the accuracy of traffic flow prediction, can better capture long-term and short-term changes, achieve more accurate predictions, and has good interpretability and computing efficiency.
Smart Images
Figure CN115545152B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of traffic information processing, and particularly relates to a traffic flow prediction method based on a long short-term Transformer model. Background Art
[0002] Traffic flow prediction is crucial for traffic network management, urban public safety, and the daily life of the public, and is an important part of the Intelligent Traffic System (ITS) field. How to utilize massive traffic data to analyze and predict dynamic traffic conditions is of great significance for the planning and construction of smart city roads and traffic management in the new era.
[0003] In recent years, deep learning models have achieved superior performance in the field of traffic prediction due to their ability to model complex non-linear dependencies. Existing deep learning-based prediction methods can be divided into three categories: models based on Recurrent Neural Network (RNN), models based on Temporal Convolutional Network (TCN), and models based on Transformer. Methods based on recurrent neural networks have problems of gradient disappearance and gradient explosion, so it is difficult to capture long-term dependencies. Methods based on temporal convolutional networks need to stack multiple layers to achieve a larger receptive field range. For these two types of methods, the number of operations required to associate two elements increases with the increase of the time distance. Different from them, methods based on Transformer models can directly model the relationship between any pair of elements without considering the distance, can better capture long-term dependencies, and show great potential in prediction tasks.
[0004] However, although the Transformer model has advantages in capturing long-term dependencies, only a few works have paid attention to its problem of being insensitive to local information. Real-world traffic flow data is usually a mixture of long-term change patterns and short-term change patterns. For example, the long-term pattern reflects the difference between weekdays and weekends, and the short-term pattern reflects the contrast between morning rush hours. Considering modeling the characteristics of the two patterns can help achieve more accurate predictions. Summary of the Invention
[0005] The object of the present invention is to solve the above-mentioned defects of traffic flow prediction problems in the prior art, and provide a traffic flow prediction method based on a long-short-term Transformer model, which comprehensively considers the temporal dependence from the long-term mode and the short-term mode, explicitly models the long-term dependence and the short-term dependence relationship, uses the long-term attention mechanism to capture the long-term change pattern, and further uses the short-term attention mechanism to flexibly capture different short-term change patterns, so as to achieve more accurate traffic flow prediction.
[0006] The object of the present invention can be achieved by adopting the following technical solutions:
[0007] A traffic flow prediction method based on a long-short-term Transformer model, the traffic flow prediction method includes the following steps:
[0008] S1. Obtain traffic flow data and perform standardized preprocessing;
[0009] S2. Perform positional encoding and temporal encoding on the traffic flow data to extract multi-angle feature representations;
[0010] S3. Input the multi-angle feature representations into the long-short-term Transformer model, and the long-short-term Transformer model includes a long-short-term attention layer, a feed-forward network layer, a residual connection layer, a normalization layer, and a fully connected layer connected in sequence, and obtain the predicted result of the output;
[0011] S4. According to the predicted result of the output and the true result of the data, with the goal of minimizing the loss function, perform iterative training and update the parameters;
[0012] S5. Use the trained long-short-term Transformer model to obtain the traffic flow prediction result on the test set.
[0013] Furthermore, encode the traffic flow data to extract the temporal information and positional information of the data. In step S2, for the traffic flow data X 1:T ={x1, x2, …, x T},X 1:T is simply referred to as X, where represents the traffic flow at time T, T represents the length of the sequence data, and perform encoding Embed(·) to obtain multi-angle feature representations. The encoding process is as follows:
[0014] X emb =Embed(X)=E pos (X)+E temporal (X)
[0015] is the multi-angle feature representation of the traffic flow sequence X, d modelis the dimension of the coding space, E pos (·) and E temporal (·) represent position encoding and time encoding respectively;
[0016] The position encoding E pos (·) adopts sine and cosine position encoding as follows:
[0017]
[0018] E pos (X) (p,d) represents the position feature of traffic flow data X at position p in the time dimension and position d in the space dimension, represents rounding down;
[0019] The time encoding E temporal (·) uses a fully connected network to extract features from traffic flow data X as follows:
[0020] E temporal (X) = W temporal X + b temporal
[0021] where W temporal is the weight matrix of the fully connected network, and b temporal is the bias term.
[0022] Furthermore, the time series data in the real world usually contains long-term change patterns and short-term change patterns. The step S3 explicitly models the long-term and short-term change patterns as follows:
[0023] S3.1. Input features from multiple perspectives into the long-short-term attention layer. The long-short-term attention layer includes parallel long-term attention mechanism and short-term attention mechanism. Use the long-term attention mechanism to capture the global long-term feature M long of the traffic flow data. At the same time, use the short-term attention mechanism to extract the local short-term feature M short of the traffic flow data;
[0024] S3.2. Fuse the above global long-term feature and local short-term feature to obtain the long-short-term feature M long-short , and the calculation formula is as follows:
[0025]
[0026] where is an affine transformation composed of trainable parameters, and Concat(·) is the concatenation operation between vectors along the feature dimension;
[0027] S3.3. The long-short-term feature M long-shortThrough a sequentially connected feed-forward network layer, residual connection layer, normalization layer, and fully connected layer, the process is as follows:
[0028] FFN(M long-short ) = max(0, M long-short W1 + b1)W2 + b2
[0029] W1 and W2 are the weight matrices of the feed-forward network layer, b1 and b2 are the bias terms, and max(·) is the operation of taking the maximum element in the elements;
[0030] H = LayerNorm(FFN(M long-short ) + M long-short )
[0031] H is the hidden layer feature, LayerNorm(·) represents the normalization operation along the feature dimension, and the element-wise addition in the normalization layer represents the residual connection layer;
[0032] The hidden layer feature passes through the fully connected layer to generate the prediction sequence, and the formula is as follows:
[0033]
[0034] represents the prediction sequence for the next H steps, W pre is the weight matrix of the fully connected layer, b pre is the bias term.
[0035] Furthermore, in step S3.1, the long-term attention mechanism is used to capture the global long-term feature M long of the traffic flow data. By considering the global long-term feature, the prediction accuracy can be improved, and the process is as follows:
[0036] First, the multi-angle feature representation X emb of the traffic flow data X is mapped to Q, K, and V, as follows:
[0037] Q = W q X emb , K = W k X emb , V = W v X emb
[0038] Among them [[ID=6३]] respectively represent the query, key, and value in the attention mechanism, are the learnable first, second, and third weight matrices for implementing the mapping, d q , d k , d v are respectively W q , W k , Wv Dimension, d model is the dimension of the coding space;
[0039] The correlation between pairwise time steps is calculated using the dot product, and the calculation formula is as follows:
[0040]
[0041] e is the attention score matrix;
[0042] Then, the softmax function is applied to each element of the attention score matrix e to obtain the normalized attention score as the attention weight, and the calculation is as follows:
[0043]
[0044] where α ij represents the attention weight between time step i and time step j, A long is the attention weight matrix, and e ij represents the attention score between time step i and time step j;
[0045] The global long-term feature M is obtained by weighting V using the attention weight matrix A long , and the formula is as follows: long , the formula is as follows:
[0046]
[0047] Furthermore, in step S3.1, a short-term attention mechanism is used to extract the local short-term feature M of traffic flow data short . The short-term attention mechanism first performs chunking and then executes the attention mechanism within the subsequence. It is a way to explicitly model temporal dependencies and can clearly discover short-term patterns. The process is as follows:
[0048] First, a learnable prototype block vector is defined to represent the high-level feature at the block level, with a length of N seg and a dimension of d s , represents the block vectors of the 1st, 2nd,..., N seg th chunks. The prototype block vector P is randomly initialized and updated along with the network parameters;
[0049] An adaptive chunking loss is introduced. This adaptive chunking loss is based on the dynamic time warping (DTW) distance, which is used to measure the similarity between two sequences of unequal lengths. The DTW distance with a smoothing factor γ > 0 is used, and the definition is as follows:
[0050]
[0051] $e'$ is the natural constant, and $B$ represents the alignment matrix of $P$ and $Q$. $\mathcal{B}$ represents the set of alignment matrices $B$, $C$ represents the cost matrix of the alignment of $P$ and $Q$, and $\langle B,C\rangle$ represents the inner product of matrices $B$ and $C$.
[0052] By minimizing the DTW distance, the optimal alignment matrix is obtained, and the calculation is as follows:
[0053]
[0054] $B'$ is the optimal alignment matrix, and $\text{argmin}$ represents the optimal alignment matrix that minimizes $\text{DTW}(P,Q)$ in the set of alignment matrices. γ (P,Q) at the minimum value.
[0055] According to the optimal alignment matrix, $Q$, $K$, and $V$ are partitioned as follows:
[0056]
[0057] where $s = \{1, 2, \ldots, N\}$, $\text{Seg}(Q,B')$ is the partitioning operation that uses the starting positions of the non-zero values in each row of seg as the splitting points, thereby partitioning $Q$ into $N$ non-overlapping subsequences. Each subsequence has a different length. $Q_s$ seg is the $s$-th subsequence of $Q$ with length $L_s$. Similarly, $K$ and $V$ also undergo the partitioning operation. $K_s$ s is the $s$-th subsequence of $K$, and $V_s$ s is the $s$-th subsequence of $V$. s s
[0058] Then, through the attention mechanism, the short-term features of different subsequences are extracted, and the calculation is as follows:
[0059]
[0060] $e$ s represents the attention score matrix of the $s$-th subsequence.
[0061] The normalized exponential function is applied to each element of $e$ s to obtain the normalized attention scores as attention weights, as follows:
[0062]
[0063] represents the attention weight between time step $m$ and time step $n$ within the $s$-th subsequence. represents the attention weight matrix of the $s$-th subsequence. represents the attention score between time step $m$ and time step $n$ within the $s$-th subsequence.
[0064] Using the attention weight matrix weight V s to obtain the short-term features, the formula is as follows:
[0065]
[0066] where s = {1, 2,..., N seg}, represents the short-term features within the s-th subsequence, and splice the short-term features within the N seg subsequences to recombine into a complete sequence to obtain the local short-term feature M short :
[0067]
[0068] is the splicing operation of vectors along the time dimension.
[0069] Furthermore, in step S4, the DTW distance is introduced to measure the similarity between two unequal-length sequences, and the block result can be adaptively obtained according to the data. The process is as follows:
[0070] Use the DTW distance as the adaptive block loss, as follows:
[0071] L proto (p) = DTW(p, Q)
[0072] Define another loss function as the mean square error loss between the predicted value and the true value. The calculation formula is as follows:
[0073]
[0074] X T+1:T+H and respectively represent the true sequence and the predicted sequence in the future H steps;
[0075] The loss function L for iterative training is composed of the adaptive block loss and the shape constraint loss. The calculation formula is as follows:
[0076]
[0077] F is the size of the batch during batch training, λ represents the weight of the loss term, X f represents the input sequence of the f-th sample during batch training, and P f represents the prototype block vector of the f-th sample. Use the backpropagation algorithm to minimize the loss function L to achieve the optimum.
[0078] The present invention has the following advantages and effects compared with the prior art:
[0079] 1. For the long-term and short-term temporal dependencies in time series data, the present invention uses a short-term attention mechanism to capture the local feature information and different short-term patterns of the time series, and a long-term attention mechanism to capture long-term patterns. For traffic prediction problems, by combining the long-term feature representation and short-term feature representation of the time series, the long-short-term Transformer model proposed by the present invention can improve the prediction accuracy and achieve better results than conventional RNNs and Transformers.
[0080] 2. The short-term attention mechanism first performs chunking and then executes the attention mechanism within the subsequences. It is an explicit way to model temporal dependencies, which can clearly discover short-term patterns, effectively extract short-term feature representations, and has good interpretability.
[0081] 3. The purpose of chunking is to discover different short-term time patterns. Therefore, the length of each optimal sub-chunk is not the same as that of other chunks in most cases. Different from directly dividing the sequence into equal-length subsequences or convolution operations that destroy the sequence shape, the adaptive chunking method in the present invention is more flexible. By defining a learnable prototype chunk vector and introducing an adaptive chunking loss, the sequence can be dynamically divided into subsequences of different lengths according to short-term features. Moreover, the calculation of subsequences is processed in parallel, so the computational complexity is not increased. BRIEF DESCRIPTION OF THE DRAWINGS
[0082] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:
[0083] Figure 1 is a schematic flowchart of the traffic flow prediction method based on the long-short-term Transformer model in the present invention;
[0084] Figure 2 is a schematic diagram of the long-short-term attention layer in the long-short-term Transformer model of the present invention;
[0085] Figure 3 is a traffic flow curve graph of the Traffic dataset within a two-week period in the present invention;
[0086] Figure 4 is an example diagram of the adaptive chunking method in the present invention for chunking the sequence on the Traffic dataset. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.
[0088] Embodiment 1
[0089] This embodiment takes the traffic flow data recorded by sensors on the highways in the San Francisco Bay Area from 2015 to 2016 as an example. This data contains the records of 862 sensors, with a recording frequency of 1 hour and a sequence length of 17,544. In this example, a multi-step prediction task is performed. We set the size of the time window to 72, and the prediction target is to predict the future traffic flow of all sensors for 24 steps through 48-step historical observations. Each traffic flow sequence with a length of 48 and a dimension of 862 is input into the model for training, and the model that obtains the optimal result on the validation set is tested on the test set to obtain the traffic flow prediction result.
[0090] Figure 1 is the flowchart of the present invention. As Figure 1 shown, the traffic flow prediction method based on the long short-term Transformer model includes the following steps:
[0091] S1. Obtain the traffic flow sequence and perform standardized preprocessing on it;
[0092] Use a sliding window to divide the long sequence data into time slices of a fixed length. We predict the future traffic flow data based on the historical observations within the slices. For a traffic flow sequence X 1:48 ={x1, x2, …, x 48}, abbreviated as X, at time t = {1, 2, …, 48}, the traffic flow observations of all sensors are represented as 1:48 The traffic flow sequence of each sensor for the next H steps is X ={x 49:72 49 , x 50 , …, x 72}.
[0093] S2. Sequence encoding representation, perform positional encoding and temporal encoding on the traffic flow sequence to obtain multi-angle feature representations;
[0094] In specific applications, as Figure 1 shown, use positional encoding E pos , temporal encoding E temporal Extract the location features and time features of the traffic flow sequence through the encoding method, and then obtain the multi-angle feature representation of the traffic flow sequence X:
[0095] X emb = Embed(X) = E pos (X) + E temporal (X)
[0096] The dimension size of the encoding space is d model , which is set to 512 dimensions here. The position encoding E pos (·) adopts the sine and cosine position encoding functions, as follows:
[0097]
[0098] E pos (X) (p,d) represents the position feature of the traffic flow sequence X at position p in the time dimension and position d in the spatial dimension, represents rounding down;
[0099] The time encoding E temporal (·) uses a fully connected network to extract features from the traffic flow sequence X, as follows:
[0100] E temporal (X) = W temporal X + b temporal
[0101] where W temporal is the weight matrix of the fully connected network, and b temporal is the bias term.
[0102] S3. Input the multi-angle feature representation of the traffic flow sequence into the long-short-term Transformer model. The long-short-term attention model includes long-short-term attention layers, feed-forward network layers, residual connection layers, normalization layers, and fully connected layers connected in sequence;
[0103] S3.1. In specific applications, as Figure 2 shown, the multi-angle features are input into the long-short-term attention layer. The long-short-term attention layer includes parallel long-term attention mechanisms and short-term attention mechanisms. Use the long-term attention mechanism to capture the global long-term features M long of the traffic flow data. At the same time, use the short-term attention mechanism to extract the local short-term features M short of the traffic flow data;
[0104] S3.2. Fuse the above global long-term features and local short-term features to obtain the long-short-term features M long-short , and the calculation formula is as follows:
[0105]
[0106] Among them is an affine transformation composed of trainable parameters, and Concat(·) is an operation of concatenating vectors along the feature dimension;
[0107] S3.3. The long-term and short-term features pass through the sequentially connected feed-forward network layer, residual connection layer, normalization layer, and fully connected layer. The process is as follows:
[0108] In specific applications, the feed-forward network layer consists of two sequentially connected linear transformations. The process is as follows:
[0109] FFN(M long-short ) = max(0, M long-short W1 + b1)W2 + b2
[0110] W1 and W2 are the weight matrices of the feed-forward network layer, b1 and b2 are the bias terms, and max(·) is an operation to take the maximum element in the elements;
[0111] H = LayerNorm(FFN(M long-short ) + M long-short )
[0112] H is the hidden layer feature, LayerNorm(·) represents the normalization operation along the feature dimension, and the element-by-element addition in the normalization layer represents the residual connection layer;
[0113] The hidden layer feature passes through the fully connected layer to generate the prediction sequence. The formula is as follows:
[0114]
[0115] represents the prediction sequence for the next 24 steps, W pre is the weight matrix of the fully connected layer, b pre is the bias term.
[0116] Furthermore, in step S3.1, the long-term attention mechanism is used to capture the global long-term feature M long of the traffic flow data. The process is as follows:
[0117] First, the multi-angle feature representation X emb of the traffic flow data X is mapped to Q, K, and V as follows:
[0118] Q = W q X emb , K = W k X emb , V = W v X emb
[0119] in Represent the query, key and value in the attention mechanism respectively, A learnable weight matrix to achieve the mapping;
[0120] The dot product is used to calculate the correlation between two time steps. The calculation formula is as follows:
[0121]
[0122] e is the attention score matrix;
[0123] The normalized exponential function is then applied to each element of e to obtain the normalized attention score, which is used as the attention weight, calculated as follows:
[0124]
[0125] where α ij represents the attention weight between time step i and time step j, A long is the attention weight matrix, e ij represents the attention score between time step i and time step j;
[0126] Use the attention weight matrix to weight V and obtain the global long-term feature M long , the formula is as follows:
[0127]
[0128] Furthermore, in step S3.1, the short-term attention mechanism is used to extract the local short-term features M of the traffic flow data. short , the process is as follows:
[0129] Define a learnable prototype block vector Represents high-level features at the block level, Indicates the 1st, 2nd, ..., Nth seg The block vectors of the blocks are divided into blocks. In this embodiment, the length of the block vectors is set to N. seg =6, dimension d s =512, randomly initialize the prototype block vector P and update it together with the network parameters;
[0130] Adaptive block loss is introduced. This loss is based on the dynamic time warping (DTW) distance and is used to measure the similarity between two sequences of unequal lengths. The DTW distance with a smoothing factor γ>0 is used and is defined as follows:
[0131]
[0132] e′ is a natural constant, B represents the alignment matrix of P and Q, Let \(B\) denote the set of alignment matrices, \(C\) denote the cost matrix for the alignment of \(P\) and \(Q\), and \(\langle B,C\rangle\) denote the inner product of matrices \(B\) and \(C\);
[0133] By minimizing the DTW distance, the optimal alignment matrix is obtained and calculated as follows:
[0134]
[0135] Let \(B'\) be the optimal alignment matrix, and \(\text{argmin}\) represents the optimal alignment matrix that minimizes DTW γ (P,Q) among the set of alignment matrices;
[0136] According to the optimal alignment matrix, \(Q\), \(K\), and \(V\) are partitioned as follows:
[0137]
[0138] where \(s = \{1,2,\ldots,6\}\), \(\text{Seg}(Q,B')\) is the partitioning operation that uses the starting positions of the non - zero values in each row of as the segmentation points, thereby partitioning \(Q\) into 6 non - overlapping subsequences, each with a different length. Let \(Q s be the \(s\) - th subsequence of \(Q\), with length \(L s , and similarly, \(K\) and \(V\) also undergo the partitioning operation. Let \(k s be the \(s\) - th subsequence of \(k\), and \(V s be the \(s\) - th subsequence of \(V\).
[0139] Then, through the attention mechanism, the short - term features of different subsequences are extracted and calculated as follows:
[0140]
[0141] Let \(e s represent the attention score matrix for the \(s\) - th subsequence;
[0142] Apply the softmax function to each element of \(e s to obtain the normalized attention scores as attention weights as follows:
[0143]
[0144] Let \(\alpha mn represent the attention weight between time step \(m\) and time step \(n\) within the \(s\) - th subsequence, represent the attention weight matrix for the \(s\) - th subsequence, represent the attention score between time step \(m\) and time step \(n\) within the \(s\) - th subsequence;
[0145] Use the attention weight matrix to weight \(V sPerform weighting to obtain short-term features, with the formula as follows:
[0146]
[0147] where s = {1, 2, …, 6}, represents the short-term feature within the s-th subsequence. Concatenate the short-term features within the 6 subsequences and recombine them into a complete sequence to obtain the local short-term feature M short :
[0148]
[0149] is the concatenation operation of vectors along the time dimension;
[0150] S4. According to the output prediction result and the true data result, with the goal of minimizing the loss function, perform iterative training to update the parameters;
[0151] In specific applications, use the DTW distance as the adaptive block loss, as follows:
[0152] L proto (P) = DTW(P, Q)
[0153] Define another loss function as the mean square error loss between the predicted value and the true value, and the calculation formula is as follows:
[0154]
[0155] X T+1:T+H and respectively represent the true sequence and the predicted sequence for the next H steps;
[0156] The loss function L for iterative training consists of the adaptive block loss and the shape constraint loss, and the calculation formula is as follows:
[0157]
[0158] F is the size of the batch during batch training, λ represents the weight of the loss term, X f represents the input sequence of the f-th sample during batch training, P f represents the prototype block vector of the f-th sample. Use the backpropagation algorithm to minimize the loss function L to achieve the optimum.
[0159] S5. Use the trained long short-term Transformer model to obtain the traffic flow prediction result on the test set.
[0160] Example 2
[0161] To verify the effectiveness of the present invention in traffic flow prediction, in this embodiment, the long - short - term Transformer model is compared with current advanced time - series prediction models, including Transformer - based models: Informer, LogTrans; RNN - based models: LSTNet, TPA - LSTM. The experimental results of the traffic flow prediction method based on the long - short - term Transformer model disclosed in Embodiment 1 on the Traffic dataset of the San Francisco Bay Area highway are shown in Table 1.
[0162] Among them, for different prediction sequence lengths, namely 4, 12, 24, 48, 96, 5 experiments were conducted respectively, and the average results of the 5 tests were taken.
[0163] Among them, the selected evaluation metrics are the mean square error (MSE) and the mean absolute error (MAE). The smaller the values of these two metrics, the better the traffic flow prediction effect. The best results are marked in bold.
[0164] Table 1. Experimental results of the long - short - term Transformer model and other models on the Traffic dataset
[0165]
[0166] As shown in Table 1, among all the models, the long - short - term Transformer model achieved the best traffic flow prediction performance. We can observe that the Transformer - based models achieved better results compared with the RNN - based models (such as LSTNet), which indicates that the Transformer - based models can trace back to more historical information and capture long - term dependencies, so the prediction error is lower. In this case, the long - short - term Transformer model is also superior to other Transformer - based models. There are two reasons according to the analysis: (1) The long - short - term Transformer model comprehensively considers the time - series dependencies from long - term patterns and short - term patterns, explicitly models long - term and short - term dependencies, and thus can better learn the feature representation of the traffic flow sequence; (2) The long - short - term Transformer model uses an adaptive chunking method to automatically segment subsequences, explicitly models short - term dependencies, and can more flexibly capture different short - term patterns.
[0167] Figure 3A traffic flow curve graph of the Traffic dataset over a two-week period is provided. The data is normalized. The Traffic dataset shows long-term and short-term time patterns. It can be observed that the long-term time pattern is a weekly periodic pattern, and the traffic flow on weekdays and weekends is very different; the short-term time pattern is a daily periodic pattern, and the traffic flow at different times of the day varies greatly. Therefore, making accurate predictions is challenging.
[0168] The present invention uses an adaptive chunking method to clearly discover different short-term patterns to enhance the prediction ability. In this embodiment, a sequence chunking graph is drawn to further study whether adaptive chunking can meaningfully chunk the sequence to verify the effectiveness of the adaptive chunking method. Figure 4 It is an example graph of chunking the sequence by the adaptive chunking method on the Traffic dataset. The dotted line represents the chunking position. As Figure 4 shown, for the Traffic dataset with an obvious periodic pattern, the adaptive chunking method successfully identifies each period, which helps to model the short-term time pattern within the period.
[0169] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.
Claims
1. A traffic flow prediction method based on the long-term and short-term Transformer model, characterized by: The traffic flow prediction method comprises the following steps: S1. Obtain traffic flow data and perform standardized preprocessing; S2, position encoding and time encoding of traffic flow data, and extraction of multi-angle feature representation; S3. Input the multi-angle feature representation into the long-short term Transformer model, which includes a sequentially connected long-short term attention layer, a feedforward network layer, a residual connection layer, a normalization layer, and a fully connected layer to obtain the output prediction result; the process is as follows: S3.
1. Input the multi-angle feature into the long-short term attention layer, which includes a parallel long-term attention mechanism and a short-term attention mechanism. The long-term attention mechanism is used to capture the global long-term features M of the traffic flow data. long At the same time, the short-term attention mechanism is used to extract the local short-term features M of traffic flow data. short ; The long-term attention mechanism is used to capture the global long-term features M of traffic flow data. long , the process is as follows: First, the multi-angle feature representation of traffic flow data X is emb The mapping is Q, K, V as follows: Q=W q X emb ,K=W k X emb ,V=W v X emb in Represent the query, key and value in the attention mechanism respectively, To achieve the mapping of the learnable first, second, and third weight matrices, d q ,d k ,d v W q ,W k ,W v The dimension, d model is the dimension of the encoding space; The dot product is used to calculate the correlation between two time steps. The calculation formula is as follows: e is the attention score matrix; Then the normalized exponential function is applied to each element of the attention score matrix e to obtain the normalized attention score as the attention weight, which is calculated as follows: where α ij represents the attention weight between time step i and time step j, A long is the attention weight matrix, e ij represents the attention score between time step i and time step j; Using the attention weight matrix A long Weight V to obtain the global long-term feature M long , the formula is as follows: S4. Based on the output prediction results and the actual data results, iterative training is performed to update the parameters with the goal of minimizing the loss function; S5. Use the trained long-term and short-term Transformer model to obtain traffic flow prediction results on the test set.
2. The traffic flow prediction method based on the long-term and short-term Transformer model according to claim 1 is characterized in that: In step S2, the traffic flow data X 1:T ={x1,x2,…,x T }, X 1:T Abbreviated as X, where represents the traffic flow at time T, where T represents the length of the sequence data. The encoding is performed using Embed(·) to obtain a multi-angle feature representation. The encoding process is as follows: X emb =Embed(X)=E pos (X)+E temporal (X) is the multi-angle feature representation of the traffic flow sequence X, d model is the dimension of the encoding space, E pos (·) and E temporal (·) represents position code and time code respectively; Position code E pos (·) Use sine and cosine position encoding as follows: E pos (X) (p,d) Represents the location feature of traffic flow data X with position p in the time dimension and position d in the space dimension, Indicates rounding down; Time Code E temporal (·) Use a fully connected network to extract features from traffic flow data X as follows: E temporal (X)=W temporal X+b temporal Where W temporal is the weight matrix of the fully connected network, b temporal is the bias term.
3. The traffic flow prediction method based on the long-term and short-term Transformer model according to claim 2 is characterized in that: The step S3 further includes: S3.
2. Fusion of global long-term features and local short-term features to obtain long-term and short-term features M long-short , the calculation formula is as follows: in It is an affine transformation composed of trainable parameters, and Concat(·) is a concatenation operation between vectors along the feature dimension; S3.
3. Long-term and short-term characteristics M long-short After the sequentially connected feedforward network layer, residual connection layer, normalization layer and fully connected layer, the process is as follows: FFN(M long-short )=max(0,M long-short W1+b1)W2+b2 W1 and W2 are the weight matrices of the feedforward network layer, b1 and b2 are bias terms, and max(·) is the operation of taking the maximum element among the elements; H=LayerNorm(FFN(M long-short )+M long-short ) H is the hidden layer feature, LayerNorm(·) represents the normalization operation along the feature dimension, and the elements in the normalization layer are added item by item to represent the residual connection layer; The hidden layer features pass through the fully connected layer to generate a prediction sequence. The formula is as follows: represents the predicted sequence for the next H steps, W pre is the weight matrix of the fully connected layer, b pre is the bias term.
4. The traffic flow prediction method based on the long-term and short-term Transformer model according to claim 1 is characterized in that: The process of step S4 is as follows: Use DTW distance as the adaptive block loss as follows: L proto (P)=DTW(P,Q) Another loss function is defined as the mean square error loss between the predicted value and the true value, and the calculation formula is as follows: X T+1:T+H and Represent the real sequence and predicted sequence for the next H steps respectively; The loss function L of iterative training is composed of adaptive block loss and shape constraint loss, and the calculation formula is as follows: F is the batch size during batch training, λ represents the weight of the loss term, and X f represents the input sequence of the fth sample during batch training, P f The prototype block vector representing the f-th sample is used to minimize the loss function L using the back-propagation algorithm to achieve the optimal value.
5. The traffic flow prediction method based on the long-term and short-term Transformer model according to claim 1, wherein in step S3.1, a short-term attention mechanism is used to extract local short-term features M of traffic flow data. short , the process is as follows: First, define a learnable prototype block vector Represents high-level features at the block level, with a length of N seg , dimension d s , Indicates the 1st, 2nd, ..., Nth seg The prototype block vector P is randomly initialized and updated along with the network parameters; Adaptive block loss is introduced. The adaptive block loss is based on the dynamic time warping (DTW) distance to measure the similarity between two sequences of unequal lengths. The DTW distance with a smoothing factor γ>0 is used and is defined as follows: e′ is a natural constant, B represents the alignment matrix of P and Q, represents the set of alignment matrices B, C represents the cost matrix of alignment between P and Q,<B,C> represents the inner product of matrices B and C; By minimizing the DTW distance, the optimal alignment matrix is obtained, which is calculated as follows: B′ is the optimal alignment matrix, argmin represents the optimal alignment matrix for DTW. γ The optimal alignment matrix when (P,Q) takes the minimum value; According to the optimal alignment matrix, Q, K, and V are divided into blocks as follows: in, s={1,2,…,N seg }, Seg(Q,B′) is a block operation, The non-zero starting position of each row in is used as the split point to divide Q into N seg non-overlapping subsequences, each subsequence has a different length, Q s is the sth subsequence of Q, with a length of L s , Similarly, K and V also undergo block operations, K s is the sth subsequence of K, V s is the s-th subsequence of V; Then, through the attention mechanism, the short-term features of different subsequences are extracted and calculated as follows: e s Represents the attention score matrix of the s-th subsequence; Apply the normalized exponential function to e s Each element of , to obtain the normalized attention score, as the attention weight, is as follows: represents the attention weight between time step m and time step n in the s-th subsequence, represents the attention weight matrix of the s-th subsequence, represents the attention score between time step m and time step n in the s-th subsequence; Using the attention weight matrix V s Weighted, short-term features are obtained, the formula is as follows: Where s={1,2,…,N seg }, Represents the short-term features in the s-th subsequence, for N seg The short-term features in the strip sequence are spliced and recombined to synthesize the complete sequence to obtain the local short-term feature M short : It is a splicing operation between vectors along the time dimension.