Traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention

Abstract

This invention discloses a traffic flow prediction method based on a spatiotemporal frequency adaptive embedding layer and multi-scale trend-aware attention, relating to the transportation field. The problem it addresses is improving the accuracy and real-time performance of traffic prediction, overcoming the limitations of existing traffic flow prediction models, thereby more effectively supporting urban traffic management and decision-making. Regarding this problem, this invention proposes a method based on a spatiotemporal frequency adaptive embedding layer and multi-scale trend-aware attention, encompassing the following aspects: A spatiotemporal frequency adaptive embedding layer is designed, combining spatiotemporal adaptive embedding and frequency enhancement to obtain an effective representation of the data. A multi-scale trend-aware attention mechanism is designed to address some shortcomings of existing attention mechanisms, further enhancing the model's adaptability.

Description

Technical Field

[0001] This invention relates to the field of transportation, and specifically to a traffic flow prediction method based on a spatiotemporal frequency adaptive embedding layer and multi-scale trend-aware attention. Background Technology

[0002] Spatiotemporal series prediction is an important research area with wide applications in various industries, including traffic management, weather forecasting, and financial market analysis. This invention focuses on traffic flow prediction, exploring how to effectively predict future traffic conditions to support urban traffic management and decision-making.

[0003] Traffic flow data is typically stored in a serialized format, meaning it contains traffic information over consecutive time periods. The data at each point in time is called a "time step," and the traffic flow for the next n+T time steps is usually predicted using the data from the previous n time steps. This method effectively utilizes historical data to infer future trends, thus enabling real-time traffic flow prediction.

[0004] One characteristic of traffic flow data is that it originates from multiple sensor nodes, such as roadside cameras, traffic sensors, and GPS devices. These sensors are distributed across different locations within the city, monitoring and recording traffic flow in real time. Because changes in traffic flow are influenced not only by temporal factors but also by strong spatial factors, they constitute complex spatiotemporal relationships. For example, traffic flow is affected by time factors such as daily peak hours, weekends, and holidays, leading to significant variations in flow. Changes in traffic flow between different road segments are interconnected. For instance, when a main road becomes congested, surrounding secondary streets may be rapidly affected as vehicles may choose to detour.

[0005] Traditional traffic flow prediction models, such as linear regression or simple time series analysis, often fail to fully capture the complex spatiotemporal dynamics of traffic data. Here are some key limitations: Many traditional models are based on linear assumptions and cannot handle nonlinear relationships, leading to poor performance in complex scenarios. They also fail to capture sufficient local features, such as sudden events (like traffic accidents or construction) within a specific timeframe, whose impact on traffic flow is short-term and localized. Furthermore, they lack sufficient spatial dependency modeling. When processing data from multiple sensor nodes, traditional models typically employ static modeling, failing to dynamically consider the interactions between different road segments. This static modeling cannot effectively reflect the spatial propagation and evolution of traffic flow. Summary of the Invention

[0006] This invention provides a traffic flow prediction method based on a spatiotemporal frequency adaptive embedding layer and multi-scale trend-aware attention. This method includes a multi-scale trend-aware attention mechanism and a spatiotemporal frequency adaptive embedding mechanism. This invention aims to better capture the complex spatiotemporal dynamic relationships in traffic flow data and fully utilize historical data for accurate prediction. By combining advanced deep learning technology and abundant traffic flow data, the accuracy and real-time performance of traffic prediction are significantly improved, thereby more effectively supporting urban traffic management and decision-making.

[0007] Traffic flow prediction based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention is a novel traffic flow prediction method. By integrating multi-scale trend-aware attention and spatiotemporal frequency adaptive embedding mechanisms, it overcomes the limitations of existing model algorithms, more accurately models the spatial and temporal relationships of traffic flow data, and applies it to real-world problem scenarios. This invention designs multi-scale trend-aware attention and spatiotemporal frequency adaptive embedding mechanisms, improving prediction efficiency and quality. Experimental comparisons with other mainstream algorithms demonstrate that this invention achieves more accurate prediction results, providing a better solution for practical problems.

[0008] To achieve the above objectives, this invention provides a traffic flow prediction method based on a spatiotemporal frequency adaptive embedding layer and multi-scale trend-aware attention, the method comprising:

[0009] Step 1: Obtain input data, specifically preprocessed traffic flow data;

[0010] Step 2: Input traffic flow data into the spatiotemporal frequency adaptive embedding layer to obtain feature embedding weights; wherein, the spatiotemporal frequency adaptive embedding layer includes a serial spatiotemporal adaptive embedding layer and a frequency enhancement layer, as detailed below:

[0011] Step 2.1: First, obtain the spatiotemporal feature representation through a spatiotemporal adaptive embedding layer; specifically, apply a fully connected layer to the traffic flow data to obtain the feature embedding E. f Using time information to obtain periodic embedded E p Define trainable E a As an adaptive embedding, the three embeddings are finally concatenated to obtain the spatiotemporal feature representation;

[0012] Step 2.2: Then, the embedding is enhanced through a frequency enhancement layer; the frequency information in the spatiotemporal feature representation is enhanced using discrete cosine transform to obtain the final feature embedding weights;

[0013] Step 3: The output of Step 2 passes through a Transformer layer. Specifically, the Transformer layer includes a parallel multi-scale trend-aware attention layer based on the time dimension and an attention mechanism layer based on the spatial dimension. Feature embedding weights are performed simultaneously along both the spatial and time dimensions, and finally, the outputs from both dimensions are aggregated. Details are as follows:

[0014] Step 3.1: Obtain the output based on the time dimension through a multi-scale trend-aware attention layer; specifically, replace the fully connected operation in the attention mechanism with a convolution operation to obtain local information, and use multiple convolution heads to extract it simultaneously;

[0015] Step 3.2: Based on the spatial dimension, use the Transformer layer in the existing technology to capture spatial relationships;

[0016] Step 3.3: Aggregate the outputs of 3.1 and 3.2 and use them as input for step 4;

[0017] Step 4: Obtain the final output after passing through the regression layer;

[0018] Step 5: Practical Training

[0019] Step 5.1: Preprocess historical traffic flow data, including removing outliers and normalizing;

[0020] Step 5.2: Determine the input length and output length;

[0021] Step 5.3: Divide the training set, test set, and validation set according to the set ratio;

[0022] Step 5.4: Model training; Train the deployed model using the training set, and use the test set as the basis for optimization and as the basis for performance verification.

[0023] Step 2.1 is as follows:

[0024] The acquired traffic flow data is represented as X t-T+1:t Where t represents the current time and T represents the length of historical data; a fully connected layer is applied to the original traffic data to obtain the feature embedding E. f , which is expressed as follows:

[0025] E f =FC(X) t-T+1:t );

[0026] Using the week embedding mapping matrix W t And the day embedding mapping matrix D t These two time codes are used to extract the corresponding day of the week and intraday time features, and these features are mapped to the week embedding. Japanese Embedded Furthermore and These features form a periodic embedding E through the ∥ operation. p The operation ;|| specifically merges two matrices into a single matrix;

[0027] Adaptive Embedding E a The trainable parameters are defined, and the initial values ​​are randomly initialized.

[0028] Finally, the above three embeddings are concatenated to obtain the spatiotemporal feature representation Z:

[0029] Z = E f ∥E p ∥E a .

[0030] The process of enhancing embedding through the frequency enhancement layer is as follows:

[0031] First, a discrete cosine transform is performed on the spatiotemporal feature representation. According to the formula:

[0032]

[0033] Wherein, Gap represents the lowest frequency component channel representation. L represents the lowest frequency component in a one-dimensional DCT. s The total number of samples representing the spatiotemporal features is represented by ; x is a variable representing the spatiotemporal feature representation, and i is the sample count. After a one-dimensional discrete cosine transform, a one-dimensional vector Freq0 is obtained; similarly, a multi-dimensional vector Freq is obtained. n-1 ;

[0034] Then, the entire frequency channel vector is obtained through a stacking operation:

[0035] Freq=DCT(V)=stack([Freq0,Freq1,…,Freq n-1 ]);

[0036] Freq i ,i=0,1,…n-1 represents an L-dimensional vector after discrete cosine transform, therefore Freq∈R C×L The result of performing a discrete cosine transform on vector V is represented by C, where C represents the number of channels and L represents the length. Vector V represents the result of performing a discrete cosine transform on each channel of the spatiotemporal feature representation Z; stack represents the stacking operation.

[0037] Finally, after linear transformation and activation function processing, the attention weight F for each channel is obtained. c-att The specific formula is as follows:

[0038] F c-attn=σ(W2δ(W1DCT(V)));

[0039] Where W1 and W2 represent linear transformations, δ is a nonlinear activation function, and σ is a linear activation function.

[0040] Step 3.1 is as follows:

[0041] A multi-scale temporal trend-aware attention mechanism is adopted, which uses convolutional operations to obtain contextual information and employs different convolutional heads to obtain information at different scales, as formalized as follows:

[0042]

[0043] In this context, MSTTSelfAttention represents a multi-scale trend-aware attention mechanism, Q, K, and V represent the query, key, and value, respectively, and d represents the size of the convolutional kernel. W represents the merge operation. O This is the final output projection matrix; MSTTHead d,j Representing a single head in a multi-scale dataset, the computation of each head can be expressed as:

[0044]

[0045] These represent the corresponding convolution operations. It is the projection matrix on the value V; the size of the convolution kernel d represents the range of the convolution kernel's convolution.

[0046] Step 3.2 is as follows:

[0047] Given the hidden spatiotemporal feature representation Where T represents the number of time frames, N represents the number of spatial nodes, and d h This is the dimension of the hidden layer. The query Q, key K, and value V matrix are obtained through the spatial transformer layer, as shown below:

[0048]

[0049] in, It is a learnable parameter matrix, from which the query matrix Q is obtained. (sp) Key matrix K (sp) Value matrix V (sp) ;

[0050] Then calculate the self-attention score Z. (sp) The operation through the space transformer layer is as follows:

[0051] Z (sp) =SelfAttention(Q) (sp) ,K (sp)V (sp) )

[0052] Here, SelfAttention represents the self-attention mechanism, which follows these computational steps:

[0053]

[0054] Where, d k The dimension of the key vector is used. The similarity is obtained by using the dot product operation. The similarity is normalized and converted into a probability distribution using the softmax function to obtain the attention weights. The attention weights are then used to perform a weighted summation of the values ​​V to obtain the final output.

[0055] Replace the self-attention mechanism in step 3.2 with a multi-head attention mechanism; assuming there are H attention heads, the calculation process of multi-head attention (MultiHead) is as follows:

[0056] MultiHead(Q,K,V)=Concat(head1,head2,…,head H W O

[0057] Concat means concatenating the results of multiple headers and finally mapping them to the mapping matrix W. O Multiply them to get the final result;

[0058] The calculation for each head is as follows:

[0059]

[0060] These are the learnable parameters of the i-th attention head.

[0061] Step 3.3 is as follows:

[0062] After the multi-scale time trend perception attention mechanism in step 3.1 and the multi-head attention mechanism in step 3.2, residual connections are performed and layer normalization is applied, as shown in the following formula:

[0063] Z MT =LayerNorm(Z+MSTTHead) d,i (Q,K,V))

[0064] Z MH =LayerNorm(Z+MultiHead(Q) (sp) ,K (sp) V (sp) ))

[0065] LayerNorm is a normalization operation performed on the input of each layer; Z MT Z is the output obtained by the multi-scale time trend perception attention mechanism after the above operations. MH This is the output obtained by the multi-head attention mechanism after the above operations;

[0066] Then, the feedforward neural network FFN is applied, and residual connections and layer normalization are added again:

[0067] Z MT′ =LayerNorm(Z) MT +FFN(Z MT ))

[0068] Z MH′ =LayerNorm(Z) MH +FFN(Z MH ))

[0069] The feedforward neural network takes the following form:

[0070] FFN(x)=ReLU(xW1+b1)W2+b2

[0071] W1, W2 and b1, b2 are learnable parameters, and x is a variable;

[0072] Then Z MT′ and Z MH′ By concatenating the components, we obtain the final output Z. ′ .

[0073] The regression layer is described as follows:

[0074]

[0075] in, It is the prediction result, T ′ d represents the time range of the prediction, d represents the dimension of the output feature, and FC represents a fully connected layer.

[0076] The beneficial effects of this invention are as follows:

[0077] Compared with traditional prediction methods, this invention mainly improves the effective representation and spatiotemporal modeling capabilities of data.

[0078] Regarding effective data representation: Previous predictive models often focused on complex model design and novel network architectures, neglecting input embedding, a simple yet powerful representation technique. Furthermore, most studies have overlooked the frequency components of the data, leading to models being frequently affected by temporal noise during training. Moreover, frequency domain analysis techniques, such as Fourier transform, can introduce high-frequency noise and boundary effects (Gibbs phenomenon), impacting model performance and accuracy. This invention increases the types of data embedding and adaptive embedding, and utilizes frequency enhancement to ensure effective data representation.

[0079] In terms of spatiotemporal modeling, most existing prediction models directly use multi-head self-attention mechanisms. These mechanisms perform well when processing discrete data such as natural language. However, in continuous time-series data, this approach fails to effectively identify and utilize local temporal trends and patterns. For example, in traffic flow data, flow rates at different points in time may be consistent, but flow rates can vary significantly due to short-term events (such as traffic accidents or road closures). Traditional self-attention mechanisms mistakenly assume a correlation between these two points, ignoring the preceding and following local information. This requires the model to perceive and adapt to such local changes. To address this issue, this invention designs a multi-scale temporal trend-aware attention mechanism. It uses convolutional operations to acquire contextual information and employs different convolutional heads to obtain information at different scales. The model can simultaneously process and fuse global information and local trends. This comprehensive understanding output not only reflects long-term dependencies in the time series but also reacts to immediate events. This mechanism is particularly suitable for applications requiring responses to multiple time-scale influencing factors (such as traffic flow prediction), as dynamic changes in such scenarios are often influenced by both recent events and long-term trends. Attached Figure Description

[0080] Figure 1 This is an architectural diagram of the method of the present invention. Detailed Implementation

[0081] To better understand the purpose, structure, and function of this invention, the following detailed description of the traffic flow prediction method based on spatiotemporal frequency adaptive embedding layer and multi-scale trend perception attention, in conjunction with the accompanying drawings, is provided.

[0082] Figure 1The diagram shows the architecture of the method of this invention. The traffic flow prediction method based on spatio-frequency adaptive embedding layer and multi-scale trend-aware attention (STAE-TA) includes the following: The model includes a spatio-frequency adaptive embedding layer, a temporally and spatially parallel Transformer layer, and finally a linear prediction layer to output the final result. Specifically, it includes the following steps:

[0083] Step 1: Obtain input. This is generally preprocessed traffic flow data.

[0084] Step 2: Obtain the feature embedding output through the spatiotemporal frequency adaptive embedding layer:

[0085] Step 2.1: Obtain the embedding through a spatiotemporal adaptive embedding layer. Apply a fully connected layer to the original traffic data to obtain the feature embedding E. f Using time information to obtain periodic embedded E p Define trainable E a As an adaptive embedding, the three embeddings are eventually concatenated into one embedding.

[0086] Step 2.2: Enhance the embedding through a frequency enhancement layer. Discrete cosine transform is used to enhance the frequency information in the embedded data obtained from step 2.1, reducing high-frequency noise and avoiding the Gibbs phenomenon.

[0087] Specifically, time-series data such as traffic data often contain complex spatiotemporal dependencies and frequency characteristics, which are crucial for accurate prediction. Previous prediction models have often focused on complex model design and novel network architectures, neglecting input embedding, a simple yet powerful representation technique. Furthermore, most studies have overlooked the frequency component of the data, leading to models being frequently affected by temporal noise during training. In frequency domain analysis, techniques such as Fourier transform can introduce high-frequency noise and boundary effects (Gibbs phenomenon), impacting model performance and accuracy. Therefore, this invention proposes a spatiotemporal frequency adaptive embedding layer to address both of these issues simultaneously. This layer mainly consists of two parts: a spatiotemporal adaptive embedding layer and a frequency enhancement layer, which are sequentially connected. Data flows from the spatiotemporal adaptive embedding layer to the frequency enhancement layer, ultimately serving as the input for the next module.

[0088] Spatiotemporal adaptive embedding layer: raw traffic data X t-T+1:t t represents the current time, T represents the length of historical data, and a fully connected layer is applied to the original traffic data to obtain the feature embedding E. f This process can be represented by the following formula:

[0089] Ef =FC(X) t-T+1:t )

[0090] Here, the fully connected layer FC(·) acts as a feature transformation layer. It maps the original traffic data to a new feature space with dimensions of . R represents the data matrix, where N is the number of spatial nodes, which in this example is the number of days in a week, N = N. w =7.

[0091] Considering the daily time resolution, let N be... d =288, representing the number of time scales in a day. For each time series, the week embedding mapping matrix W is first used... t And the day embedding mapping matrix D t These two time codes are used to extract the corresponding day of the week and intraday time features, and these features are mapped to the week embedding. Japanese Embedded Furthermore and These features form a periodic embedding E through the ∥ operation. p The ∥ operation merges two matrices into a single matrix.

[0092] In addition, an adaptive embedding E is introduced. a Adaptive embeddings are defined by trainable parameters, with initial values ​​randomly initialized, and dimensions equal to... Similarly, from a temporal perspective, a time frame in a traffic time series should exhibit high similarity to its neighboring time frames. This temporal continuity is an indispensable part of traffic flow forecasting, as traffic flows typically exhibit significant short-term autocorrelation.

[0093] Spatially, data captured by sensors from different locations often reflects different traffic patterns. For example, traffic flow in city centers can differ significantly from that in suburbs. a The spatial component is designed to understand and integrate these geospatial differences. Although each sensor may observe traffic flows with unique temporal dynamics, E a Designed to be shareable across different data sources, this means that regardless of the diversity of data sources, the model can make effective predictions by learning common spatiotemporal features. a When processing data from different sensors, it is not necessary to relearn completely independent parameters for each case, which improves the model's generalization ability and training efficiency. Through E a Processing spatiotemporal data within a unified framework simplifies model complexity while ensuring flexibility and accuracy in handling various traffic scenarios. Based on this, a comprehensive spatiotemporal feature representation Z is constructed by concatenating these different embeddings:

[0094] Z = E f ∥E p ∥E a

[0095] Ultimately, this integrated feature representation helps the model achieve higher accuracy in traffic prediction tasks. This approach further enhances the model's ability to interpret and predict traffic time series data.

[0096] Frequency Enhancement Layer: After obtaining the output Z of the spatiotemporal adaptive layer, an effective representation of the time series can be obtained. However, this data is mostly information in the time domain, while frequency is an indispensable element of the time series. Real-world datasets typically contain rich frequency information, allowing deep learning models to better utilize their capabilities. However, most Fourier transform-based methods use Fourier transform to obtain frequency information and inverse Fourier transform to reconstruct time information to avoid complex training, introducing new computational overhead. Furthermore, recent research shows that the frequency domain information of real-world datasets is concentrated in the low-frequency range, while the implicit periodicity of the Discrete Fourier Transform (DFT) leads to boundary discontinuities, producing significant high-frequency content—a phenomenon known as the Gibbs Phenomenon. After quantization, the Gibbs Phenomenon causes boundary points to exhibit erroneous values. Therefore, using Discrete Cosine Transform (DCT) to introduce frequency information can fundamentally avoid the Gibbs Phenomenon and the inverse transform operation.

[0097] The Discrete Cosine Transform (DCT) fundamentally avoids the Gibbs phenomenon caused by periodic boundary treatment in the Discrete Fourier Transform (DFT) and its inverse DFT, and has higher energy compression efficiency compared to the Fourier Transform. The DCT is essentially equivalent to the DFT of real even functions. Because the DCT employs symmetric extension during periodic continuation, according to the formula:

[0098]

[0099] Gap represents the lowest frequency component channel representation. L represents the lowest frequency component in a one-dimensional DCT. S This represents the total number of samples in the input spatiotemporal feature representation. This is because the entire signal needs to be processed during the transformation to extract frequency components. In this embodiment, x is the spatiotemporal feature representation Z, obtained through step 2.1, and the subscript i represents the sample count; after a one-dimensional discrete cosine transform, a one-dimensional vector Freq0 is obtained, and similarly, a multi-dimensional vector Freq is obtained. n-1 .

[0100] Then, the entire frequency channel vector is obtained through a stacking operation:

[0101] Freq=DCT(V)=stack([Freq0,Freq1,…,Freq n-1 ])

[0102] Freq i ,i=0,1,…n-1 represents an L-dimensional vector after discrete cosine transform, therefore Freq∈R C×L (C represents the number of channels, L represents the length) represents the vector V∈R C×L (V represents the result of performing a Discrete Cosine Transform (DCT) on each channel of Z); stack represents the stacking operation. After obtaining Freq, the attention weights can be learned through the neural network structure. The entire frequency enhancement layer framework can be described as a combination of Discrete Cosine Transform enhancement and attention mechanisms:

[0103] F c-att =σ(W2δ(W1DCT(V)))

[0104] The formula indicates that after the input vector V undergoes DCT transformation, it is processed by two linear transformations W1 and W2, a nonlinear activation function δ, and a linear activation function σ, ultimately resulting in the output F. c-att The W2 transformation is the result of applying the activation function σ, representing the attention weight for each channel. These weights can be used to weight input features, thereby enhancing important features and suppressing unimportant ones.

[0105] This method allows the features of each channel to interact with their corresponding frequency components, comprehensively capturing key temporal information from the frequency domain. This process greatly enriches the network's feature extraction capabilities. Subsequent experiments will demonstrate the visualization of the frequency channel attention tensors, showcasing the importance of each channel learned in the frequency domain and the relative importance of different frequency components within each channel. By enhancing the extraction and utilization of frequency information, this method significantly improves the performance of time series models, enabling them to exhibit superior predictive capabilities in various practical applications, particularly in complex traffic flow prediction problems. Combining the output of the spatiotemporally adaptive embedding layer as input further enhances the overall model performance.

[0106] Step 3: After passing through the Transformer layer, the output of Step 2 is received and processed simultaneously along both the spatial and temporal dimensions:

[0107] Step 3.1: Obtain the output by passing through a multi-scale trend-aware attention layer along the time dimension. Replace the fully connected Q, K, and V in the attention mechanism with convolutional operations to obtain local information, and extract it simultaneously using multiple convolutional heads.

[0108] Step 3.2: Obtain the output along the spatial dimension using an attention mechanism. Use a Transformer layer to capture spatial relationships.

[0109] Step 3.3: Combine the outputs of 3.1 and 3.2 as input for step 4.

[0110] Specifically, the Transformer layer consists of a series of identical layers. The Transformer layer captures the relationships between data from both spatial and temporal dimensions, and finally merges the outputs and passes them to the regression layer to obtain the final result.

[0111] Multi-scale Trend-Aware Attention Layer (Time Dimension): Self-attention is a specific attention mechanism where queries, keys, and values ​​come from symbolic representations of the same sequence. In practice, multi-head self-attention is the most commonly used form of self-attention, capable of processing information from different representation subspaces simultaneously. The core operation of multi-head self-attention is scaling the dot product attention, representing similarity; the greater the similarity, the stronger the correlation between data. All queries, keys, and values ​​are representations of the same sequence. This method first linearly maps queries, keys, and values ​​to different representation subspaces, then computes the attention function in parallel. Finally, all outputs are concatenated and further projected to generate the desired output. Its formal definition is as follows:

[0112]

[0113] head j =Attention(QW j Q ,KW j K VW j V )

[0114] Where Q, K, and V represent query, key, and value, respectively, and W... j Q W j K W j V W is the projection matrix on Q, K, V. O It is the final output projection matrix. MHSelfAttention represents the merging operation, MHSelfAttention represents the multi-head attention mechanism, and Attention represents attention to a single head. jAttention operations are performed. For the time dimension, most existing time series prediction models directly use multi-head self-attention mechanisms. These mechanisms perform well when processing discrete data such as natural language. However, in continuous time series data, this method cannot effectively identify and utilize local temporal trends and patterns. For example, in traffic flow data, traffic flow data at different time points may be consistent, but the flow may vary significantly due to short-term events (such as traffic accidents or road closures). Traditional self-attention mechanisms may mistakenly assume that these two points are correlated, ignoring the local information before and after them. This requires the model to be able to perceive and adapt to such local changes. To solve the above problems, this invention designs a multi-scale time trend-aware attention mechanism, which obtains contextual information through convolution operations and uses different convolution heads to obtain information at different scales. Its formal definition is as follows:

[0115]

[0116] The calculation for each head can be represented as:

[0117]

[0118] Where Q, K, and V represent query, key, and value, respectively, and d represents the size of the convolution kernel. These represent the convolution operation, and the kernel size represents the range of the convolution operation. W is the projection matrix on V. O It is the final output projection matrix.

[0119] The self-attention mechanism for spatiotemporal trend perception is achieved through the following key steps to deeply capture and understand the complex dynamics in time series data:

[0120] Trend-aware attention weights: When calculating self-attention weights, in addition to traditional content-based weights (calculated through the relationships between queries, keys, and values), weights based on local trends are introduced. This involves applying one-dimensional convolution to capture the local context before and after each time point, performing convolution operations on one-dimensional data (such as time series). It extracts local features by sliding a convolutional kernel (filter) across the input sequence. The convolutional kernel performs a weighted sum operation at each location, resulting in a new feature map that captures local patterns in the input data. The size of the convolutional kernel determines the size of the captured context window. By applying convolution to time series, the model can learn a certain amount of contextual information before and after each time point. For example, if the convolutional kernel size is 3, the model will consider the values ​​at the current time point and one time point before and after it. This approach allows the model to consider not only their immediate similarity but also their local trends of change in the time series when evaluating the correlation between two time points.

[0121] Multi-head trend perception mechanism: Drawing inspiration from traditional multi-head attention mechanisms, the trend perception self-attention also employs multiple parallel attention heads, each capturing different features of the sequence through an independent one-dimensional convolutional kernel. This design enables the model to have higher sensitivity and discriminative ability to various patterns in time series, thereby improving the diversity of information and the depth of processing.

[0122] Multi-scale information acquisition: The model uses one-dimensional convolutional kernels of different sizes (e.g., 3, 5, 7) to capture multi-scale features of time series. Small convolutional kernels can capture minute temporal changes, while larger kernels help to grasp more macroscopic temporal trends. This is similar to the application in image processing, where filters of different sizes are used to extract features at different levels. The features captured by convolutional kernels of various sizes reflect dynamic characteristics at different time scales, thus forming a comprehensive understanding from short-term fluctuations to long-term trends. The model integrates these features along a specific dimension (e.g., feature or channel dimension) to form a comprehensive feature representation, ensuring that information from multiple time scales is fully considered when making decisions.

[0123] Integrated processing of global and local information: Through a trend-aware self-attention mechanism, the model can simultaneously process and integrate global information and local trends. This comprehensive understanding output not only reflects long-term dependencies in the time series but also responds to immediate events. This mechanism is particularly suitable for applications requiring responses to multiple time-scale influencing factors (such as traffic flow forecasting), as dynamic changes in these scenarios are typically influenced by both recent events and long-term trends.

[0124] Spatial Attention Layer: In the spatial dimension, existing Transformer layers are directly used to capture spatial relationships. Specifically, simple vanilla transformers are applied along the time and spatial axes to capture complex traffic relationships. Given the hidden spatiotemporal representation... Where T represents the number of time frames, N represents the number of spatial nodes, and d h This refers to the dimension of the hidden layer. The query, key, and value matrices are obtained through the spatial transformer layer, as shown below:

[0125]

[0126] in, It is a learnable parameter matrix, where Q, K, and V represent the query, key, and value, respectively, resulting in the query matrix Q. (sp) Key matrix K (sp) Value matrix V (sp) .

[0127] Next, we calculate the self-attention score Z. (sp) The operation of the space transformer layer is as follows:

[0128] Z (sp) =SelfAttention(Q) (sp) ,K (sp) V (sp) )

[0129] Here, SelfAttention represents the self-attention mechanism, which follows these computational steps:

[0130]

[0131] Where, d k It is the dimension of the key vector, usually related to d. h In this process, Q represents the query, K represents the key, and V represents the value. The similarity is obtained by using the dot product operation, and the similarity is normalized. The softmax function is used to convert it into a probability distribution to obtain the attention weights. The attention weights are then used to perform a weighted summation on the value vector (V) to obtain the final output.

[0132] The complete operation of the space transformer layer includes the following steps:

[0133] To enhance the model's expressive power, a multi-head attention mechanism is employed. Specifically, assuming there are H attention heads, each head has a dimension of... The calculation process for MultiHead attention is as follows:

[0134] MultiHead(Q,K,V)=Concat(head1,head2,…,head H W O

[0135] Concat means concatenating the results of multiple headers and finally mapping them to the mapping matrix W. O Multiply them to get the final result.

[0136] The calculation for each head is as follows:

[0137]

[0138] These are the learnable parameters of the i-th attention head. It is the output weight matrix.

[0139] After the multi-scale time trend perception attention mechanism in step 3.1 and the multi-head attention mechanism in step 3.2, residual connections are performed and layer normalization is applied, as shown in the following formula:

[0140] Z MT =LayerNorm(Z+MSTTHead) d,j (Q,K,V))

[0141] Z MH =LayerNorm(Z+MultiHead(Q) (sp) ,K (sp) V (sp) ))

[0142] LayerNorm is a normalization operation performed on the input of each layer, normalizing the features of each sample; where Z MT Z is the output obtained by the multi-scale time trend perception attention mechanism after the above operations. MH This is the output obtained by the multi-head attention mechanism after the above operations;

[0143] Then, the feedforward neural network FFN is applied, and residual connections and layer normalization are added again:

[0144] Z MT′ =LayerNorm(Z) MT +FFN(Z MT ))

[0145] Z MH′ =LayerNorm(Z) MH +FFN(Z MH ))

[0146] The feedforward neural network takes the following form:

[0147] FFN(x)=ReLU(xW1+b1)W2+b2

[0148] W1, W2 and b1, b2 are learnable parameters, and x is a variable, which is Z in this embodiment. MT′ and Z MH′ ;

[0149] Then Z MT′ and Z MH′ By concatenating the components, the final output is obtained.

[0150] Step 4: Obtain the final output after passing through the regression layer.

[0151] Using the output Z of the spacetime transformer layer ′ To generate predictions. The regression layer can be described as:

[0152]

[0153] in, It is the prediction result, T ′ d represents the prediction time range, and d is the dimension of the output feature. In this embodiment, d = 1. Therefore, the fully connected layer (FC) will... ′ The dimensions in the T×N×d range from h Return to T in ′ ×N×d.

[0154] Step 5: Practical Training

[0155] Step 5.1: Preprocess the historical traffic flow data, including removing outliers and normalizing.

[0156] Step 5.2: Determine the input length and output length. Generally, the input length and output length are the same. For example, if the historical time interval is 1 minute per data point, the input length is 90, and the output length is 90, then the input data is the data from the previous 90 minutes, and the output is the predicted data from the next 90 minutes.

[0157] Step 5.3: Split the training set, test set, and validation set. The ratio is typically 7:1:2 or 6:2:2.

[0158] Step 5.4: Model Training. Train the deployed model using the training set, and use the test set as the basis for optimization and performance verification.

[0159] Experimental results

[0160] As shown in Table 1, the method provided in this invention achieves better performance on most metrics across all four datasets. The method addresses the previous model's lack of ability to capture different frequency patterns in the data by employing Discrete Cosine Transform (DCT) to enhance the channel attention mechanism, avoiding high-frequency noise issues caused by Fourier Transform methods (such as the Gibbs phenomenon). This method enriches the model's understanding of periodic components in the data and improves its predictive ability. This research complements spatiotemporal embedding by presenting periodic behavior in traffic data in detail. Furthermore, a multi-scale trend-aware attention mechanism is employed to address the problem of traditional attention mechanisms' inability to utilize contextual information. The multi-scale model enables the model to learn temporal patterns at different scales, from local changes to broader trends.

[0161] Table 1 compares the performance of the method of this invention with that of the reference algorithm.

[0162]

[0163]

[0164] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention, characterized in that, The following steps are involved: Step 1: Obtain input data, specifically preprocessed traffic flow data; Step 2: Input traffic flow data into the spatiotemporal frequency adaptive embedding layer to obtain feature embedding weights; wherein, the spatiotemporal frequency adaptive embedding layer includes a serial spatiotemporal adaptive embedding layer and a frequency enhancement layer, as detailed below: Step 2.1: First, obtain the spatiotemporal feature representation through a spatiotemporal adaptive embedding layer; specifically, apply a fully connected layer to the traffic flow data to obtain feature embeddings. Utilizing time information to obtain periodic embeddings Define trainable As an adaptive embedding, the three embeddings are finally concatenated to obtain the spatiotemporal feature representation; Step 2.2: Then, the embedding is enhanced through a frequency enhancement layer; the frequency information in the spatiotemporal feature representation is enhanced using discrete cosine transform to obtain the final feature embedding weights; Step 3: The output of Step 2 passes through a Transformer layer. Specifically, the Transformer layer includes a parallel multi-scale trend-aware attention layer based on the time dimension and an attention mechanism layer based on the spatial dimension. Feature embedding weights are performed simultaneously along both the spatial and time dimensions, and finally, the outputs from both dimensions are aggregated. Details are as follows: Step 3.1: Obtain the output based on the time dimension through a multi-scale trend-aware attention layer; specifically, replace the fully connected operation in the attention mechanism with a convolution operation to obtain local information, and use multiple convolution heads to extract it simultaneously; Step 3.2: Based on the spatial dimension, use the Transformer layer in the existing technology to capture spatial relationships; Step 3.3: Aggregate the outputs of 3.1 and 3.2 and use them as input for step 4; Step 4: Obtain the final output after passing through the regression layer.

2. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention as described in claim 1, characterized in that, Step 2.1 is as follows: The acquired traffic flow data is represented as Where t represents the current time and T1 represents the length of historical data; a fully connected layer is applied to the original traffic data to obtain feature embeddings. , which is expressed as follows: ； Wherein, FC represents a fully connected layer; Using the week embedding mapping matrix and day embedding mapping matrix These two time codes are used to extract the corresponding day of the week and intraday time features, and these features are mapped to the week embedding. Japanese Embedded , and thus and These features are achieved through The operation forms a periodic embedding ; The specific operation involves merging two matrices into a single matrix; Adaptive Embedding The trainable parameters are defined, and the initial values ​​are randomly initialized. Finally, the above three embeddings are concatenated to obtain the spatiotemporal feature representation Z: 。 3. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention according to claim 1, characterized in that, The process of enhancing embedding through the frequency enhancement layer is as follows: First, a discrete cosine transform is performed on the spatiotemporal feature representation. According to the formula: ； Where gap represents the lowest frequency component channel representation. This represents the lowest frequency component in a one-dimensional discrete cosine transform (DCT). This represents the total number of samples representing the spatiotemporal features of the input. Let be a variable, representing the spatiotemporal feature representation, where i is the sample count. After a one-dimensional discrete cosine transform, a one-dimensional vector is obtained. Similarly, we can obtain multidimensional vectors. ; Then, the entire frequency channel vector is obtained through a stacking operation: ； in Freq represents an L-dimensional vector after discrete cosine transform. The result of performing a discrete cosine transform on vector A is represented by C, where C represents the number of channels and L represents the length. Vector A represents the result of performing a discrete cosine transform on each channel of the spatiotemporal feature representation Z; stack represents the stacking operation. Finally, after linear transformation and activation function processing, the attention weights for each channel are obtained. The specific formula is as follows: ； Where W1 and W2 represent linear transformations. It is a non-linear activation function. It is a linear activation function.

4. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention according to claim 1, characterized in that, Step 3.1 is as follows: A multi-scale temporal trend-aware attention mechanism is adopted, which uses convolutional operations to obtain contextual information and employs different convolutional heads to obtain information at different scales, as formalized as follows: ； in, This represents a multi-scale trend-aware attention mechanism. , and These represent the query, key, and value, respectively. Indicates the size of the convolution kernel. This indicates a merge operation. It is the final output projection matrix; Representing a single head in a multi-scale dataset, the computation of each head can be expressed as: ； , These represent the corresponding convolution operations. It is the projection matrix onto value V; the size of the convolution kernel d represents the range of the convolution kernel's convolution. Indicates a single head Perform attention manipulation.

5. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention according to claim 4, characterized in that, Step 3.2 is as follows: Given the hidden spatiotemporal feature representation ,in Indicates the number of time frames. Indicates the number of spatial nodes. This is the dimension of the hidden layer. The query Q, key K, and value V matrix are obtained through the spatial transformer layer, as shown below: ； in, It is a learnable parameter matrix, from which the query matrix is ​​obtained. Key matrix Value matrix ; Then calculate the self-attention score. The operation through the space transformer layer is as follows: ； in, This represents the self-attention mechanism, which follows these computational steps: ； in, This refers to the dimension of the key vector. Similarity is obtained using the dot product operation, and the similarity is then normalized. The function transforms it into a probability distribution, obtains attention weights, and uses these attention weights to perform a weighted summation of the value V, resulting in the final output.

6. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention according to claim 5, characterized in that, Replace the self-attention mechanism in step 3.2 with a multi-head attention mechanism; assuming there is Individual attention, multi-headed attention The calculation process is as follows: ； This means concatenating the results from multiple heads and finally combining them with the output projection matrix. Multiply them to get the final result; The calculation for each head is as follows: ； It is the first Learnable parameters for each attention head.

7. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention according to claim 6, characterized in that, Step 3.3 is as follows: After the multi-scale time trend perception attention mechanism in step 3.1 and the multi-head attention mechanism in step 3.2, residual connections are performed and layer normalization is applied, as shown in the following formula: ； ； It is a normalization operation performed on the input of each layer; The output obtained by the multi-scale time trend perception attention mechanism after the above operations is shown. This is the output obtained by the multi-head attention mechanism after the above operations; Then, the feedforward neural network FFN is applied, and residual connections and layer normalization are added again: ； ； The feedforward neural network takes the following form: ； and It is a learnable parameter, and x is a variable; Then and By concatenating the components, the final output is obtained. .

8. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention according to claim 7, characterized in that, The regression layer is described as follows: ； in, It is a prediction result. It is the predicted time range. It represents the dimension of the output features, and FC stands for fully connected layer.

9. A traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend-aware attention as described in claim 8, characterized in that, The specific steps for actually training the method are as follows: Step 5.1: Preprocess historical traffic flow data, including removing outliers and normalizing; Step 5.2: Determine the input length and output length; Step 5.3: Divide the training set, test set, and validation set according to the set ratio; Step 5.4: Model training; Train the deployed model using the training set, and use the test set as the basis for optimization and as the basis for performance verification.

Images