Traffic flow prediction method based on space-time frequency adaptive embedding and multi-scale trend perception attention

Abstract

The invention discloses a traffic flow prediction method based on a space-time frequency adaptive embedded layer and multi-scale trend perception attention, relates to the field of traffic, and aims at improving the accuracy and real-time performance of traffic prediction aiming at the limitation of an existing traffic flow prediction model so as to support urban traffic management and decision making more effectively. For the problem, the invention provides a method based on a space-time frequency adaptive embedding layer and multi-scale trend perception attention, which covers the following aspects: the space-time frequency adaptive embedding layer is designed, and space-time adaptive embedding and frequency enhancement are combined to obtain effective representation of data. And a multi-scale trend perception attention mechanism is designed to solve some defects of the existing attention mechanism and further improve the adaptability of the model.

Description

Technical Field

[0001] The present invention relates to the field of transportation, and in particular to a traffic flow prediction method based on a spatiotemporal frequency adaptive embedding layer and multi-scale trend perception attention. Background Art

[0002] Spatiotemporal sequence prediction tasks are an important research area and are widely used in many industries, including traffic management, weather forecasting, financial market analysis, etc. This paper will focus on traffic flow prediction and explore how to effectively predict future traffic conditions to support urban traffic management and decision making.

[0003] Traffic flow data usually exists in a serialized form, which means that the data contains traffic information for consecutive time periods. The data at each time point is called a "time step", and the data of the first n time steps are usually used to predict the traffic flow of the next n+T time steps. This method can effectively use historical data to infer future situations, thereby achieving real-time prediction of traffic flow.

[0004] One of the characteristics of traffic flow data is that it comes from multiple sensor nodes, such as road cameras, traffic sensors, and GPS devices. These sensors are distributed in different locations in the city to monitor and record traffic flow in real time. Since changes in traffic flow are not only affected by time factors, but also strongly affected by spatial factors, a complex spatiotemporal relationship is formed. For example, traffic flow is affected by time factors such as daily peak hours, weekends, and holidays, which can cause significant changes in traffic flow. Traffic flow changes between different road sections are interrelated. For example, when a main road is congested, the surrounding secondary streets may be quickly affected because vehicles may choose to detour.

[0005] Traditional traffic flow prediction models, such as linear regression or simple time series analysis, often fail to fully capture the complex spatiotemporal dynamic relationships in traffic data. Here are some of the main limitations: Many traditional models are based on linear assumptions and cannot handle nonlinear relationships, resulting in poor performance of the models in complex scenarios. Insufficient capture of local features. These models often have difficulty capturing local features, such as emergencies within certain time periods (such as traffic accidents or construction), which have a short-term and local impact on traffic flow. Insufficient modeling of spatial dependencies. Traditional models usually adopt a static modeling approach when processing data from multiple sensor nodes, and fail to dynamically consider the interactions between road sections. This static modeling cannot effectively reflect the propagation and evolution of traffic flow in space. Summary of the invention

[0006] The present invention provides a traffic flow prediction method based on a spatiotemporal frequency adaptive embedding layer and multi-scale trend perception attention. The method includes a multi-scale trend perception attention mechanism and a spatiotemporal frequency adaptive embedding mechanism. The method of the present invention can better capture the complex spatiotemporal dynamic relationship in traffic flow data and make full use of historical data for accurate prediction. By combining advanced deep learning technology and rich traffic flow data, the accuracy and real-time performance of traffic prediction can be significantly improved, thereby more effectively supporting urban traffic management and decision-making.

[0007] Traffic flow prediction based on spatiotemporal frequency adaptive embedding layer and multi-scale trend-aware attention is a new traffic flow prediction method. By integrating multi-scale trend-aware attention mechanism, spatiotemporal frequency adaptive embedding mechanism and other mechanisms, it solves the limitations of existing model algorithms, more accurately models the spatial and temporal relationship of traffic flow data, and applies it to practical problem scenarios. The method of the present invention designs mechanisms such as multi-scale trend-aware attention and spatiotemporal frequency adaptive embedding to improve the efficiency and quality of prediction. By experimentally comparing with other mainstream algorithms, the method of the present invention can obtain more accurate prediction results. It provides a better solution to solve practical problems.

[0008] To achieve the above object, the present invention provides a traffic flow prediction method based on spatiotemporal frequency adaptive embedding layer and multi-scale trend perception attention, the method comprising:

[0009] Step 1: Obtain input data, specifically pre-processed traffic flow data;

[0010] Step 2: Input traffic flow data into the spatiotemporal frequency adaptive embedding layer to obtain feature embedding weights; wherein the spatiotemporal frequency adaptive embedding layer includes a serial spatiotemporal adaptive embedding layer and a frequency enhancement layer, as follows:

[0011] Step 2.1: First, the spatiotemporal feature representation is obtained through the spatiotemporal adaptive embedding layer; specifically, the fully connected layer is applied to the traffic flow data to obtain the feature embedding E f , using time information to obtain periodic embedding E p , define the trainable E a As an adaptive embedding, the three embeddings are finally concatenated to obtain the spatiotemporal feature representation;

[0012] Step 2.2: Then the embedding is enhanced through the frequency enhancement layer; the frequency information in the spatiotemporal feature representation is enhanced using discrete cosine transform to obtain the final feature embedding weight;

[0013] Step 3: The output of step 2 passes through the Transformer layer. Specifically, the Transformer layer includes a parallel multi-scale trend perception attention layer based on the time dimension and an attention mechanism layer based on the spatial dimension. The feature embedding weights are performed simultaneously along the spatial dimension and the time dimension, and finally the output results of the two dimensions are aggregated. The details are as follows:

[0014] Step 3.1: Obtain output through a multi-scale trend-aware attention layer based on the time dimension; specifically, replace the fully connected operation in the attention mechanism with a convolution operation to obtain local information, and use multiple convolution heads to extract simultaneously;

[0015] Step 3.2: Based on the spatial dimension, use the Transformer layer in the existing technology to capture the spatial relationship;

[0016] Step 3.3: Aggregate the output results of 3.1 and 3.2 as the input of step 4;

[0017] Step 4: Get the final output through the regression layer;

[0018] Step 5: Actual Training:

[0019] Step 5.1: Preprocess the historical traffic flow data, including removing outliers and normalizing;

[0020] Step 5.2: Determine the input length and output length;

[0021] Step 5.3: Split the training set, test set, and validation set according to the set ratio;

[0022] Step 5.4: Model training: Use the training set to train the deployed model, use the test set as the basis for tuning, and use the test set as the effect verification.

[0023] The step 2.1 is as follows:

[0024] The traffic flow data obtained is represented by X t-T+1:t , where t represents the current time and T represents the length of historical data; a fully connected layer is applied to the original traffic data to obtain the feature embedding E f , which is expressed as follows:

[0025] E f =FC(X t-T+1:t );

[0026] Using Zhou embedding mapping matrix W t and the daily embedding mapping matrix D t These two time codes are used to extract the corresponding day of the week and intraday time features and map these features into the weekly embedding Japanese Embed Then and These features form a periodic embedding E through the ∥ operation p ; The specific operation of ∥ is to merge two matrices into the same matrix;

[0027] Adaptive Embedding a Defined by trainable parameters, the initial values ​​are randomly initialized;

[0028] Finally, the above three embeddings are concatenated to obtain the spatiotemporal feature representation Z:

[0029] Z=E f ∥E p ∥E a .

[0030] The process of embedding through frequency enhancement layer enhancement is as follows:

[0031] First, perform discrete cosine transform on the spatiotemporal feature representation. According to the formula:

[0032]

[0033] Among them, Gap represents the lowest frequency component channel representation, Represents the lowest frequency component in one-dimensional DCT, L s Represents the total number of samples represented by the input spatiotemporal feature; x is a variable, referring to the spatiotemporal feature representation, i is the sample count, and after a one-dimensional discrete cosine transform, a one-dimensional vector Freq0 is obtained. Similarly, a multidimensional vector Freq n-1 ;

[0034] Then the entire frequency channel vector is obtained by stacking:

[0035] Freq=DCT(V)=stack([Freq0,Freq1,…,Freq n-1 ]);

[0036] Where Freq i , i = 0, 1, ... n-1 represents the L-dimensional vector after discrete cosine transform, so Freq ∈ R C×L Represents the result of discrete cosine transform of vector V, where C represents the number of channels, L represents the length, and vector V represents the result of discrete cosine transform of each channel of spatiotemporal feature representation Z; stack represents stacking operation;

[0037] Finally, after linear transformation and activation function processing, the attention weight F of each channel is obtained c-att ; The specific formula is as follows:

[0038] F c-attn=σ(W2δ(W1DCT(V)));

[0039] Where W1 and W2 represent linear transformations, δ is a nonlinear activation function, and σ is a linear activation function.

[0040] The step 3.1 is as follows:

[0041] A multi-scale time-trend-aware attention mechanism is used to obtain contextual information through convolution operations, and different convolution heads are used to obtain information at different scales, which is formalized as follows:

[0042]

[0043] Among them, MSTTSelfAttention represents the multi-scale trend-aware attention mechanism, Q, K, and V represent query, key, and value respectively, and d represents the size of the convolution kernel. represents the merge operation, W O is the final output projection matrix; MSTTHead d,j Representing a single head in multiple scales, the calculation of each head can be expressed as:

[0044]

[0045] They represent the corresponding convolution operations respectively. It is the projection matrix on the value V; the size of the convolution kernel d indicates the range of the convolution kernel convolution.

[0046] The step 3.2 is as follows:

[0047] Given a hidden spatiotemporal feature representation Where T represents the number of time frames, N represents the number of spatial nodes, and d h is the dimension of the hidden layer, and the query Q, key K, value V matrix is ​​obtained through the spatial transformer layer as shown below:

[0048]

[0049] in, is a learnable parameter matrix, and the query matrix Q is obtained (sp) , the bond matrix K (sp) , value matrix V (sp) ;

[0050] Then calculate the self-attention score Z (sp) , through the spatial transformer layer operation as follows:

[0051] Z (sp) =SelfAttention(Q (sp) ,K (sp),V (sp) )

[0052] Among them, SelfAttention represents the self-attention mechanism, which follows the following calculation steps:

[0053]

[0054] Among them, d k is the dimension of the key vector, and the dot product operation is used to obtain the similarity. The similarity is normalized and converted into a probability distribution using the softmax function to obtain the attention weight. The attention weight is used to perform a weighted summation on the value V to obtain the final output.

[0055] Replace the self-attention mechanism in step 3.2 with a multi-head attention mechanism; assuming there are H attention heads, the calculation process of MultiHead is as follows:

[0056] MultiHead(Q,K,V)=Concat(head1,head2,…,head H )W O

[0057] Concat means concatenating the results of multiple headers and finally adding them to the mapping matrix W. O Multiply to get the final result;

[0058] The calculation for each head is:

[0059]

[0060] is the learnable parameter of the i-th attention head.

[0061] The step 3.3 is as follows:

[0062] After the multi-scale time trend-aware attention mechanism in step 3.1 and the multi-head attention mechanism in step 3.2, residual connections are performed and layer normalization is performed, respectively. The formula is as follows:

[0063] Z MT =LayerNorm(Z+MSTTHead d,i (Q,K,V)

[0064] Z MH =LayerNorm(Z+MultiHead(Q (sp) ,K (sp) ,V (sp) ))

[0065] LayerNorm is a normalization operation performed on the input of each layer; Z MT is the output of the multi-scale time trend perception attention mechanism after the above operations, Z MH is the output of the multi-head attention mechanism after the above operations;

[0066] Then apply the feed-forward neural network FFN separately, and add residual connections and layer normalization again:

[0067] Z MT′ =LayerNorm(Z MT +FFN(Z MT ))

[0068] Z MH′ =LayerNorm(Z MH +FFN(Z MH ))

[0069] Among them, the form of the feedforward neural network is:

[0070] FFN(x)=ReLU(xW1+b1)W2+b2

[0071] W1, W2 and b1, b2 are learnable parameters, and x is a variable;

[0072] Then Z MT′ and Z MH′ Splice to get the final output Z ′ .

[0073] The regression layer is expressed as follows:

[0074]

[0075] in, is the prediction result, T ′ is the predicted time range, d is the dimension of the output feature, and FC represents the fully connected layer.

[0076] The beneficial effects of the present invention are as follows:

[0077] Compared with traditional prediction methods, the present invention mainly improves the effective representation ability of data and the spatiotemporal modeling ability.

[0078] Regarding the effective representation of data: previous prediction models often focus on complex model design and novel network architecture, and input embedding, as a simple and powerful representation technology, is often overlooked. It is also found that most studies ignore the frequency components of the data, which causes previous models to be often affected by time domain noise data during the training process. Moreover, in frequency domain analysis, such as Fourier transform, high-frequency noise and boundary effects (Gibbs phenomenon) may be introduced, which will affect the performance and accuracy of the model. The present invention increases the data embedding type and adaptive embedding, and uses frequency enhancement to ensure the effective representation of data.

[0079] For spatiotemporal modeling: Most of the existing prediction models directly use multi-head self-attention mechanisms. This type of attention mechanism performs well when processing discrete data such as natural language. However, in continuous time series data, this method cannot effectively identify and utilize local time trends and patterns of data. For example, in traffic flow data, the flow data at different time points may be consistent, but the flow may be significantly different due to events in a short period of time (such as traffic accidents or road blockages). The traditional self-attention mechanism will mistakenly believe that these two points are related, while ignoring the local information before and after, which requires the model to be able to perceive and adapt to such local changes. In order to solve the above problems, the present invention designs a multi-scale time trend perception attention mechanism, which obtains contextual information through convolution operations and uses different convolution heads to obtain information at different scales. The model can simultaneously process and fuse global information and local trends. The output of this comprehensive understanding not only reflects the long-term dependencies in the time series, but also responds to immediate events. This mechanism is particularly suitable for application scenarios that need to respond to multiple time scale influencing factors (such as traffic flow prediction), because the dynamic changes in such scenarios are usually affected by recent events and long-term trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0081] In order to better understand the purpose, structure and function of the present invention, the traffic flow prediction method based on spatiotemporal frequency adaptive embedding layer and multi-scale trend perception attention of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0082] Figure 1The architecture diagram of the method of the present invention is as follows: the traffic flow prediction method based on spatio-frequency adaptive embedding layer and multi-scale trend perception attention (referred to as STAE-TA) includes the following contents: the model includes a spatio-frequency adaptive embedding layer (Spatio-Fr equency Adaptive Network), a temporal and spatial parallel Transformer layer, and finally a linear prediction layer is used to output the final result. Specifically, it includes the following steps:

[0083] Step 1: Get input, which is usually pre-processed traffic flow data.

[0084] Step 2: Get the feature embedding output through the spatiotemporal frequency adaptive embedding layer:

[0085] Step 2.1: Obtain embedding through spatiotemporal adaptive embedding layer. Apply fully connected layer to the original traffic data to obtain feature embedding E f , using time information to obtain periodic embedding E p , define the trainable E a As an adaptive embedding, the three embeddings are finally concatenated into one embedding.

[0086] Step 2.2: Enhance the embedding through the frequency enhancement layer. Use discrete cosine transform to enhance the frequency information in the embedded data obtained from 2.1, reduce high-frequency noise, and avoid Gibbs phenomenon.

[0087] Specifically, time series data such as traffic usually contain complex spatiotemporal dependencies and frequency characteristics, which are crucial for accurate prediction. Previous prediction models often focus on complex model design and novel network architectures, and input embedding, as a simple and powerful representation technology, is often overlooked. It is also found that most studies ignore the frequency components of the data, which causes previous models to be often affected by time domain noise data during the training process. In addition, in frequency domain analysis, such as Fourier transform, high-frequency noise and boundary effects (Gibbs phenomenon) may be introduced, which will affect the performance and accuracy of the model. Therefore, the present invention proposes to use a spatiotemporal frequency adaptive embedding layer to solve these two problems at the same time. This layer is mainly divided into the following two parts, the spatiotemporal adaptive embedding layer and the frequency enhancement layer. The two are in a serial relationship. The data flows from the spatiotemporal adaptive embedding layer to the frequency enhancement layer, and finally serves as the input of the next module.

[0088] Spatiotemporal adaptive embedding layer: original traffic data X t-T+1:t , t represents the current time, T represents the length of historical data, and a fully connected layer is applied to the original traffic data to obtain the feature embedding E f This process can be expressed by the following formula:

[0089] Ef =FC(X t-T+1:t )

[0090] Here, the fully connected layer FC(·) plays the role of feature conversion. It maps the original traffic data into a new feature space with the dimension of R represents the data matrix, where N is the number of spatial nodes, in this case the number of days of the week N = N w =7.

[0091] Considering the daily time resolution, let N d = 288, representing the number of time scales in a day. For each time series, we first use the weekly embedding mapping matrix W t and the daily embedding mapping matrix D t These two time codes are used to extract the corresponding day of the week and intraday time features and map these features into the weekly embedding Japanese Embed Then and These features form a periodic embedding E through the ∥ operation p The __init__ operation combines two matrices into one matrix.

[0092] In addition, the adaptive embedding E is introduced a , the adaptive embedding is defined by trainable parameters, the initial values ​​are randomly initialized, and the dimensions are In terms of time, a time frame in a traffic time series should show high similarity with its adjacent time frames. This temporal continuity is an indispensable part of traffic flow prediction, as traffic flow usually shows obvious short-term autocorrelation.

[0093] From a spatial perspective, data captured by sensors at different locations often reflect different traffic patterns. For example, the traffic flow in the city center may be very different from that in the suburbs. a The spatial part of E is designed to understand and integrate these geospatial differences. Although each sensor may observe traffic flows with unique temporal dynamics, a Designed to be sharable across different data sources, this means that regardless of the diversity of the data sources, the model can make effective predictions by learning common spatiotemporal features. a When processing data from different sensors, there is no need to relearn completely independent parameters for each case, which improves the generalization ability and training efficiency of the model. a Processing spatiotemporal data in a unified framework simplifies the complexity of the model while ensuring flexibility and accuracy in handling various traffic scenarios. On this basis, a comprehensive spatiotemporal feature representation Z is constructed by concatenating these different embeddings:

[0094] Z=E f ∥E p ∥E a

[0095] Ultimately, this integrated feature expression helps the model achieve higher accuracy in traffic prediction tasks. In this way, the model's ability to interpret and predict traffic time series is further improved.

[0096] Frequency enhancement layer: After obtaining the output Z of the spatiotemporal adaptation layer, an effective expression of the time series can be obtained, but these data are more information in the time domain, and the time series frequency is an indispensable information in the time series. Real-world data sets usually contain rich frequency information, which makes better use of the capabilities of deep learning models. However, most Fourier transform-based methods use Fourier transform to obtain frequency information and use inverse Fourier transform to reconstruct time information to avoid complex number training, which introduces new computational complexity. In addition, recent studies have shown that the frequency domain information of real-world data sets is concentrated in the low-frequency range, while the implicit periodicity of the discrete Fourier transform (DFT) leads to boundary discontinuities and produces significant high-frequency content, a phenomenon known as the Gibbs phenomenon. After quantization, the Gibbs phenomenon causes the boundary points to present erroneous values. Therefore, the discrete cosine transform (DCT) is used to introduce frequency information, so that the Gibbs phenomenon and inverse transform operations can be fundamentally avoided.

[0097] The discrete cosine transform can essentially avoid the Gibbs phenomenon caused by periodic boundary processing in the discrete Fourier transform and its inverse transform, and has higher energy compression efficiency than the Fourier transform. The discrete cosine transform is essentially equivalent to the discrete Fourier transform of the real even function. Because the discrete cosine transform uses symmetrical expansion when extending the period, according to the formula:

[0098]

[0099] Gap represents the lowest frequency component channel representation. Represents the lowest frequency component in one-dimensional DCT, L S Represents the total number of samples of the input spatiotemporal feature representation. This is because when performing the transformation, the entire signal needs to be processed to extract the frequency component. In this embodiment, x is the spatiotemporal feature representation Z, obtained by step 2.1, and the subscript i is the sample number count; after a one-dimensional discrete cosine transform, a one-dimensional vector Freq0 is obtained, and similarly, a multidimensional vector Freq n-1 .

[0100] Then the entire frequency channel vector is obtained by stacking:

[0101] Freq=DCT(V)=stack([Freq0,Freq1,…,Freq n-1 ])

[0102] Where Freq i , i = 0, 1, ... n-1 represents the L-dimensional vector after discrete cosine transform, so Freq ∈ R C×L (C represents the number of channels, L represents the length) represents the vector V∈R C×L (V represents the result of performing discrete cosine transform on each channel of Z) The result after performing DCT; stack represents the stacking operation. After obtaining Freq, the attention weight can be learned through the neural network structure. The entire frequency enhancement layer framework can be expressed as a combination of discrete cosine transform enhancement and attention mechanism:

[0103] F c-att =σ(W2δ(W1DCT(V)))

[0104] The formula indicates that after the input vector V is transformed by DCT, it is processed by two linear transformations W1 and W2, the nonlinear activation function δ, and the linear activation function σ, and the final output is F c-att It is the result of applying the activation function σ to W2 after transformation, representing the attention weight of each channel. These weights can be used to weight the input features, thereby strengthening important features and suppressing unimportant features.

[0105] Through this method, the features of each channel interact with its corresponding frequency components, and the key time information is fully captured from the frequency domain. This process greatly enriches the network's feature extraction capabilities. In the subsequent experimental part, the visualization of the frequency channel attention tensor will be shown to show the importance of each channel learned in the frequency domain and the relative importance of different frequency components in each channel. This method significantly improves the performance of the time series model by enhancing the extraction and utilization of frequency information, enabling it to show better prediction capabilities in various practical applications, especially in complex traffic flow prediction problems. The output of the spatiotemporal adaptive embedding layer is combined as input to further improve the overall effect of the model.

[0106] Step 3: After passing through the Transformer layer, accept the output of step 2 and perform the following operations simultaneously along the spatial and temporal dimensions:

[0107] Step 3.1: Obtain output through a multi-scale trend-aware attention layer along the time dimension. Replace Q, K, and V in the attention mechanism with a convolution operation from full connection to obtain local information, and use multiple convolution heads to extract simultaneously.

[0108] Step 3.2: Obtain outputs through the attention mechanism along the spatial dimension. Use Transformer layers to capture spatial relationships.

[0109] Step 3.3: Aggregate the output results of 3.1 and 3.2 as the input of step 4.

[0110] Specifically, the Transformer layer is composed of a series of identical layers. The Transformer layer captures the relationship between data from the spatial dimension and the temporal dimension respectively, and finally merges the output results and sends them to the regression layer to obtain the final result.

[0111] Multi-scale trend-aware attention layer (starting from the time dimension): Self-attention is a specific attention mechanism in which queries, keys, and values ​​come from symbolic representations of the same sequence. In practice, multi-head self-attention is the most commonly used form of self-attention, which can process information from different representation subspaces simultaneously. The core operation of multi-head self-attention is scaled dot product attention, which represents similarity. The greater the similarity, the stronger the correlation between the data. All queries, keys, and values ​​are representations of the same sequence. The method first linearly maps queries, keys, and values ​​to different representation subspaces, and then calculates the attention function in parallel. Finally, all outputs are concatenated together and further projected to generate the desired output. Its formal definition is as follows:

[0112]

[0113] head j =Attention(QW j Q ,KW j K ,VW j V )

[0114] Among them, Q, K and V represent query, key and value respectively, and W j Q ,W j K ,W j V is the projection matrix on Q, K, V, W O is the final output projection matrix, Indicates a merge operation, MHSelfAttention indicates a multi-head attention mechanism, and Attention indicates a single head jPerform attention operations. For the time dimension, most of the existing time series prediction models directly use a multi-head self-attention mechanism. This type of attention mechanism performs well when processing discrete data such as natural language. However, in continuous time series data, this method cannot effectively identify and utilize local time trends and patterns of the data. For example, in traffic flow data, the flow data at different time points may be consistent, but the flow may vary significantly due to events in a short period of time (such as traffic accidents or road blockages). The traditional self-attention mechanism will mistakenly believe that the two points are related and ignore the local information before and after. This requires the model to be able to perceive and adapt to such local changes. In order to solve the above problems, the present invention designs a multi-scale time trend-aware attention mechanism, which obtains contextual information through convolution operations, and uses different convolution heads to obtain information at different scales. Its formal definition is as follows:

[0115]

[0116] The calculation of each head can be expressed as:

[0117]

[0118] Among them, Q, K and V represent query, key and value respectively, d represents the size of the convolution kernel, They represent convolution operations respectively, and the size of the kernel represents the range of convolution kernel convolution. is the projection matrix on V, W O is the final output projection matrix.

[0119] The spatiotemporal trend-aware self-attention mechanism is implemented through the following key steps to deeply capture and understand the complex dynamics in time series data:

[0120] Trend-aware attention weights: When calculating the weights of self-attention, in addition to the traditional content-based weights (calculated by the relationship between query, key, and value), local trend-based weights are introduced. 1D Convolution is applied to capture the local context before and after each time point, which is a convolution operation on one-dimensional data (such as time series). It slides a convolution kernel (filter) over the input sequence to extract local features. The convolution kernel performs a weighted sum operation at each position, and the result is a new feature map that captures the local pattern in the input data. The size of the convolution kernel determines the size of the captured context window. By applying convolution to time series, the model can learn a certain amount of contextual information before and after each time point. For example, if the convolution kernel size is 3, the model will consider the value of the current time point and the time point before and after. . This approach allows the model to consider not only their immediate similarity but also their local change trend in the time series when evaluating the relevance of two time points.

[0121] Multi-head trend perception mechanism: Drawing on the ideas of the traditional multi-head attention mechanism, the trend-aware self-attention also uses multiple parallel attention heads, each of which captures different features of the sequence through an independent one-dimensional convolution kernel. This design enables the model to have higher sensitivity and discrimination ability for various patterns in the time series, thereby improving the diversity of information and the depth of processing.

[0122] Multi-scale information acquisition: The model uses one-dimensional convolution kernels of different sizes (such as 3, 5, 7) to capture the multi-scale features of the time series. Small-sized convolution kernels can capture tiny time changes, while larger-sized convolution kernels help grasp more macroscopic time trends. This is similar to the application in image processing, where filters of different sizes are used to extract features at different levels. The features captured by convolution kernels of various sizes reflect the dynamic characteristics at different time scales, thus forming a comprehensive understanding from short-term fluctuations to long-term trends. The model integrates these features along specific dimensions (such as feature or channel dimensions) to form a comprehensive feature representation to ensure that information at multiple time scales is fully considered when making decisions.

[0123] Comprehensive processing of global and local information: Through the trend-aware self-attention mechanism, the model can simultaneously process and fuse global information and local trends. The output of this comprehensive understanding not only reflects the long-term dependencies in the time series, but also responds to immediate events. This mechanism is particularly suitable for application scenarios that need to respond to factors affecting multiple time scales (such as traffic flow prediction), because the dynamic changes in such scenarios are usually affected by both recent events and long-term trends.

[0124] Spatial attention layer: In the spatial dimension, the existing Transformer layer is directly used to capture spatial relationships. Specifically, simple transformers are applied along the temporal and spatial axes to capture complex traffic relationships. Given a hidden spatiotemporal representation Where T represents the number of time frames, N represents the number of spatial nodes, and d h is the dimension of the hidden layer. The query, key, and value matrices are obtained through the spatial transformer layer as shown below:

[0125]

[0126] in, is a learnable parameter matrix, where Q, K, and V represent query, key, and value, respectively, and the query matrix Q is obtained. (sp) , the bond matrix K (sp) , value matrix V (sp) .

[0127] Next, calculate the self-attention score Z (sp) , the operation of the spatial transformer layer is as follows:

[0128] Z (sp) =SelfAttention(Q (sp) ,K (sp) ,V (sp) )

[0129] Among them, SelfAttention represents the self-attention mechanism, which follows the following calculation steps:

[0130]

[0131] Among them, d k is the dimension of the key vector, usually the same as d h Similarly, Q represents query, K represents key, and V represents value. The dot product operation is used to obtain the similarity, normalize the similarity, and use the softmax function to convert it into a probability distribution to obtain the attention weight. The attention weight is used to perform weighted summation on the value vector (V) to obtain the final output.

[0132] The complete operation of the spatial transformer layer consists of the following steps:

[0133] In order to enhance the expressiveness of the model, a multi-head attention mechanism is adopted. Specifically, assuming there are H attention heads, the dimension of each head is The calculation process of MultiHead is as follows:

[0134] MultiHead(Q,K,V)=Concat(head1,head2,…,head H )W O

[0135] Concat means concatenating the results of multiple headers and finally adding them to the mapping matrix W. O Multiply to get the final result.

[0136] The calculation for each head is:

[0137]

[0138] is the learnable parameter of the ith attention head, is the output weight matrix.

[0139] After the multi-scale time trend-aware attention mechanism in step 3.1 and the multi-head attention mechanism in step 3.2, residual connections are performed and layer normalization is performed, respectively. The formula is as follows:

[0140] Z MT =LayerNorm(Z+MSTTHead d,j (Q,K,V)

[0141] Z MH =LayerNorm(Z+MultiHead(Q (sp) ,K (sp) ,V (sp) ))

[0142] LayerNorm is a normalization operation performed on the input of each layer, which normalizes the features of each sample; MT is the output of the multi-scale time trend perception attention mechanism after the above operations, Z MH is the output of the multi-head attention mechanism after the above operations;

[0143] Then apply the feed-forward neural network FFN separately, and add residual connections and layer normalization again:

[0144] Z MT′ =LayerNorm(Z MT +FFN(Z MT ))

[0145] Z MH′ =LayerNorm(Z MH +FFN(Z MH ))

[0146] Among them, the form of the feedforward neural network is:

[0147] FFN(x)=ReLU(xW1+b1)W2+b2

[0148] W1, W2 and b1, b2 are learnable parameters, x is a variable, in this example, Z MT′ and Z MH′ ;

[0149] Then Z MT′ and Z MH′ Splice to get the final output

[0150] Step 4: Get the final output through the regression layer.

[0151] Using the output Z of the spatiotemporal transformer layer ′ to generate predictions. The regression layer can be expressed as:

[0152]

[0153] in, is the prediction result, T ′ is the predicted time range, and d is the dimension of the output feature. In this embodiment, d = 1. Therefore, the fully connected layer (FC) converts Z ′ The dimensions in T×N×d h Return to T ′ ×N×d.

[0154] Step 5: Actual Training:

[0155] Step 5.1: Preprocess the historical traffic flow data, including removing outliers and normalizing.

[0156] Step 5.2: Determine the input length and output length. Generally speaking, the input length and output length are the same length. For example, if the historical time interval is 1 minute, the input length is 90, and the output length is 90, then the input data is the data of the first 90 minutes, and the output is the data of the next 90 minutes.

[0157] Step 5.3: Split the training set, test set, and validation set. The ratio is usually 7:1:2 or 6:2:2.

[0158] Step 5.4: Model training. The deployed model is trained using the training set, and the test set is used as the basis for tuning and the test set is used for effect verification.

[0159] Experimental results

[0160] As shown in Table 1, on all four data sets, the method provided by the present invention achieved better performance in most indicators. In the inventive method, attention was first paid to the previous model's lack of ability to capture different frequency patterns in the data, and discrete cosine transform (DCT) was used to enhance the channel attention mechanism, avoiding the high-frequency noise problem caused by Fourier transform methods (such as the Gibbs phenomenon). This method enriches the model's understanding of the periodic components in the data and improves the model's predictive ability. This study supplements spatiotemporal embedding by presenting the periodic behavior in traffic data in detail. On the other hand, a multi-scale trend-aware attention mechanism is used to solve the problem that the traditional attention mechanism cannot utilize contextual information, and a multi-scale model is used to enable the model to have the ability to learn temporal patterns of different scales, from local changes to broader trends.

[0161] Table 1 is a comparison of the effects of the method of the present invention and the reference algorithm

[0162]

[0163]

[0164] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention, characterized in that: The following steps are involved: Step 1: Obtain input data, specifically pre-processed traffic flow data; Step 2: Input traffic flow data into the spatiotemporal frequency adaptive embedding layer to obtain feature embedding weights; wherein the spatiotemporal frequency adaptive embedding layer includes a serial spatiotemporal adaptive embedding layer and a frequency enhancement layer, as follows: Step 2.1: First, the spatiotemporal feature representation is obtained through the spatiotemporal adaptive embedding layer; specifically, the fully connected layer is applied to the traffic flow data to obtain the feature embedding E f , using time information to obtain periodic embedding E p , define the trainable E a As an adaptive embedding, the three embeddings are finally concatenated to obtain the spatiotemporal feature representation; Step 2.2: Then the embedding is enhanced through the frequency enhancement layer; the frequency information in the spatiotemporal feature representation is enhanced using discrete cosine transform to obtain the final feature embedding weight; Step 3: The output of step 2 passes through the Transformer layer. Specifically, the Transformer layer includes a parallel multi-scale trend perception attention layer based on the time dimension and an attention mechanism layer based on the spatial dimension. The feature embedding weights are performed simultaneously along the spatial dimension and the time dimension, and finally the output results of the two dimensions are aggregated. The details are as follows: Step 3.1: Obtain output through a multi-scale trend-aware attention layer based on the time dimension; specifically, replace the fully connected operation in the attention mechanism with a convolution operation to obtain local information, and use multiple convolution heads to extract simultaneously; Step 3.2: Based on the spatial dimension, use the Transformer layer in the existing technology to capture the spatial relationship; Step 3.3: Aggregate the output results of 3.1 and 3.2 as the input of step 4; Step 4: Get the final output through the regression layer.

2. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention according to claim 1 is characterized in that: The step 2.1 is as follows: The traffic flow data obtained is represented by X t-T+1:t , where t represents the current time and T represents the length of historical data; a fully connected layer is applied to the original traffic data to obtain the feature embedding E f , which is expressed as follows: E f =FC(X t-T+1:t ); Using Zhou embedding mapping matrix W t and the daily embedding mapping matrix D t These two time codes are used to extract the corresponding day of the week and intraday time features and map these features into the weekly embedding Japanese Embed Then and These features form a periodic embedding E through the ∥ operation p ; The specific operation of ∥ is to merge two matrices into the same matrix; Adaptive Embedding a Defined by trainable parameters, the initial values ​​are randomly initialized; Finally, the above three embeddings are concatenated to obtain the spatiotemporal feature representation Z: Z=E f ∥E p ∥E a 。 3. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention according to claim 1 is characterized in that: The process of embedding through frequency enhancement layer enhancement is as follows: First, perform discrete cosine transform on the spatiotemporal feature representation. According to the formula: Among them, Gap represents the lowest frequency component channel representation, Represents the lowest frequency component in one-dimensional DCT, L S Represents the total number of samples represented by the input spatiotemporal feature; x is a variable, referring to the spatiotemporal feature representation, i is the sample count, and after a one-dimensional discrete cosine transform, a one-dimensional vector Freq0 is obtained. Similarly, a multidimensional vector Freq n-1 ; Then the entire frequency channel vector is obtained by stacking: Freq=DCT(V)=stack([Freq0,Freq1,…,Freq n-1 ]); Where Freq i , i = 0, 1, ... n-1 represents the L-dimensional vector after discrete cosine transform, so Freq ∈ R C×L Represents the result of discrete cosine transform of vector V, where C represents the number of channels, L represents the length, and vector V represents the result of discrete cosine transform of each channel of spatiotemporal feature representation Z; stack represents stacking operation; Finally, after linear transformation and activation function processing, the attention weight F of each channel is obtained c-attn ; The specific formula is as follows: F c-a =σ(W2δ(W1DCT(V))); Where W1 and W2 represent linear transformations, δ is a nonlinear activation function, and σ is a linear activation function.

4. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention according to claim 1 is characterized in that: The step 3.1 is as follows: A multi-scale time-trend-aware attention mechanism is used to obtain contextual information through convolution operations, and different convolution heads are used to obtain information at different scales, which is formalized as follows: Among them, MSTTSelfAttention represents the multi-scale trend-aware attention mechanism, Q, K, and V represent query, key, and value respectively, and d represents the size of the convolution kernel. represents the merge operation, W O is the final output projection matrix; MSTTHead d,j Representing a single head in multiple scales, the calculation of each head can be expressed as: They represent the corresponding convolution operations respectively. It is the projection matrix on the value V; the size of the convolution kernel d indicates the range of the convolution kernel convolution.

5. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention according to claim 4 is characterized in that: The step 3.2 is as follows: Given a hidden spatiotemporal feature representation Where T represents the number of time frames, N represents the number of spatial nodes, and d h is the dimension of the hidden layer, and the query Q, key K, value V matrix is ​​obtained through the spatial transformer layer as shown below: in, is a learnable parameter matrix, and the query matrix Q is obtained (sp) , the bond matrix K (sp) , value matrix V (sp) ; Then calculate the self-attention score Z (sp) , through the spatial transformer layer operation as follows: Z (sp) =SelfAttention(Q (sp) ,K (sp) ,V (sp) ) Among them, SelfAttention represents the self-attention mechanism, which follows the following calculation steps: Among them, d k is the dimension of the key vector, and the dot product operation is used to obtain the similarity. The similarity is normalized and converted into a probability distribution using the softmax function to obtain the attention weight. The attention weight is used to perform a weighted summation on the value V to obtain the final output.

6. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention according to claim 5 is characterized in that: Replace the self-attention mechanism in step 3.2 with a multi-head attention mechanism; assuming there are H attention heads, the calculation process of MultiHead is as follows: MultiHead(Q,K,V)=Concat(head1,head2,…,head H )W O Concat means concatenating the results of multiple headers and finally adding them to the mapping matrix W. O Multiply to get the final result; The calculation for each head is: head i =SelfAttention(QW i Q ,KW i K ,VW i V ) is the learnable parameter of the i-th attention head.

7. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention according to claim 6 is characterized in that: The step 3.3 is as follows: After the multi-scale time trend-aware attention mechanism in step 3.1 and the multi-head attention mechanism in step 3.2, residual connections are performed and layer normalization is performed, respectively. The formula is as follows: Z MT =LayerNorm(Z+MSTTHead d，j (Q,K,V)) From MH =LayerNorm(Z+MultiHead(Q (sp) ,K (sp) ,In (sp) )) LayerNorm is a normalization operation performed on the input of each layer; Z MT is the output of the multi-scale time trend perception attention mechanism after the above operations, Z MH is the output of the multi-head attention mechanism after the above operations; Then apply the feed-forward neural network FFN separately, and add residual connections and layer normalization again: Z MT′ =LayerNorm(Z MT +FFN(Z MT )) Z MH′ =LayerNorm(Z MH +FFN(Z MH )) Among them, the form of the feedforward neural network is: FFN(x)=ReLU(xW1+b1)W2+b2 W1, W2 and b1, b2 are learnable parameters, and x is a variable; Then Z MT′ and z MM′ The final output Z' is obtained by concatenation.

8. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention according to claim 7 is characterized in that: The regression layer is expressed as follows: in, is the prediction result, T′ is the predicted time range, d is the dimension of the output feature, and FC represents the fully connected layer.

9. The traffic flow prediction method based on spatiotemporal frequency adaptive embedding and multi-scale trend perception attention according to claim 8 is characterized in that: The actual training of the method is as follows: Step 5.1: Preprocess the historical traffic flow data, including removing outliers and normalizing; Step 5.2: Determine the input length and output length; Step 5.3: Split the training set, test set, and validation set according to the set ratio; Step 5.4: Model training: Use the training set to train the deployed model, use the test set as the basis for tuning, and use the test set as the effect verification.

Images