High-speed traffic flow prediction method combining self-attention mechanism and state space model

Abstract

The invention relates to a high-speed traffic flow prediction method combining a self-attention mechanism and a state space model, belongs to the field of intelligent traffic systems, and aims at reducing the calculation complexity of capturing a long-range dependency relationship by introducing the state space model and modeling the dynamic change of a time dimension through the self-attention mechanism. In order to further improve the performance of the model, the invention also designs a learnable space-time embedding module which can adaptively learn unique traffic characteristics of different highway nodes and time periods, including multi-dimensional information such as traffic flow, time, date, node position and the like, so that the prediction precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of highway traffic flow prediction technology, and in particular to a highway traffic flow prediction method combining a self-attention mechanism with a state space model. The method can effectively capture the spatiotemporal dependency of highway traffic flow data, improve the accuracy and computational efficiency of traffic flow prediction, and belongs to the field of intelligent transportation systems. Background Art

[0002] With the continuous expansion of the highway network and the rapid increase in the number of vehicles, highway traffic management faces increasingly severe challenges, especially when traffic flow increases dramatically and traffic congestion occurs frequently. As a key technology in intelligent transportation systems, traffic flow prediction can provide important decision-making support for traffic management departments, effectively alleviate traffic congestion problems, and improve traffic safety and traffic efficiency.

[0003] Highway traffic flow is usually collected by sensors deployed on the road and is presented as typical time series data. However, highway traffic flow data is obviously nonlinear and dynamic, and is affected by many factors such as holidays, weather, and emergencies, which brings great challenges to traffic flow prediction.

[0004] Traditional traffic flow prediction methods mainly rely on statistical models and machine learning methods, such as historical average method, autoregressive integrated moving average model and support vector regression, etc. These methods have great limitations when dealing with complex nonlinear and dynamic changes of highways, resulting in low prediction accuracy.

[0005] In recent years, advances in deep learning technology have brought new solutions to traffic flow prediction. Researchers have tried to solve the problem of highway traffic flow prediction through graph neural networks and attention mechanisms. Since highways have a natural network structure, graph neural networks are widely used to model the dependencies between traffic nodes. Graph neural networks can aggregate information between adjacent nodes through graph convolution and capture short-range dependencies in space. However, the ability to model long-range dependencies between remote nodes is relatively weak, especially in widely distributed scenarios such as highways, making it difficult to effectively capture the global characteristics of the overall traffic flow.

[0006] In recent years, the attention mechanism has achieved remarkable results in fields such as computer vision and natural language processing, and has also been introduced into traffic flow prediction. The attention mechanism can adaptively assign weights between different time steps and is suitable for modeling temporal dependencies in time series data. However, the attention mechanism has a high computational complexity when processing long sequence data, especially in highway scenarios with many sensor nodes and huge amounts of data. The computational cost is relatively high, which affects the real-time performance and efficiency of the model.

[0007] In order to deal with the above problems, existing research has also introduced solutions based on state space models. An emerging state space model uses a selective state space mechanism to more efficiently capture relevant information in long sequence data, which is particularly suitable for long-range dependency modeling and global feature extraction in high-speed traffic flow. However, the application of state space models in high-speed traffic flow prediction in the existing technology is still insufficient. Summary of the invention

[0008] The present invention relates to a model for high-speed traffic flow prediction, which aims to improve the spatiotemporal dependency modeling capability in large-scale traffic systems while optimizing computational efficiency and prediction accuracy. In view of the shortcomings of the models in the prior art, such as the limitations in long-range dependency modeling and computational complexity, the present invention proposes a traffic flow prediction model with both high efficiency and prediction performance by designing a stacked spatiotemporal block structure. The specific technical solution is as follows:

[0009] A high-speed traffic flow prediction method combining self-attention mechanism and state-space model adopts stacked spatiotemporal blocks as core modules; each spatiotemporal block consists of two time blocks and one space block, and the space block is sandwiched between the two time blocks; the time block is used to capture the temporal dependency in traffic data, and the space block is used to capture the spatial dependency between different sensor nodes;

[0010] The specific prediction method includes the following steps:

[0011] 1) Input Embedding Layer

[0012] Step 1.1: Convert the raw data into a high-dimensional representation; the raw data includes traffic flow, time information, and sensor node information;

[0013] 1.1.1 Traffic flow data Convert to high-dimensional representation through fully connected layers

[0014] X data =FC(X)

[0015] 1.1.2 In order to consider the different traffic patterns between weekdays and weekends, a learnable parameterization matrix is ​​introduced Convert the day of the week of time information into a high-dimensional representation X day ,Each row in the matrix corresponds to a high-dimensional representation of a specific day of the week, which is used to capture the changes in traffic patterns between weekdays and non-weekdays;

[0016] In order to capture the changes in traffic patterns between day and night, another learnable parameterized matrix is ​​introduced Convert the time of day of temporal information into a high-dimensional representation X time ;

[0017] 1.1.3 Considering the unique traffic characteristics of each sensor node, a learnable parameterized tensor is introduced The time and node information are embedded to express the unique traffic characteristics of each sensor node and the relative position between time steps; this tensor not only reflects the specific traffic characteristics of each node, but also contains the relative position information of the time step. The addition of relative position information is necessary because traffic patterns are affected not only by absolute time but also by the intervals between observations. By capturing these intervals, the model can better understand the time dependency and periodicity in traffic data. Therefore, each node is mapped to a high-dimensional representation X at each time position. pn ;

[0018] Step 1.2: Concatenate the embedded data obtained above to get the output of the input embedding layer:

[0019] X emb =Concat(X data ,X day ,X time ,X pn )

[0020] in d h =4d. By combining X data , X day , X time and X pn The model can provide a comprehensive embedding form that contains the core features required for accurate traffic prediction. By integrating these diverse traffic data features, the model can effectively learn and generalize complex patterns in the input data, thereby improving performance in various traffic prediction tasks. emb is input into the space-time block; for convenience, X is used instead of X below emb ;

[0021] 2) Space-time block calculation

[0022] Step 2.1 There is a dependency between traffic conditions at different time steps. In the time block, the time dependency in traffic data is modeled through the self-attention mechanism; the traffic data of each sensor node at different time steps is mapped to query, key and value to generate query matrix, key matrix and value matrix. Represents the data of node n at all time steps;

[0023] Q n =X :n: W Q

[0024] Kn =X :n: W K

[0025] V n =X :n: W V

[0026] in is a learnable parameter. Then calculate the attention score:

[0027]

[0028] Finally, the output of the attention mechanism is obtained by the following formula:

[0029] Attention(Q n ,K n ,V n )=A n V n

[0030] For each node n, the attention score matrix A n is unique, which enables the attention mechanism to capture dynamic temporal dependencies specific to each node. In addition, it can effectively represent long-range dependencies and capture complex interactions between arbitrary time steps. The output of the time block It is obtained through a fully connected feedforward neural network. This network helps to further process and optimize the high-dimensional representation generated by the attention mechanism. In order to ensure the stability and efficiency of the training process, residual connections are introduced to help alleviate the gradient disappearance problem.

[0031] Step 2.2 In the spatial block, a state-space model is used to capture spatial dependencies; the reason for this choice is that the traffic system contains a large number of sensor nodes, and the time complexity of the self-attention mechanism grows quadratically with the increase in the number of nodes. In contrast, the state-space model has a lower linear time complexity while still being able to effectively model long-term dependencies. Therefore, the state-space model is more efficient and scalable in dealing with spatial dependencies in large-scale traffic systems.

[0032] The output of the time block calculation first passes through the linear layer, the convolutional layer, and then enters the state space model, and finally multiplied by the gated nonlinear unit. Represents the data of all nodes at time t. t:: After inputting the state space model, we get SSM(H t:: ), which captures the dependencies between different nodes. Similar to the temporal block, the state-space model is followed by a fully connected feed-forward network to obtain the output H of the spatial block s .

[0033] Step 2.3: Transform the output H of the spatial block s Then input it into another time block based on the self-attention mechanism to generate the final output H of the spatiotemporal block;

[0034] 3) Output and loss function calculation

[0035] Step 3.1 uses skip connection to convert the output of each spatiotemporal block into the prediction result of that layer through linear transformation, and adds the prediction results of all layers to obtain the final prediction value;

[0036]

[0037] Among them, H l is the output of the l-th layer spatiotemporal block, W l is the linear transformation matrix.

[0038] Step 3.2 Predicted value The mean squared error with the true value Y is used as the loss function;

[0039]

[0040] Through this design, the model can integrate the outputs of spatiotemporal blocks at each layer to improve prediction accuracy, and optimize model parameters through loss functions to improve performance.

[0041] The traffic flow prediction method of the present invention takes multimodal data such as basic traffic flow, time information, and sensor node location, and converts these multidimensional data into high-dimensional representations through the input embedding layer to capture the complex features therein. Specifically, the input embedding part converts traffic flow data, day of the week, time period, and sensor node information into high-dimensional vectors, and generates embedded representations of these information through a fully connected network, a learning matrix, and a learnable tensor, thereby providing a basis for subsequent spatiotemporal dependency modeling. The model also takes into account the relative positions between time steps to better capture the periodicity and temporal dependencies of traffic flow.

[0042] In order to more effectively capture the temporal dependency of traffic flow, the present invention adopts a self-attention mechanism in the temporal part of the spatiotemporal block. By performing query, key and value mapping operations on the traffic data of each sensor node at different time steps, the model can adaptively focus on the relationship between different time points, thereby modeling short-term and long-term temporal dependencies. The self-attention mechanism has a unique advantage in capturing long-range dependencies and can effectively cope with the complexity of high-speed traffic flow.

[0043] However, the time complexity of the self-attention mechanism is high, especially when facing a large number of sensor nodes in high-speed scenarios, the computational cost will increase significantly. In order to solve this problem, the present invention introduces a state-space model in spatial dependency modeling. The spatial dependency relationship between sensor nodes is efficiently modeled through linear transformation, convolutional layer and state-space model. Compared with the self-attention mechanism, the state-space model has linear time complexity, which greatly reduces the computational cost of the model, while retaining the ability to model long-range spatial dependencies, making it more suitable for processing large-scale transportation systems.

[0044] The present invention gradually extracts spatiotemporal dependency information by stacking multiple spatiotemporal block layers, and ensures the stability of the model training process and the effective transmission of deep features through residual connections. At the same time, in the output part, the model adopts a skip connection mechanism to fuse the prediction results of each layer to further improve the final prediction accuracy.

[0045] Finally, the present invention uses mean square error as the loss function to optimize the difference between the model output and the actual traffic flow, thereby achieving accurate prediction of future traffic flow. Through multi-level feature extraction and efficient computing structure, the present invention significantly improves the prediction ability in high-speed traffic flow scenarios, and is suitable for traffic management and intelligent transportation applications in actual traffic systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a model framework diagram of the time block based on the self-attention mechanism of the present invention;

[0047] Figure 2 It is a model framework diagram of the space block based on the state space model of the present invention;

[0048] Figure 3 It is the overall framework diagram of the model of the present invention;

[0049] Figure 4 This is a training flow chart of the model of the present invention. DETAILED DESCRIPTION

[0050] In order to better understand the present invention, the invention content will be described in detail below in conjunction with specific embodiments. It should be noted that these embodiments are only for illustrating the technical solutions of the present invention and do not limit the scope of the present invention.

[0051] A high-speed traffic flow prediction method combining self-attention mechanism and state-space model adopts stacked spatiotemporal blocks as core modules; each spatiotemporal block consists of two time blocks and one space block, and the space block is sandwiched between the two time blocks; the time block is used to capture the temporal dependency in traffic data, and the space block is used to capture the spatial dependency between different sensor nodes;

[0052] The specific prediction method includes the following steps:

[0053] 1) Input Embedding Layer

[0054] Step 1.1: Convert the raw data into a high-dimensional representation; the raw data includes traffic flow, time information, and sensor node information;

[0055] 1.1.1 Traffic flow data Convert to high-dimensional representation through fully connected layers

[0056] X data =FC(X)

[0057] 1.1.2 In order to consider the different traffic patterns between weekdays and weekends, a learnable parameterization matrix is ​​introduced Convert the day of the week of time information into a high-dimensional representation X day ,Each row in the matrix corresponds to a high-dimensional representation of a specific day of the week, which is used to capture the changes in traffic patterns between weekdays and non-weekdays;

[0058] In order to capture the changes in traffic patterns between day and night, another learnable parameterized matrix is ​​introduced Convert the time of day of temporal information into a high-dimensional representation X time ;

[0059] 1.1.3 Considering the unique traffic characteristics of each sensor node, a learnable parameterized tensor is introduced The time and node information are embedded to express the unique traffic characteristics of each sensor node and the relative position between time steps; this tensor not only reflects the specific traffic characteristics of each node, but also contains the relative position information of the time step. The addition of relative position information is necessary because traffic patterns are affected not only by absolute time but also by the intervals between observations. By capturing these intervals, the model can better understand the time dependency and periodicity in traffic data. Therefore, each node is mapped to a high-dimensional representation X at each time position. pn ;

[0060] Step 1.2: Concatenate the embedded data obtained above to get the output of the input embedding layer:

[0061] X emb =Concat(X data ,X day ,X time ,X pn )

[0062] in d h =4d. By combining Xdata , X day , X time and X pn The model can provide a comprehensive embedding form that contains the core features required for accurate traffic prediction. By integrating these diverse traffic data features, the model can effectively learn and generalize complex patterns in the input data, thereby improving performance in various traffic prediction tasks. emb is input into the space-time block; for convenience, X is used instead of X below emb ;

[0063] Space-time block computing

[0064] Step 2.1 There is a dependency between traffic conditions at different time steps. In the time block, the time dependency in traffic data is modeled through the self-attention mechanism; the traffic data of each sensor node at different time steps is mapped to query, key and value to generate query matrix, key matrix and value matrix. Represents the data of node n at all time steps;

[0065] Q n =X :n: W Q

[0066] K n =X :n: W K

[0067] V n =X :n: W V

[0068] in is a learnable parameter. Then calculate the attention score:

[0069]

[0070] Finally, the output of the attention mechanism is obtained by the following formula:

[0071] Attention(Q n ,K n ,V n )=A n V n

[0072] For each node n, the attention score matrix A n is unique, which enables the attention mechanism to capture dynamic temporal dependencies specific to each node. In addition, it can effectively represent long-range dependencies and capture complex interactions between arbitrary time steps. The output of the time block It is obtained through a fully connected feedforward neural network. This network helps to further process and optimize the high-dimensional representation generated by the attention mechanism. In order to ensure the stability and efficiency of the training process, residual connections are introduced to help alleviate the gradient disappearance problem.

[0073] Step 2.2 In the spatial block, a state-space model is used to capture spatial dependencies; the reason for this choice is that the traffic system contains a large number of sensor nodes, and the time complexity of the self-attention mechanism grows quadratically with the increase in the number of nodes. In contrast, the state-space model has a lower linear time complexity while still being able to effectively model long-term dependencies. Therefore, the state-space model is more efficient and scalable in dealing with spatial dependencies in large-scale traffic systems.

[0074] The output of the time block calculation first passes through the linear layer, the convolutional layer, and then enters the state space model, and finally multiplied by the gated nonlinear unit. Represents the data of all nodes at time t. t:: After inputting the state space model, we get SSM(H t:: ), which captures the dependencies between different nodes. Similar to the temporal block, the state-space model is followed by a fully connected feed-forward network to obtain the output H of the spatial block s .

[0075] Step 2.3: Transform the output H of the spatial block s Then input it into another time block based on the self-attention mechanism to generate the final output H of the spatiotemporal block;

[0076] Output and loss function calculation

[0077] Step 3.1 uses skip connection to convert the output of each spatiotemporal block into the prediction result of that layer through linear transformation, and adds the prediction results of all layers to obtain the final prediction value;

[0078]

[0079] Among them, H l is the output of the l-th layer spatiotemporal block, W l is the linear transformation matrix.

[0080] Step 3.2 Predicted value The mean squared error with the true value Y is used as the loss function;

[0081]

[0082] Through this design, the model can integrate the outputs of spatiotemporal blocks at each layer to improve prediction accuracy, and optimize model parameters through loss functions to improve performance.

[0083] The traffic flow prediction model proposed in this paper adopts stacked spatiotemporal blocks as the core module. Each spatiotemporal block consists of two time blocks and one space block, and the space block is sandwiched between the two time blocks. The time block is used to capture the temporal dependency in traffic data, and the space block is used to capture the spatial dependency between different sensor nodes. The entire model is composed of multiple such spatiotemporal blocks stacked together, extracting more complex spatiotemporal features layer by layer.

[0084] The purpose of the input embedding layer is to transform the raw input data into a high-dimensional representation for better subsequent spatiotemporal feature extraction. Specifically, the input data of the model includes historical traffic flow, time information (such as the specific time of day and day of the week), the location of sensor nodes and other data. First, the traffic flow data is transformed into a high-dimensional representation through a fully connected layer. Then, two learnable matrices are introduced to map the day of the week and the time of day information to a high-dimensional space, respectively, so as to capture the differences between weekdays and non-workdays, and day and night traffic patterns. In order to better express the characteristics of each sensor node at different time steps, a learnable tensor is also introduced to embed the information of time and node, reflecting the unique traffic characteristics of the node and the relative position between time steps.

[0085] Each time block models the temporal dependencies in traffic data through a self-attention mechanism. The self-attention mechanism can learn the dynamic relationship between different time steps, thereby effectively capturing long-term and short-term temporal dependency features. For each sensor node, the model generates a matrix of queries, keys, and values ​​based on its traffic data at different time steps. Then, by calculating the correlation between time steps, the model can adaptively focus on those historical time points that are most relevant to the current moment, thereby providing a basis for future traffic predictions.

[0086] Unlike time blocks, spatial blocks are used to capture the spatial dependencies between sensor nodes. In the present invention, the spatial block does not use the traditional self-attention mechanism because its computational complexity will increase quadratically with the increase in the number of nodes, which is not scalable in large-scale transportation systems. Therefore, the present invention uses a state-space model to replace the self-attention mechanism. The state-space model effectively models the spatial dependencies between sensor nodes through linear transformations and convolution operations while maintaining linear time complexity, greatly improving the computational efficiency of the model.

[0087] During model training, residual connections are used to combine the output of each spatiotemporal block with the original input to ensure the effective transmission of deep features while alleviating the gradient vanishing problem. In addition, the model uses a skip connection mechanism to weightedly fuse the intermediate prediction results of each spatiotemporal block to obtain the final prediction output. This design not only improves the stability of the model, but also enhances the accuracy of the prediction.

[0088] The final output of the model is the prediction result of future traffic flow. Through linear transformation, the model maps the output of the spatiotemporal block to the traffic flow prediction value. In order to optimize the prediction ability of the model, the present invention uses the mean square error as the loss function, calculates the difference between the predicted value and the actual traffic flow, and updates the parameters of the model based on this error, so that the model gradually approaches the optimal prediction result.

[0089] The present invention is applicable to real-time traffic flow prediction of large-scale transportation systems, and has significant application value in highway traffic flow management and scheduling. By accurately predicting future traffic flow, traffic management departments can take measures in advance to optimize traffic signals, plan traffic routes, reduce traffic congestion, and improve road utilization. At the same time, the model can also provide support for intelligent transportation systems, such as traffic information services in autonomous driving and vehicle networking systems.

Claims

1. A high-speed traffic flow prediction method combining self-attention mechanism and state-space model, characterized in that stacked spatiotemporal blocks are used as core modules; each spatiotemporal block consists of two time blocks and one space block, and the space block is sandwiched between the two time blocks; the time block is used to capture the temporal dependency in traffic data, and the space block is used to capture the spatial dependency between different sensor nodes; The specific prediction method includes the following steps: 1) Input Embedding Layer Step 1.1: Convert the raw data into a high-dimensional representation; the raw data includes traffic flow, time information, and sensor node information; 1.1.1 Convert traffic flow data into high-dimensional representation through a fully connected layer; 1.1.2 Introduce a learnable parameterized matrix to convert the day of the week of time information into a high-dimensional representation to capture the changes in traffic patterns between weekdays and non-workdays; introduce another learnable parameterized matrix to convert the time of day of time information into a high-dimensional representation to capture the changes in traffic patterns between day and night; 1.1.3 Introduce a learnable parameterized tensor that embeds time and node information to express the unique traffic characteristics of each sensor node and the relative position between time steps; Step 1.2: Concatenate the embedded data obtained above to get the output of the input embedding layer, and then input it into the spatiotemporal block; 2) Space-time block calculation Step 2.1: In the time chunk, the temporal dependency in the traffic data is modeled through the self-attention mechanism. Perform query, key and value mapping operations on the traffic data of each sensor node at different time steps to generate query matrix, key matrix and value matrix. By calculating the correlation between time steps, adaptively focus on the historical time points that are most relevant to the current moment, and model short-term and long-term time dependencies. Then, through a fully connected feedforward network, the output of the time block is obtained. Step 2.2: In the spatial block, the state space model is used to capture the spatial dependency. The output of the temporal block calculation is passed through a linear layer, a convolutional layer, and a state space model to model the spatial dependency between sensor nodes. The output of the spatial block is then passed through a fully connected feedforward network to obtain the output of the spatial block. Step 2.3: Input the output of the spatial block into another temporal block based on the self-attention mechanism to generate the final output of the spatial-temporal block. 3) Output and loss function calculation Step 3.1 uses skip connections to convert the output of each layer of spatiotemporal blocks into the prediction results of that layer through linear transformation, and adds the prediction results of all layers to obtain the final prediction value.

2. According to claim 1, a high-speed traffic flow prediction method combining self-attention mechanism and state space model is characterized in that: After step 3.1, the model parameters are optimized through the loss function.

3. The high-speed traffic flow prediction method combining the self-attention mechanism and the state space model according to claim 1 is characterized in that: In step 2.1 and step 2.2, residual connections are introduced before the fully connected feed-forward network.

Images