Adaptive space-time coding mask pre-training traffic flow prediction method and device

Abstract

The invention discloses an adaptive space-time coding mask pre-training traffic flow prediction method and equipment. The equipment comprises an upstream mask pre-training auto-encoder and a downstream space-time predictor. In the pre-training stage, long-term time dependence is learned through mask reconstruction; in the prediction stage, multi-scale trend is adopted to perceive time attention and structure to perceive space attention, and traffic flow trend and road network structure features are respectively captured. Adaptive space-time coding is introduced in the two stages to enhance space-time feature extraction. The method effectively learns long-term dependence under low video memory overhead, combines trend and structure perception, significantly improves traffic flow prediction accuracy, and supports accurate operation of an intelligent traffic system.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation, and more specifically, relates to a traffic flow prediction method and device enhanced by adaptive spatiotemporal coding and mask pre-training. Background Technology

[0002] Intelligent Transportation Systems (ITS) play a crucial role in modern smart cities, undertaking the important tasks of predicting, planning, and managing urban traffic. As a core technology of ITS, traffic flow forecasting aims to predict future traffic flow based on historical observation data. Accurate traffic flow forecasting can guide route planning and alleviate traffic congestion, thus attracting extensive research. Nevertheless, current research still has the following two limitations.

[0003] The first limitation is the difficulty in fully modeling long-term traffic flow. Traffic flow exhibits regular, periodic changes in sync with people's daily routines, with a complete cycle lasting up to one week. Most existing traffic flow prediction models are trained end-to-end, and their input lengths are often limited to very small values ​​(typically one hour, 12 steps). Compared to traffic flow cycles lasting up to one week, existing prediction methods can only capture fragmented time segments and may fall into spatiotemporal illusions due to complex spatiotemporal heterogeneity. A spatiotemporal illusion refers to a situation where different input windows of the same sensor have similar traffic sequences, but the traffic sequences differ in the prediction window. A spatial illusion refers to a situation where two sensors located at different locations have similar traffic sequences in the same input window, but the traffic sequences differ in the same prediction window.

[0004] The second limitation is that the performance gains from architectural improvements are diminishing. Deep learning models, such as spatiotemporal graph neural networks and Transformers, have been extensively studied to better capture the spatiotemporal dependencies of traffic flow. Researchers have designed complex graph convolutional networks, attention mechanisms, and other methods for traffic flow prediction. However, on the one hand, as models are stacked, their structures become increasingly complex. On the other hand, the performance gains from architectural improvements are gradually diminishing, exhibiting marginal utility. Therefore, to further improve performance, the research focus needs to shift from designing new architectures to designing more effective representation methods. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an adaptive spatiotemporal coding and mask pre-training enhanced traffic flow prediction method and device, the purpose of which is to make full use of long-term historical traffic flow data to accurately predict future traffic flow and ensure intelligent scheduling and safety management of smart transportation.

[0006] This traffic prediction method consists of an upstream masked pre-trained autoencoder and a downstream spatiotemporal predictor. The masked pre-trained autoencoder comprises an encoder and a decoder, both based on the Transformer model. Partially masked long-term traffic flow is fed into the encoder, and the decoder reconstructs the masked portion, thus providing the spatiotemporal predictor with a contextual representation containing long-term dependencies. The spatiotemporal predictor consists of a temporal Transformer module and a spatial Transformer module. In the temporal Transformer module, a multi-scale trend-aware self-attention mechanism is designed to capture the trend changes in traffic flow. In the spatial Transformer module, the shortest path features between nodes are incorporated into the self-attention mechanism to capture the structural features of the road network. To enhance the model's representational capability, both the masked pre-trained autoencoder and the spatiotemporal predictor use spatiotemporal adaptive coding to learn the intrinsic spatiotemporal correlations of traffic flow.

[0007] To achieve the above objectives, according to one aspect of the present invention, an adaptive spatiotemporal coding and mask pre-training enhancement method for traffic flow prediction is provided, comprising a pre-training stage and a prediction stage: Pre-training stage: S1. Historical traffic flow data of at least one day is processed using a block-based method to perform random masking in the time dimension; S2. The masked data is input into a mask autoencoder for training. Through the block-based processing and time-dimensional masking, a pre-trained temporal encoder specifically designed to extract long-term time dependencies is obtained while reducing memory usage; Prediction stage: S3 adds time, date, road network spatial, and adaptive spatiotemporal coding to the short-term traffic flow data to form coded data; S4 inputs the coded data into a customized downstream predictor for processing: First, the temporal trend features are extracted from the encoded data using a multi-scale trend-aware temporal attention module with one-dimensional convolutional kernels of different scales. Then, the temporal trend features are input into a structure-aware spatial attention module, and spatial structure features are extracted as the output features of the predictor by incorporating road network topology information into the attention calculation in the form of a learnable bias matrix. S5 uses the pre-trained time encoder to extract the time representation of the long-term historical traffic flow data after block processing, which corresponds to the short-term traffic flow data, and fuses it with the features output in step S4 to generate an enhanced representation. S6 integrates the enhanced representation to predict future traffic flow.

[0008] In a specific embodiment of the present invention, the block processing method in step S1 uses a one-dimensional convolution kernel to perform non-overlapping sliding convolution on the historical traffic flow data in the time dimension, dividing the long sequence into a data block sequence of equal length.

[0009] In a specific embodiment of the present invention, in step S3, the adaptive spatiotemporal coding is a learnable tensor initialized by the Xavier method and participating in gradient updates during training, used to adaptively fuse the spatiotemporal context information of traffic flow.

[0010] In one specific embodiment of the present invention, the multi-scale trend-aware temporal attention module includes at least two attention layers, wherein the first layer uses a one-dimensional convolutional kernel of size 1 to generate queries and keys, and the second layer uses a one-dimensional convolutional kernel of size greater than 1 to generate queries and keys.

[0011] In one specific embodiment of the present invention, the size of the one-dimensional convolution kernel with a value greater than 1 is 3 or 5.

[0012] In a specific embodiment of the present invention, in the structure-aware spatial attention module, the road network topology information is the shortest path distance matrix between nodes calculated based on the road network graph structure.

[0013] In one specific embodiment of the present invention, the fusion in step S5 is performed as follows: The last block representation output by the pre-trained temporal encoder after encoding long-term historical data is extracted, projected through a linear layer, and then added element-wise with the features output in step S4.

[0014] In a specific embodiment of the present invention, the pre-training of the mask autoencoder in step S2 and the overall training in the prediction stage are both optimized using the Huber loss function.

[0015] Based on the same inventive concept, the present invention also designs an electronic device, comprising: One or more processors; memory for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement traffic flow prediction methods such as adaptive spatiotemporal coding and mask pre-training enhancement.

[0016] Based on the same inventive concept, the present invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described adaptive spatiotemporal coding and mask pre-training enhanced traffic flow prediction method.

[0017] The advantages of this invention are: (1) A two-stage traffic flow prediction architecture of "pre-training-training". Traffic flow exhibits a periodicity lasting up to one day or even one week, depending on human daily routines. Due to the time and space complexity and end-to-end nature of existing traffic flow prediction technologies, these algorithms incur significant memory overhead when inputting long-term traffic flow, making it difficult to learn long-term time dependencies. This invention divides traffic flow prediction into two stages: "pre-training-training". In the pre-training stage, traffic flow is compressed into blocks using a convolutional network, and the masked parts are reconstructed in a self-supervised manner using a masked autoencoder. This provides the downstream spatiotemporal predictor with the time dependency of long-term historical traffic flow under low memory overhead, helping the downstream spatiotemporal predictor to escape spatiotemporal illusions and improve the accuracy of traffic flow prediction.

[0018] (2) More accurate traffic flow prediction. The original Transformer calculates attention scores point by point, which cannot perceive the trend information over time; the original Transformer can only learn the similarity of road network nodes and cannot perceive the structural features of the road network. This invention designs a multi-scale trend-aware temporal attention, which aggregates the temporal features of adjacent moments through multi-scale convolutional kernels, thereby perceiving the trend changes of traffic flow; and designs a structure-aware spatial attention, which learns the structural features of the road network by introducing the degree matrix of nodes, making up for the shortcomings of the original Transformer in perceiving temporal trends and spatial structures, fully learning the spatiotemporal features of traffic flow, and effectively improving the accuracy of traffic flow prediction.

[0019] (3) More efficient spatiotemporal representation methods. Existing technologies often improve prediction accuracy by stacking models, which increases model complexity, and endless stacking will also encounter performance bottlenecks. This invention seeks to improve the model's representation ability through other means, by adding adaptive spatiotemporal coding to both the upstream mask autoencoder and the downstream spatiotemporal predictor, in order to enhance the model's ability to learn spatiotemporal data representation and further improve the prediction accuracy of traffic flow. Attached Figure Description

[0020] Figure 1 This is a flowchart of the traffic flow prediction method enhanced by adaptive spatiotemporal coding and mask pre-training in an embodiment of the present invention; Figure 2 This is a schematic diagram of the traffic flow prediction method enhanced by adaptive spatiotemporal coding and mask pre-training in an embodiment of the present invention. Figure 3 This is a comparison of the MAE, RMSE, and MAPE metrics of the present invention (MPTSAformer) with existing embodiments of the invention on the PEMS03, PEMS04, and PEMS08 datasets. Figures 4-12This diagram illustrates the changes in MAE, RMSE, and MAPE of the present invention (MPTSAformer) and existing embodiments (STAEformer, PDG2Seg, PDFormer) on the PEMS03, PEMS04, and PEMS08 datasets. The horizontal axis of the diagram represents the predicted time step, and the vertical axis represents the MAE, RMSE, and MAPE values. Figure 4 A comparative illustration of MAE in the prediction scenarios of the PEMS03 dataset; Figure 5 A comparative diagram illustrating the RMSE in the prediction scenarios of the PEMS03 dataset; Figure 6 A comparative illustration of MAPE prediction scenarios for the PEMS03 dataset; Figure 7 A comparative diagram showing the MAE in the prediction scenarios of the PEMS04 dataset; Figure 8 A comparative diagram illustrating the RMSE in the prediction scenarios of the PEMS04 dataset; Figure 9 A comparative illustration of MAPE prediction scenarios for the PEMS04 dataset; Figure 10 A comparative diagram showing the MAE in the prediction scenarios of the PEMS08 dataset; Figure 11 A comparative diagram illustrating RMSE in prediction scenarios for the PEMS08 dataset; Figure 12 A comparative illustration of MAPE prediction scenarios in the PEMS08 dataset; Figure 13 The MAE, RMSE, and MAPE metrics of the mask autoencoder module of this invention on the PEMS03, PEMS04, and PEMS08 datasets under different input lengths (288, 864, and 1440).

[0021] Figure 14 The results show the MAE, RMSE, and MAPE metrics of the mask autoencoder module of this invention on the PEMS03, PEMS04, and PEMS08 datasets at different mask rates (5%, 25%, 50%, and 75%).

[0022] Figure 15The Masked Pre-training Transformer Autoencoder (MPformer) of this invention enhances the MAE, RMSE, and MAPE metrics of existing embodiments (AGCRN, STDN, and STAEformer) on the PeMSD4 and PeMSD8 datasets. Here, w / o is an abbreviation for without, indicating the original prediction performance of these three methods without MPformer; w / is an abbreviation for with, indicating the prediction performance of these three methods with MPformer.

[0023] Figure 16 This is the result of ablation experiments conducted on the PEMS03, PEMS04, and PEMS08 datasets. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0025] The technical terminology of this invention will be explained and described below: Transportation network: The transportation network is defined as an undirected graph. ,in Yes A set of vertices for a node (e.g., a flow sensor). It is an edge set. It is an adjacency matrix.

[0026] Traffic flow sequence: using express Time-based transportation network Traffic flow in, of which It's a dimension of traffic flow. Continuous. Traffic flow at each moment constitutes a traffic flow sequence. .according to Based on the length of the traffic flow sequence, traffic flow sequences are broadly divided into short-term traffic flow sequences. (e.g., one hour) and long-term traffic flow sequences (e.g., several days).

[0027] Transformer: The Transformer is a deep learning model that employs an attention mechanism. This mechanism assigns different weights based on the importance of different parts of the input data, allowing the model to extract state information from any previous point in the sequence. The attention layer can access all previous states and weight them according to a learned relevance metric, thus providing relevant information about labels that are far apart, providing context for any position in the input sequence, enabling the Transformer model to process all input data at once.

[0028] Mask pre-training: Mask pre-training is widely used in various fields such as Natural Language Processing (NLP) and Computer Vision (CV). This technique randomly masks parts of the input and predicts the masked parts based on the unmasked parts, training the model in a self-supervised manner. The pre-trained model can greatly help downstream tasks.

[0029] Mean Absolute Error (MAE): A metric for evaluating the accuracy of a forecasting model. It measures the average absolute deviation between the model's predicted values ​​and the actual values.

[0030] Root Mean Square Error (RMSE): A metric for evaluating the accuracy of a predictive model. It is used to measure and assess the quality of reconstruction and estimation of spatial environmental variable values ​​that were not used; that is, it is a measure of the error between the estimated values ​​and the reference point values.

[0031] Mean Absolute Percentage Error (MAPE): A metric used to measure the accuracy of a forecasting model. It measures the accuracy of the model's predictions by calculating the average of the relative errors between the predicted and actual values.

[0032] The solutions to the difficulties existing in the current technology are as follows: Regarding the first issue, most existing models are trained end-to-end, making it difficult to fully model long-term traffic flow. The mask pre-trained autoencoder module of this invention effectively learns the temporal dependencies of long-term historical traffic flow through operations such as block embedding, adding masks, and reconstructing the masked parts, helping downstream spatiotemporal predictors avoid spatiotemporal illusions. Regarding the second issue, this invention does not improve model prediction accuracy by improving model architecture or stacking model size. Instead, it adds spatiotemporal adaptive encoding to the hidden representation, adaptively learning the spatiotemporal features of traffic flow, enhancing the model's representational ability, and improving prediction accuracy.

[0033] like Figure 1 As shown, the traffic flow prediction method enhanced by adaptive spatiotemporal coding and mask pre-training of the present invention includes the following steps: (1) Design a block and masking mechanism to perform block and masking operations on long-term traffic flow data in the pre-training stage in order to reduce the computation and memory overhead of the pre-training process. (1.1) During the pre-training phase, the input long-term traffic flow data spans several days. Directly using such long-term data for pre-training would incur unacceptable computational and memory overhead. To address this issue, a block-based operation is employed. Specifically, given long-term traffic flow data... ,in This is the time step of long-term traffic flow. The length used is... One-dimensional convolution kernel ,by The step size is moved along the time dimension, thus representing long-term traffic flow. Divide into non-overlapping blocks This process can be represented as:

[0034] in, In this embodiment, the value is 12. It is the length of the data block. It is the number of output channels of the convolution. This indicates a convolution operation.

[0035] (1.2) with Use a random mask for the mask rate. Covering up the time dimension The step size data, the masked block is denoted as The uncovered block is recorded as Temporal masking forces autoencoders to reconstruct masked data from visible data alone, thereby learning long-term temporal dependencies.

[0036] (2) Design a time coding mechanism to add time coding, date coding and adaptive spatiotemporal coding to the data after it is divided into blocks to enhance its representation ability; (2.1) Take each unmasked block The start time is used as an index to embed the dictionary from the time. Obtaining time code .in This indicates that a day is divided into 288 moments at 5-minute intervals, representing the dimensionality of the encoding. It is 128; (2.2) Take each unmasked block The start time date is used as an index from the date embedding dictionary. Get date code .in Representing the seven days of the week, the dimension of the encoding. It is 128; (2.3) Take the shape as The empty tensor is initialized by Xavier to obtain the adaptive spatiotemporal coding. Dimensions of encoding It is 128. This adaptive spatiotemporal coding will participate in gradient updates during pre-training, adaptively learning the temporal and spatial characteristics of traffic flow; (2.4) Uncovered blocks With time encoding Date code and adaptive spatiotemporal coding Add them together to get the input embedding. The process is represented as:

[0037] (3) Design an autoencoder module to capture the long-term time dependence of long-term traffic flow data; (3.1) Embedding the input The data is fed into an encoder composed of multiple stacked Transformers to obtain a long-term historical data representation. ; (3.2) Using learnable Filling long-term historical data representation To mark the masked block, the filled data is denoted as... ; (3.3) is the data after filling. Add time encoding Date code and adaptive spatiotemporal coding The method for obtaining the above encoding is the same as steps (2.1) to (2.3). Since the data has been padded, the length of the encoding in the time dimension is... ; (3.4) Fill in the data The data is fed into a single-layer Transformer decoder to obtain the reconstructed data. ; (3.5) Obtain the masked block and reconstructing data Then, the autoencoder is trained in a self-supervised manner using the Huber loss function as follows.

[0038]

[0039] in, is a hyperparameter of the Huber loss function, which is set to 1 in this embodiment.

[0040] (4) Design a spatiotemporal coding mechanism to add time coding, date coding, spatial coding and adaptive spatiotemporal coding to short-term traffic flow data to enhance spatiotemporal representation capabilities; (4.1) Use a fully connected layer to improve short-term traffic flow The dimension is obtained to obtain a high-dimensional representation. The process is represented as:

[0041] Among them, dimensions It is 32, and FC indicates a fully connected layer.

[0042] (4.2) Take the time of each step of short-term traffic flow as the index, and embed the dictionary from the time. Obtaining time code Dimensions of encoding It is 32; (4.3) Take the date of each step of short-term traffic flow as the index and embed the data from the date dictionary. Get date code Dimensions of encoding It is 32; (4.4) Statistics on the transportation network The degree of each node in the matrix is ​​obtained. Degree matrix As an index, a dictionary is embedded through nodes. Obtaining spatial encoding Dimensions of encoding It is 32; (4.5) Take the shape as The empty tensor is initialized by Xavier to obtain the adaptive spatiotemporal coding. Dimensions of encoding It is 128. This adaptive spatiotemporal coding will participate in gradient updates during training, adaptively learning the temporal and spatial characteristics of traffic flow; (4.6) High-dimensional representation With time encoding Date code Spatial coding and adaptive spatiotemporal coding By piecing them together, a spatiotemporal representation is obtained. The process is represented as:

[0043] in This represents the tensor concatenation operation, where the concatenated dimension is... It is 256.

[0044] (5) Design a multi-scale trend-aware time attention module to capture the time characteristics and trend changes of short-term traffic flow data; (5.1) Given a spatiotemporal representation The query is obtained through convolution. s and keys The value is obtained through linear projection. The specific process is as follows:

[0045] in This represents the convolution operation. and These are the parameters of the convolution kernel. These are learnable parameters. Performing convolution operations in the time dimension can aggregate traffic flow from adjacent time steps within a certain range, thereby perceiving traffic flow trend information. To perceive trend information at multiple scales, this invention designs a multi-scale trend perception time attention module. This module consists of three Transformer layers, each with a different convolution kernel size. From the first to the third layer, the convolution kernel... and The sizes are 1, 3, and 5, respectively. The multi-scale convolutional kernels move along the time dimension, capturing local trends at different scales in the data. To ensure that the tensor output by each convolutional operation has the same shape, the inputs of the second and third layers are padded with sizes of 1 and 2, respectively.

[0046] (5.2) Query transpose of the key The attention score is obtained by multiplication, scaling, and the softmax function. The specific process is as follows:

[0047] (5.3) Score attention AND value Multiplication yields the output of the multi-scale trend-aware time attention module. The specific process is as follows:

[0048] (6) Design a structural perception spatial attention module to capture the road network spatial characteristics of short-term traffic flow data; (6.1) Given the output of the multi-scale trend-aware time attention module The query is obtained through linear projection. ,key Sum The specific process is as follows:

[0049] in These are learnable parameters.

[0050] (6.2) Using the Floyd-Warshall algorithm based on traffic network The shortest path matrix is ​​obtained by directly calculating the shortest path between all pairs of nodes. .

[0051] (6.3) Use a learnable embedding dictionary Shortest path matrix Mapped to spatial bias matrix .

[0052] (6.4) Query transpose of the key Multiplication, after scaling, and with the spatial bias matrix Attention scores are obtained by summing and using the softmax function. The specific process is as follows:

[0053] (6.5) Attention score AND value Multiplication yields the output of the structure-aware spatial attention module. The specific process is as follows:

[0054] (7) Design a long-term time dependency fusion mechanism to integrate the long-term time dependency of long-term traffic flow data into short-term traffic flow data; (7.1) For short-term traffic flow ,Pick Time before Long-term traffic flow of step length After the block operation described in step (1.2), the data is fed into the pre-trained encoder. At this point, no mask is added, and the long-term time representation of the encoder output is obtained. During the prediction phase, the pre-trained encoder does not participate in gradient updates.

[0055] (7.2) Extracting long-term time representation The last block The last block is connected through a fully connected layer. Project to The dimension will soon be from Projection as .

[0056] (7.3) The output of the structure-aware spatial attention module Transformed into ,Will and The addition yields the enhanced characterization after fusion. The specific process is as follows:

[0057] The above operations can integrate the long-term time dependencies learned in the pre-training phase into short-term traffic flow, thereby improving the predictive performance of traffic flow.

[0058] (8) Predict future short-term traffic flow.

[0059] (8.1) Enhanced characterization Projecting the fully connected layer into the prediction data The specific process is as follows:

[0060] (8.2) Given real data and forecast data The spatiotemporal predictor is trained in a supervised manner using the Huber loss function as follows.

[0061]

[0062] in, is a hyperparameter of the Huber loss function, which is set to 1 in this embodiment.

[0063] Figure 3This paper compares the traffic flow prediction method based on masked pre-training Transformer Autoencoder & Trend-Structure-Aware Transformer Predictor (MPTSAformer) of this invention with existing embodiments on the PEMS03, PEMS04, and PEMS08 datasets in terms of MAE, RMSE, and MAPE metrics. Compared with the baseline, MPTSAformer achieves the best performance on all metrics across all datasets. Taking MAE as an example, MPTSAformer improves upon the second-best method by 3.1%, 1.8%, and 3.5% on PeMSD3, PeMSD4, and PeMSD8, respectively. From these experimental results, we can see that: 1) Except for ST-LLM, time series large language models and time series analysis models perform poorly. Traffic flow exhibits spatiotemporal coupling. Time series large language models and time series analysis models typically focus on pattern mining in the time dimension while ignoring spatial features, thus performing poorly in traffic flow prediction tasks. ST-LLM and traffic flow prediction models learn not only temporal patterns but also spatial locations, thus performing well. This demonstrates the importance of spatial features in traffic flow prediction. 2) In traffic flow prediction models, Transformer-based models outperform those based on spatiotemporal graph neural networks. This is because Transformers have an advantage in time series modeling. Their self-attention mechanism can capture long-range spatiotemporal dependencies. 3) MPTSAformer achieves state-of-the-art prediction performance. Compared to other Transformer-based models, MPTSAformer has three advantages. First, MPTSAformer effectively utilizes the temporal dependencies in long-term historical data. Second, MPTSAformer perceives trend changes at different scales in traffic flow. Third, MPTSAformer captures the structural features of the road network.

[0064] Figure 4-12This section describes the changes in MAE, RMSE, and MAPE of the present invention (MPTSAformer) and existing embodiments (STAEformer, PDG2Seg, PDFormer) on the PEMS03, PEMS04, and PEMS08 datasets. MPTSAformer exhibits the smallest error at almost every step across all metrics. Due to the increased prediction difficulty and range, the MAE, RMSE, and MAPE of all models continuously increase. However, MPTSAformer shows the smallest increase, and its improvement over the baseline method increases with step size. Taking the PeMSD8 dataset as an example, at steps 3, 6, 9, and 12, MPTSAformer is 0.33, 0.44, 0.52, and 0.75 lower than the second-place method in MAE, and 0.47, 0.66, 0.89, and 1.20 lower in RMSE, respectively. Thanks to its long-term time dependence and time trend awareness, MPTSAformer can overcome, to some extent, the difficulty of prediction error increasing with step size. This established MPTSAformer's leading position across various metrics.

[0065] To investigate the impact of different lengths of long-term historical data on the mask pre-training module of this invention, we tested lengths of 1 day (288), 3 days (864), and 7 days (1440) on the PEMS03, PEMS04, and PEMS08 datasets. The experimental results are as follows: Figure 13 As shown, the optimal length for the PeMSD3 dataset is 1 day, while the optimal length for the PeMSD4 and PeMSD8 datasets is 3 days. This result indicates that longer historical data does not necessarily lead to better performance. We hypothesize that excessively short historical data does not contain sufficient long-term time dependencies, while excessively long historical data increases the difficulty of pre-training and training, both leading to a decline in prediction performance. Therefore, it is necessary to adjust the length of the long-term historical data to be more suitable for different datasets.

[0066] To investigate the impact of mask ratio on the mask pre-training module of this invention, experiments were conducted on the PEMS03, PEMS04, and PEMS08 datasets with mask ratios varying from 0.05, 0.25, 0.5, and 0.75. The experimental results are as follows: Figure 14As shown, a mask ratio of 0.25 achieved the best performance across all datasets; further increasing the mask ratio actually led to a decrease in prediction performance. While masked autoencoders in computer vision often employ high mask ratios, we found that traffic flow modeling is better suited to relatively low mask ratios. This is because traffic flow has a long cycle, and an excessively high mask ratio can cause the decoder to lack sufficient temporal context to reconstruct the masked data, making it difficult to capture long-term dependencies. Since the pre-training phase only calculates the loss of the masked portion, an excessively low mask ratio will lead to insufficient training, thereby reducing model performance.

[0067] To verify the effectiveness of the mask pre-training module of this invention, it was combined with existing embodiments of the invention (DCRNN, MTGNN, STDN, and STAEformer), which have different backbone networks. Experiments were conducted on the PeMSD4 and PeMSD8 datasets, and the results are as follows. Figure 15 As shown, after enhancement by the mask pre-training module, all three predictors achieved some degree of performance improvement. This confirms the effectiveness and versatility of the mask pre-training module in enhancing downstream predictors. Through linear projection, the long-term temporal representation learned by the mask pre-training module can be easily incorporated into any downstream predictor.

[0068] To verify the effectiveness of each module of this invention, such as Figure 16 Ablation experiments were designed on the PEMS03, PEMS04, and PEMS08 datasets, including five variants of the present invention: (1) a variant that removes the mask pre-training (w / o P), which performs worse than MPTSAformer, demonstrating the importance of long-term time dependence; (2) a variant that replaces the multi-scale trend-aware module with a linear layer (w / o T), which performs worse than MPTSAformer because this variant follows the point-by-point calculation mode in the original Transformer and cannot perceive changes in time trends; (3) a variant that removes the spatial bias matrix (w / o S), which performs worse than MPTSAformer because this variant only focuses on the global association between nodes and ignores the structural features of the graph; (4) a variant that removes the adaptive spatiotemporal encoding in the pre-training stage (w / o EP) and a variant that removes the adaptive spatiotemporal encoding in the training stage (w / o ET), both of which perform worse than MPTSAformer, demonstrating the importance of adaptive encoding for pre-trained autoencoders and downstream predictors, and its ability to enhance the model's representational capabilities.

[0069] Example 2 Based on the same inventive concept, the present invention also provides an electronic device, including one or more processors; a storage device for storing one or more programs; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in Embodiment 1.

[0070] Since the device described in Embodiment 2 of this invention is the electronic device used to implement the traffic flow prediction method enhanced by adaptive spatiotemporal coding and mask pre-training in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this electronic device based on the method described in Embodiment 1 of this invention, and therefore will not be described again here. All electronic devices used in any method of this invention fall within the scope of protection of this invention.

[0071] Example 3 Based on the same inventive concept, the present invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0072] Since the device described in Embodiment 3 of this invention is a computer-readable medium used to implement the traffic flow prediction method enhanced by adaptive spatiotemporal coding and mask pre-training in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable media used in any method of this invention are within the scope of protection of this invention.

[0073] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A traffic flow prediction method enhanced by adaptive spatiotemporal coding and mask pre-training, characterized in that, Includes the following steps: S1. For historical traffic flow data of at least one day, a block-based processing method is used to randomly mask the data in the time dimension. S2. The masked data is then input into a masked autoencoder for training. Through the block processing and time-dimensional masking, a pre-trained time encoder specifically designed to extract long-term time dependencies is obtained while reducing GPU memory usage. S3. Time, date, road network space, and adaptive spatiotemporal coding are added to short-term traffic flow data to form encoded data. S4. The encoded data is then input into a customized downstream predictor for processing. The multi-scale trend-aware time attention module extracts time trend features from the encoded data using one-dimensional convolutional kernels of different scales. The time trend features are input into the structure-aware spatial attention module. By incorporating the road network topology information into the attention calculation in the form of a learnable bias matrix, the spatial structure features are extracted as the output features of the predictor. S5 uses the pre-trained time encoder to extract the time representation of the long-term historical traffic flow data after block processing, which corresponds to the short-term traffic flow data, and fuses it with the features output in step S4 to generate an enhanced representation. S6 integrates the enhanced representation to predict future traffic flow.

2. The method according to claim 1, characterized in that: The block processing method described in step S1 uses a one-dimensional convolution kernel to perform non-overlapping sliding convolution on the historical traffic flow data in the time dimension, dividing the long sequence into a sequence of data blocks of equal length.

3. The method according to claim 1, characterized in that: In step S3, the adaptive spatiotemporal coding is a learnable tensor initialized by the Xavier method and participating in gradient updates during training, used to adaptively fuse the spatiotemporal context information of traffic flow.

4. The method according to claim 1, characterized in that: The multi-scale trend-aware temporal attention module includes at least two attention layers, wherein the first layer uses a one-dimensional convolutional kernel of size 1 to generate queries and keys, and the second layer uses a one-dimensional convolutional kernel of size greater than 1 to generate queries and keys.

5. The method according to claim 4, characterized in that: The size of the one-dimensional convolution kernel with a value greater than 1 is 3 or 5.

6. The method according to claim 1, characterized in that: In the structure-aware spatial attention module, the road network topology information is the shortest path distance matrix between nodes calculated based on the road network graph structure.

7. The method according to claim 1, characterized in that, The fusion in step S5 is performed as follows: The last block representation output by the pre-trained temporal encoder after encoding long-term historical data is extracted, projected through a linear layer, and then added element-wise with the features output in step S4.

8. The method according to claim 1, characterized in that: In step S2, the pre-training of the mask autoencoder and the overall training in the prediction stage are both optimized using the Huber loss function.

9. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images