Urban traffic flow prediction method and system fusing space-time features and large language model

Abstract

The application discloses a kind of city traffic flow prediction method and system of spatiotemporal feature and large language model fusion, including space-time compiler module and space-time adapter module, the space-time compiler module is extracted by using time step embedding, time embedding day and space embedding Traffic flow characteristics are fused to obtain the token that encapsulates the space-time mode about traffic flow data;The space-time adapter module is frozen by pre-training Bert model Transform block, allow model to adapt to traffic flow dynamic while retaining its language processing strength, the token obtained is fused with pre-trained BERT model Transform block to obtain traffic flow final prediction.The application combines spatiotemporal feature and large language model, models the space and time mode in traffic data using the advanced learning ability of LLM, to realize highly accurate prediction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and large language model technology, specifically to a method and system for predicting urban traffic flow that integrates spatiotemporal features and a large language model. Background Technology

[0002] Urban traffic flow forecasting is a crucial component of Intelligent Transportation Systems (ITS), playing an increasingly important role in optimizing urban traffic management, alleviating congestion, and enhancing public transportation scheduling. The primary objective is to accurately predict future urban traffic conditions, such as vehicle size, congestion levels, and travel times, using historical data. However, the inherently high nonlinearity and dynamic spatiotemporal dependence of traffic flow make accurate and timely forecasting particularly challenging.

[0003] Deep learning models are widely used in traffic flow prediction due to their powerful ability to learn from large-scale spatiotemporal data. Convolutional Neural Networks (CNNs) effectively capture spatial dependencies, while Recurrent Neural Networks (RNNs) model temporal dynamics. However, CNNs struggle with non-Euclidean spatial structures, and RNNs face difficulties with long-term dependencies. To address these challenges, graph neural network (GNN)-based traffic flow prediction models have emerged as a more efficient approach, demonstrating superior performance in modeling complex spatial correlations within road networks. Notable examples include STGCN, STGM, STGAFormer, and RGDAN. Despite these advancements, these models still require carefully designed architectures to handle the complex spatiotemporal properties of traffic flow data and face challenges in generalizing to real-world scenarios with limited labeled data and novel or unseen traffic patterns.

[0004] In recent years, pre-trained large language models (LLMs) have recently made significant breakthroughs in natural language processing and computer vision, providing new approaches for urban traffic flow prediction. Models such as BERT, GPT-4, and LLAMA have demonstrated exceptional abilities in learning from sequential data, effectively capturing complex short-term and long-term dependencies, often outperforming traditional deep learning techniques. These models are well-equipped to handle large amounts of data and exhibit excellent generalization performance, even with limited supervision, making them valuable tools for time series prediction tasks. However, directly applying LLMs to traffic flow prediction remains a challenge. While LLMs demonstrate proficiency in time modeling, they often overlook critical spatial dependencies in traffic data. Traffic flow data not only contains sequential information over time but also involves complex spatial relationships between different locations, such as interactions between nearby intersections or regional traffic patterns, which are naturally ill-suited for text-based architectures. Therefore, it is common to train on LLMs, so the main challenge is to convert traffic flow data into a form that LLMs can handle while preserving spatial and temporal information. SUMMARY

[0005] The present application provides a method for predicting urban traffic flow by fusing spatio-temporal features and large language models.

[0006] Technical solution: To achieve the above purpose, the technical scheme adopted by the present application is:

[0007] A method for predicting urban traffic flow by fusing spatio-temporal features and large language models, comprising the following steps:

[0008] A spatio-temporal compiler module is established according to traffic flow data, which extracts traffic flow features and performs feature fusion to obtain tokens encapsulating spatio-temporal patterns about traffic flow data by using time step embedding, temporal embedding day, and spatial embedding.

[0009] A spatio-temporal adapter module is established according to the tokens, and the obtained tokens are fused with a pre-trained BERT model Transformer block to obtain the final prediction of traffic flow.

[0010] Preferably, the space-time compiler module comprises a time step embedding unit, a time embedding day unit, a space embedding unit, and a feature fusion unit. The time step embedding unit is configured to generate time step embeddings from the traffic flow data. The time embedding day unit is configured to apply absolute position encoding to the day resolution traffic flow data to obtain position encodings, and obtain day embeddings from the position encodings. The space embedding unit is configured to obtain space embeddings from the traffic flow data by employing an adaptive space embedding method. The feature fusion unit integrates the time step embeddings, the day embeddings, and the space embeddings by using convolution operations to obtain tokens that encapsulate space-time patterns of the traffic flow data.

[0011] Preferably, the space-time adapter module comprises an input embedding layer, a frozen pre-trained Bert model transformer unit, and a convolution output layer. The input embedding layer is configured to obtain an input vector from the token embeddings, the position embeddings, and the segment embeddings. The frozen pre-trained Bert model transformer unit comprises six transformer blocks, respectively denoted as a first transformer block, a second transformer block, a third transformer block, a fourth transformer block, a fifth transformer block, and a sixth transformer block. Each transformer block comprises a multi-head attention layer and a feed-forward neural network module. The multi-head attention layer is configured to identify relationships and connections between different positions in the input data, and the feed-forward neural network module is configured to capture temporal patterns within each token. The multi-head attention layer in the first transformer block is frozen, and the learned spatial semantics are preserved from pre-training. The remaining five transformer blocks are fine-tuned for the multi-head attention layer, the feed-forward neural network module, and layer normalization to effectively adapt the model to the prediction task. The convolution output layer is configured to convert the output features of the last transformer block into the final traffic flow prediction.

[0012] Preferably, for the frozen first transformer block, the output of the input embedding H1 is processed by the frozen multi-head attention layer, followed by layer normalization, as follows:

[0013]

[0014] wherein, FrozenMHA(H1) represents applying the frozen multi-head attention component to H1, and LN represents normalizing the output of each layer.

[0015] The calculation of FrozenMHA is kept frozen, and is represented as:

[0016] FrozenMHA(H1) = Concat(FrozenAttention1,...FrozenAttention n )W O ;

[0017] where each FrozenAttention i is computed as follows:

[0018]

[0019] where FrozenAttention i denotes the i-th frozen attention head, i = 1, 2,..., n, n denotes the number of heads, Concat denotes the concatenation operation, E T denotes the temporal embedding, E D denotes the day embedding, E S denotes the spatial embedding, Conv2D(·) denotes the convolution operation applied to the three embeddings, || denotes the concatenation along the channel dimension, θ e denotes the learnable parameters of the Conv2D convolution operation, PE denotes the position embedding, SE denotes the segment embedding, W i Q , W i K and W i V denote trainable weight matrices used to generate the query, key and value representations, d k denotes the dimension of the keys;

[0020] While the remaining 5 Transformer blocks are fine-tuned for multi-head attention layers, feed-forward neural network modules and layer normalization to effectively adapt the model to the prediction task;

[0021] The computation of the l-th layer is given as follows:

[0022]

[0023] where, denotes the output of the previous layer, the final output is given by , l = 2,., 6, LN denotes layer normalization, FFN denotes feed-forward neural network, MHA denotes multi-head attention;

[0024] The computation of MHA is given as follows:

[0025]

[0026] where each attention Attention i is computed using the following formula:

[0027]

[0028] where, is the input of the previous layer, W O represents the projection matrix of the output, W i K and W i V are trainable weight matrices for generating query, key and value representations, d k represents the dimension of the key.

[0029] After MHA, a feed-forward neural network module is introduced to capture the temporal dependencies within the output of the layer;

[0030] FFN(H) = ReLU(HW1 + b1)W2 + b2

[0031] where FFN(H) represents the output after the feed-forward neural network module, H represents the output after multi-head attention, W1, W2, b1 and b2 are adjustable parameters related to the network;

[0032] Finally, by normalizing the output features of each layer, layer normalization is applied to ensure stability during training, and the calculation of layer normalization is:

[0033]

[0034] where LN(H (l-1) ) represents the layer normalization operation on the output of the previous layer, H (l - 1) is the feature representation or output of the previous layer, μ and σ 2 represent the mean and variance of H (l - 1) respectively, parameters γ and β are learnable scale and shift terms, ⊙ represents multi-element multiplication, and ∈ is a small constant added.

[0035] Preferably, the calculation formula of the input vector is as follows:

[0036] H1 = Conv2D(Conv2D(X T , θ T ) || X day · W day || ReLU(W s · X T + b s ), θ e ) + PE + SE

[0037] where H1 represents the input vector, Conv2D(.) refers to point convolution operation using a filter with a kernel size of 1x1, XT denotes traffic flow data, θ T denotes learnable weights associated with the convolutional layer, D denotes the dimension of the hidden representation of the embedding at each time step, X day denotes position encodings, W day denotes a learnable parameter matrix associated with the time embedding day, is a learnable weight matrix, is a bias vector, ReLU denotes the activation function ReLU, PE denotes position embeddings, SE denotes segment embeddings.

[0038] Preferred: The traffic flow final prediction is computed as follows:

[0039]

[0040] where Y S denotes the traffic flow final prediction, is the feature representation obtained by the sixth Transformer block, W conv denotes learnable convolutional weights, b conv denotes learnable convolutional bias terms.

[0041] Preferred: The computation of the time step embedding is as follows:

[0042] E T = Conv2D(X T , θ T )

[0043] where E T denotes the time step embedding, Conv2D(.) denotes a point-wise convolution operation with a filter of kernel size 1x1, X T denotes traffic flow data, θ T denotes learnable weights associated with the convolutional layer, D denotes the dimension of the hidden representation of the embedding at each time step;

[0044] The computation of the day embedding is as follows:

[0045] E D = X day · W day

[0046] where: E D denotes the day embedding, X day denotes position encodings, W day denotes a learnable parameter matrix associated with the time embedding day;

[0047] The computation of the spatial embedding is as follows:

[0048] E S = ReLU(Ws • X T + b s )

[0049] where E S denotes the spatial embedding, is a learnable weight matrix, is a bias vector, and ReLU denotes the activation function ReLU.

[0050] Preferably, the formula of the token encapsulating the spatio-temporal pattern about traffic flow data is as follows:

[0051] E e = Conv2D(Conv2D(X T , θ T ) || X day · W day || ReLU(W s · X T + b s ), θ e )

[0052] where E e denotes the token, E T denotes the time step embedding, E D denotes the day embedding, E S denotes the spatial embedding, Conv2D(·) denotes the convolution operation applied to the three embeddings, || denotes the concatenation along the channel dimension, and θ e denotes the learnable parameters of the Conv2D convolution operation.

[0053] Another object of the present application is to provide a city traffic flow prediction system fusing spatio-temporal features and large language models, which adopts the city traffic flow prediction method fusing spatio-temporal features and large language models. The city traffic flow prediction method fusing spatio-temporal features and large language models comprises a spatio-temporal compiler module and a spatio-temporal adapter module. The spatio-temporal compiler module extracts traffic flow features by using time step embedding, time embedding day and spatial embedding and performs feature fusion to obtain a token encapsulating spatio-temporal patterns about traffic flow data. The spatio-temporal adapter module freezes the pre-trained Bert model Transformer block, allows the model to adapt to traffic flow dynamics while retaining its language processing strength, and fuses the obtained token with the pre-trained BERT model Transformer block to obtain traffic flow final prediction.

[0054] An electronic device comprising a processor and a memory for storing processor-executable instructions, the processor being configured to invoke the instructions stored in the memory to perform the city traffic flow prediction method fusing spatio-temporal features and large language models.

[0055] Compared with the prior art, the present application has the following beneficial effects:

[0056] 1. The present application combines spatio-temporal features with large language models (LLM), utilizing the advanced learning capabilities of LLM to model spatial and temporal patterns in traffic data, enabling highly accurate predictions.

[0057] 2. The present application designs a spatio-temporal compiler module to convert raw traffic data into LLM-compatible tokens, with a parser applying time step, temporal (day), and spatial embeddings to effectively represent spatio-temporal semantics.

[0058] 3. The present application designs a spatio-temporal adapter module to fuse the generated tokens with a pre-trained BERT model. This module freezes the first layer's multi-head attention while fine-tuning the remaining five layers, allowing the model to adapt to traffic flow dynamics while retaining its language processing strength. This provides valuable guidance and decision-making basis for traffic flow prediction, congestion management, and traffic planning. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The overall flowchart of the urban traffic flow prediction method and system provided by the present application, which combines spatio-temporal features with large language models.

[0060] Figure 2 The spatio-temporal compiler flowchart of the urban traffic flow prediction method and system provided by the present application, which combines spatio-temporal features with large language models.

[0061] Figure 3 The spatio-temporal adapter schematic diagram of the urban traffic flow prediction method and system provided by the present application, which combines spatio-temporal features with large language models.

[0062] Figure 4 The MAPE comparison result graph of the prediction performance of the METR-LA dataset in each model.

[0063] Figure 5 The RMSE comparison result graph of the prediction performance of the METR-LA dataset in each model.

[0064] Figure 6 The MAE comparison result graph of the prediction performance of the METR-LA dataset in each model.

[0065] Figure 7 The MAPE comparison result graph of the prediction performance of the PEMS-BAY dataset in each model.

[0066] Figure 8 The RMSE comparison result graph of the prediction performance of the PEMS-BAY dataset in each model.

[0067] Figure 9The MAE of the prediction performance of each model for the PEMS-BAY data set is compared. DETAILED DESCRIPTION

[0068] The application will be further clarified by the following examples, which should not be construed as limiting the scope of the application. Various equivalents for the application, which are intended to be covered by the scope of the claims below, will become apparent to those skilled in the art upon review of the application.

[0069] Example 1

[0070] A city traffic flow prediction method fusing spatio-temporal features and large language models, as shown in Figures 1-3 includes the following steps:

[0071] A spatio-temporal compiler module is established according to traffic flow data, which extracts traffic flow features by using time step embedding, time embedding day and space embedding and performs feature fusion to obtain tokens encapsulating spatio-temporal patterns about traffic flow data.

[0072] The spatio-temporal compiler module includes a time step embedding unit, a time embedding day unit, a space embedding unit and a feature fusion unit, which extracts traffic flow features by using three embedding strategies of time step embedding, time embedding (day) and space embedding, and encapsulates complex spatio-temporal patterns by a feature fusion method.

[0073] It should be noted that the data preprocessing stage includes the following steps:

[0074] A1: Load public traffic data sets such as METR-LA, PEMS-BAY, etc., extract the required fields for research, and delete useless data.

[0075] A2: Use interpolation method to process missing data, and eliminate some abnormal data caused by external factors.

[0076] ① Delete data with unmatched travel time and travel distance.

[0077] ② Delete outlier data by KNN algorithm.

[0078] A3: Normalize the preprocessed data to a limited range.

[0079] A4: Divide the data set into 70% training set, 20% test set and 10% validation set.

[0080] B1: Time step embedding

[0081] The time step embedding unit is used to generate time step embedding according to traffic flow data.

[0082] Time steps represent discrete units in the traffic flow sequence, containing key temporal information that reflects the changing traffic patterns at each spatial location. Embedding these time steps preserves their sequential structure, enabling the parser to capture complex temporal dependencies that are crucial for accurate prediction. We employ point-wise convolutions to extract features from the traffic flow data to generate time step embeddings that effectively capture temporal patterns within each time step. The computation of time step embeddings is given by:

[0083] E T = Conv2D(X T , θ T )

[0084] where E T denotes the time step embedding, Conv2D(.) refers to a point-wise convolution operation with a filter of kernel size 1 x 1, X T denotes the traffic flow data, θ T denotes the learnable weights associated with the convolutional layer, and D denotes the dimension of the hidden representation for each time step embedding.

[0085] B2: Time embedding (day)

[0086] The time embedding day unit is used to apply absolute positional encoding to the traffic flow data at day resolution, resulting in a position encoding from which the day embedding is derived.

[0087] We employ the day embedding when excluding the week embedding. This choice is based on two main reasons. First, in many traffic flow prediction tasks, the periodic feature of every week is stable and can be effectively captured through day-based embedding. Second, especially in the case of short-term prediction or limited data, introducing a week embedding can complicate the model and increase the risk of overfitting. Therefore, focusing only on the day embedding enhances the model's generalization ability, simplifies the computation, and maintains the prediction accuracy. Following the established practice of time data processing, we apply absolute positional encoding to the traffic flow data at "day" resolution, resulting in a position encoding The computation of day embedding is given by:

[0088] E D = X day · W day

[0089] where E D denotes the day embedding, X day denotes the position encoding, and W day denotes the learnable parameter matrix associated with the time embedding day.

[0090] B3: Spatial embedding

[0091] The spatial embedding unit is used to obtain spatial embedding by employing an adaptive spatial embedding method according to the traffic flow data.

[0092] To comprehensively represent the spatial relationship of different locations in the traffic flow network, we employ an adaptive spatial embedding method. The calculation formula of spatial embedding is as follows:

[0093] E S = ReLU(W s · X T + b s )

[0094] where E S represents the spatial embedding, is a learnable weight matrix, is a bias vector, and ReLU represents the activation function ReLU. The ReLU activation function is used to introduce nonlinearity, thereby improving the model expression ability.

[0095] B4: Feature fusion

[0096] The feature fusion unit integrates the time step embedding, day embedding, and spatial embedding using convolution operation to obtain tokens encapsulating spatiotemporal patterns about traffic flow data.

[0097] To obtain the final token representation We use convolution operation instead of direct connection to integrate the time step embedding time embedding and spatial embedding This method more effectively captures time and space interaction by learning feature correlation in embedding, and the calculation formula of tokens encapsulating spatiotemporal patterns about traffic flow data is as follows:

[0098] E e = Conv2D(E T || E D || E S , θ e )

[0099] = Conv2D(Conv2D(X T , θ T )|| X day · W day || ReLU(W s · X T + b s ), θ e )

[0100] where E e represents the token, and E Tdenotes the time step embedding, E D denotes the token embedding, E S denotes the spatial embedding, Conv2D(·) denotes a convolution operation applied to the three embeddings, || denotes concatenation along the channel dimension, and θ e denotes the learnable parameters of the Conv2D convolution operation.

[0101] This convolution-based fusion results in a comprehensive token representation that encapsulates complex spatio-temporal patterns, enhancing the model's ability to more accurately understand and predict traffic flow dynamics.

[0102] S2: Design a spatio-temporal adapter module

[0103] According to the token, the spatio-temporal adapter module is established, and the obtained token is fused with the pre-trained BERT model Transformer block to obtain the final prediction of traffic flow.

[0104] The spatio-temporal adapter module includes an input embedding layer, a frozen pre-trained Bert model Transformer unit, and a convolution output layer.

[0105] C1: Input embedding layer

[0106] The input embedding layer is used to obtain an input vector according to token embedding, position embedding, and segment embedding.

[0107] The calculation formula of the input vector is as follows:

[0108] H1 = E e + PE + SE

[0109] = Conv2D(E T || E D || E S , θ e ) + PE + SE

[0110] = Conv2D(Conv2D(X T , θ T ) || X day · W day || ReLU(W s · X T + b s ), θ e ) + PE + SE

[0111] where H1 denotes the input vector, E e denotes the token, PE denotes the position embedding, and SE denotes the segment embedding. PE and SE are fine-tuned parameters. This formula effectively combines position information and segment distinction, facilitating the model's understanding of the token context related to its position in the sequence.

[0112] C2: Freeze pre-trained Bert model Transformer units

[0113] The freeze pre-trained Bert model Transformer units include 6 Transformer blocks, denoted as the first Transformer block, the second Transformer block, the third Transformer block, the fourth Transformer block, the fifth Transformer block, and the sixth Transformer block, each of which includes a multi-head attention layer and a feed-forward neural network module. The multi-head attention layer is used to identify the relationships and connections between different positions in the input data, and the feed-forward neural network module is used to capture the temporal patterns within each token. The multi-head attention layer in the first Transformer block is frozen to retain the learned spatial semantics from pre-training. The remaining five Transformer blocks are fine-tuned for the multi-head attention layer, the feed-forward neural network module, and the layer normalization to effectively adapt the model to the prediction task.

[0114] In this section, we implement a strategy to fine-tune the BERT Transformer architecture, which consists of six layers. Each Transformer block includes two main components: multi-head attention and feed-forward neural network (FFN) modules. The multi-head attention mechanism helps the model identify the ability to identify spatial dependencies between tokens, enabling it to identify relationships and connections between different positions in the input data. On the other hand, the FFN module captures temporal patterns within each token, extracting sequential trends and improving the model's ability to analyze spatio-temporal dynamics.

[0115] To maintain the integrity of the pre-training knowledge, we freeze the multi-head attention layer in the first Transformer block, retaining the learned spatial semantics from pre-training. This selective freezing ensures that core semantic understanding is preserved while also reducing the number of parameters that need to be adjusted. In the remaining five layers, we fine-tune the multi-head attention, FFN, and layer normalization (LN) components to effectively adapt the model to the prediction task.

[0116] For the first frozen Transformer block, the output of the input embedding H1 is processed through the frozen multi-head attention layer, followed by layer normalization (LN), which is formulated as follows:

[0117]

[0118] where, denotes the output of the first frozen Transformer block, FrozenMHA(H1) denotes applying the frozen multi-head attention component to H1, and LN denotes layer normalization on the output of each layer.

[0119] The computation of FrozenMHA remains frozen and is denoted as:

[0120] FrozenMHA(H1) = Concat(FrozenAttention1,...FrozenAttention n )W O .

[0121] where each FrozenAttention i is computed as:

[0122]

[0123] where FrozenAttention i denotes the i-th frozen attention head, i = 1, 2,..., n, n denotes the number of heads, Concat denotes concatenation, and W i Q , W i K and W i V denote trainable weight matrices used to generate the query, key, and value representations, d k denotes the dimension of the keys.

[0124] The remaining five Transformer blocks, however, are fine-tuned for multi-head attention layers, feed-forward neural network modules, and layer normalization to effectively adapt the model to the prediction task. The model undergoes fine-tuning not only in the multi-head attention layers (MHA) but also in the feed-forward neural network modules (FFN) and layer normalization (LN).

[0125] The computation of the l-th layer proceeds as follows:

[0126]

[0127] where denotes the output of the previous layer, and the final output is given by H1 (6) , l = 2,..., 6, LN denotes layer normalization, FFN denotes a feed-forward neural network, and MHA denotes multi-head attention.

[0128] In this equation, multi-head attention (MHA) plays a central role in capturing spatial dependencies in the data. Specifically, the computation of MHA is given by:

[0129]

[0130] Each attention i Calculate using the following formula:

[0131]

[0132] in, It is the input of the previous layer, W O This represents the output projection matrix. W i K and W i V It is a trainable weight matrix used to generate query, key, and value representations, d k The dimension of the key is represented, while the softmax function is used to normalize the attention score and generate weights for each attention head.

[0133] Following MHA, a feedforward neural network module is introduced to capture the temporal dependencies within the layer's output, defined as:

[0134] FFN(H)=ReLU(HW1+b1)W2+b2

[0135] Where FFN(H) represents the output after passing through the feedforward neural network module, H represents the output after multi-head attention, and W1, W2, b1, and b2 are network-related adjustable parameters.

[0136] This two-layer feedforward network adds nonlinear transformations to improve the model's ability to capture complex temporal patterns that are crucial for accurate predictions.

[0137] Finally, layer normalization (LN) is applied to normalize the output features of each layer to ensure stability during training. The calculation of layer normalization is defined as follows:

[0138]

[0139] Among them, LN(H (l-1) H indicates that layer normalization is performed on the output of the previous layer. (l-1) μ and σ are the feature representations or outputs of the previous layer. 2 H respectively (l - 1) The mean and variance of , parameters γ and β are learnable scaling and shift terms, ⊙ denotes multi-element multiplication, and ∈ is a small constant added to avoid division by zero.

[0140] C3: Convolutional output layer

[0141] The convolutional output layer is used to transform the output features of the last Transformer block into the final traffic flow prediction.

[0142] The layer takes the feature representation obtained from the sixth layer of the pre-trained model and applies a convolution operation to generate the regression output. We compute the output prediction of the traffic flow The final prediction of the traffic flow is computed as follows:

[0143]

[0144] where Y S denotes the final prediction of the traffic flow, is the feature representation obtained from the sixth Transformer block, W conv denotes the learnable convolution weights, and b conv denotes the learnable convolution bias term.

[0145] The convolution operation is used to capture spatial dependencies from within the feature maps of the previous layer, enabling effective prediction of the traffic flow at subsequent time steps. To introduce non-linearity and enhance the capacity of the model, we apply an activation function (e.g., ReLU) to the output prediction Y S .

[0146] Embodiment 2

[0147] The embodiment provides a city traffic flow prediction system fusing spatio-temporal features and large language models, and adopts a city traffic flow prediction method fusing spatio-temporal features and large language models. The city traffic flow prediction method fusing spatio-temporal features and large language models comprises a spatio-temporal compiler module and a spatio-temporal adapter module. The spatio-temporal compiler module extracts traffic flow features by using time step embedding, time embedding day, and space embedding, and performs feature fusion to obtain tokens encapsulating spatio-temporal patterns about traffic flow data. The spatio-temporal adapter module freezes a pre-trained Bert model Transformer block, allows the model to adapt to traffic flow dynamics while retaining its language processing strength, fuses the obtained tokens with the pre-trained BERT model Transformer block to obtain a final prediction of the traffic flow.

[0148] Embodiment 3

[0149] The embodiment provides an electronic device comprising a processor and a memory. The memory is configured to store processor-executable instructions. The processor is configured to invoke the instructions stored in the memory to execute the city traffic flow prediction method fusing spatio-temporal features and large language models.

[0150] The spatio-temporal compiler module of the present application converts raw traffic data into LLM compatible tokens, and the parser applies time step, temporal and spatial embeddings to effectively represent spatio-temporal semantics. The spatio-temporal adapter module fuses the generated tokens with a pre-trained BERT model. This module freezes the multi-head attention of the first layer while fine-tuning the remaining five layers, allowing the model to adapt to traffic flow dynamics while preserving its language processing strength. The present application combines large language models with spatio-temporal features, leveraging the advanced learning capabilities of LLMs to model spatial and temporal patterns in traffic data, resulting in highly accurate predictions. The present application can provide valuable guidance and decision-making basis for traffic flow prediction, congestion management, and transportation planning.

[0151] Embodiment 4

[0152] Reference Figure 1 , 2 , 3 is an example of the present application, which provides a city traffic flow prediction method combining spatio-temporal features and large language models. In order to verify its beneficial effects, the comparative results of two schemes are provided.

[0153] The embodiments of the present application use two public traffic flow datasets METR-LA and PEMS-BAY.

[0154] The METR-LA dataset includes traffic data from 207 sensors on the Los Angeles freeway. It covers a time span from March 1, 2012 to June 30, 2012, with a total of 34,272 time steps, each representing a 5-minute interval. The PEMS-BAY dataset includes data from 325 sensors in the San Francisco Bay Area. It covers a time span from January 1, 2017 to May 31, 2017, with a total of 52,116 time steps, also at 5-minute intervals.

[0155] For these two datasets, we follow the traditional preprocessing method to normalize the 5-minute interval data and divide them into 70% for training, 10% for validation, and 20% for testing.

[0156] We use the present application to predict the traffic flow of the next 12 time steps, using data from the previous 12 time steps as input. The time dimension Td is set to 288, with each time step representing a 5-minute interval. All experiments are performed on an RTX 4090D GPU with 24GB of memory. The model training uses the Ranger21 optimizer with a learning rate of 0.0001. The LLM architecture has a 6-layer BERT model, with the attention mechanism of the first layer being frozen. The training runs for 25 epochs, with a batch size of 32, and the results are averaged in the prediction step.

[0157] To evaluate the effectiveness of our invention, we rely on three main indicators: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Mean Absolute Percentage Error (MAPE).

[0158] In the experiment, we compared our invention with 8 baseline models, which can be categorized into the following groups: statistical methods (HA, ARIMA), graph-based methods (STGCN, DCRNN, GWNet), pre-trained models (STEP), and large language models (GPT2, GATGPT).

[0159] From Figures 4-9 As shown in the comparison chart of the prediction performance of each model, our invention demonstrated outstanding performance on both METR-LA and PEMS-BAY datasets. Our invention outperformed the 8 baseline models in all three prediction ranges (15 minutes, 30 minutes, and 60 minutes). On the METR-LA dataset, our invention showed excellent performance in terms of MAE, RMSE, and MAPE in all three prediction ranges, proving its advantage in traffic flow prediction. Specifically, for 15-minute prediction, our invention achieved an MAE of 2.34, significantly lower than STEP (2.61), and an MAPE of 5.71%. This consistent performance was also verified in 30-minute and 60-minute predictions, especially in 60-minute prediction, where our invention maintained the leading position with an MAE of 3.00 and an MAPE of 8.15%. Similarly, in the PEMS-BAY dataset, our invention similarly demonstrated its superior prediction ability. It achieved an MAE of 1.05, an RMSE of 2.11, and an MAPE of 2.10%, outperforming other models in all indicators. Similar to the previous dataset, our invention showed the best performance in 15-minute, 30-minute, and 60-minute predictions, especially in 15-minute prediction, where its MAE surpassed Gemma2 (1.13). This consistent performance proves that our invention has strong generalization ability and adaptability across different datasets. In summary, our invention significantly improved the prediction accuracy on both METR-LA and PMS-BAY datasets, confirming its effectiveness in the field of traffic flow prediction. This indicates that our invention can effectively capture the complex patterns of traffic flow changes, provide more accurate short-term flow predictions, and be applicable to different flow datasets, showing broad potential for application.

[0160] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered within the scope of protection of the present application.

Claims

1. A method for predicting urban traffic flow that integrates spatiotemporal features and a large language model, characterized in that, Includes the following steps: A spatiotemporal compiler module is established based on traffic flow data. The spatiotemporal compiler module extracts traffic flow features and performs feature fusion by using time step embedding, time embedding day and spatial embedding to obtain a token that encapsulates the spatiotemporal pattern of traffic flow data. The spatiotemporal compiler module includes a time step embedding unit, a time embedding day unit, a spatial embedding unit, and a feature fusion unit. The time step embedding unit is used to generate time step embeddings based on traffic flow data. The temporal embedding unit is used to apply absolute location coding to traffic flow data at a time resolution to obtain location coding, and then to obtain the temporal embedding based on the location coding; the spatial embedding unit is used to obtain the spatial embedding based on the traffic flow data by using an adaptive spatial embedding method. The feature fusion unit uses convolution operations to integrate temporal step embeddings, day embeddings, and spatial embeddings to obtain tokens that encapsulate spatiotemporal patterns in traffic flow data; A spatiotemporal adapter module is established based on the token. The spatiotemporal adapter module freezes the pre-trained BERT model Transformer block, allowing the model to retain its language processing strength while adapting to traffic flow dynamics. The obtained token is then fused with the pre-trained BERT model Transformer block to obtain the final traffic flow prediction. The spatiotemporal adapter module includes an input embedding layer, a frozen pre-trained BERT model Transformer unit, and a convolutional output layer. The input embedding layer is used to obtain the input vector based on token embedding, position embedding, and segment embedding. The frozen pre-trained BERT model Transformer unit includes six Transformer blocks, denoted as the first Transformer block, the second Transformer block, the third Transformer block, the fourth Transformer block, the fifth Transformer block, and the sixth Transformer block. Each Transformer block includes a multi-head attention layer and a feedforward neural network module. The multi-head attention layer is used to identify the relationships and connections at different locations in the input data, and the feedforward neural network module is used to capture the temporal pattern within each token. The multi-head attention layer in the first Transformer block is frozen to retain the learned spatial semantics from pre-training. For the remaining five Transformer blocks, the multi-head attention layer, the feedforward neural network module, and layer normalization are fine-tuned to effectively adapt the model to the prediction task. The convolutional output layer is used to convert the output features of the last Transformer block into the final traffic flow prediction.

2. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 1, characterized in that: For the first frozen Transformer block, the output of the input embedding H1 is processed through a frozen multi-head attention layer, followed by layer normalization, as shown in the following formula: in, This indicates the output of the first frozen Transformer block. FrozenMHA(H1) means that the frozen multi-head attention component is applied to H1, and LN means that the output of each layer is normalized. FrozenMHA keeps the calculations frozen, represented as: FrozenMHA(H1)=Concat(FrozenAttention1,...FrozenAttention n )W O ; Each FrozenAttention i The calculation is as follows: Among them, FrozenAttention i This represents the i-th frozen attention head, i = 1, 2, ..., n, where n represents the number of heads, and Concat represents the concatenation operation. T E represents time step embedding. D Indicates embedding, E S Represents spatial embedding, Conv2D(·) represents the convolution operation applied to the three embeddings, || represents concatenation along the channel dimension, and θ e The parameters represent the learnable parameters of the Conv2D convolution operation, PE represents the position embedding, SE represents the segment embedding, and W represents the segment embedding. i Q W i K and W i V d represents the trainable weight matrix used to generate query, key, and value representations. k The dimension representing the key; The remaining 5 Transformer blocks fine-tune the multi-head attention layer, feedforward neural network module, and layer normalization to effectively adapt the model to the prediction task. The calculation for layer l is performed as follows: in, This indicates the output of the previous layer, and the final output is generated by... Given that l = 2, ..., 6, LN represents layer normalization, FFN represents feedforward neural network, and MHA represents multi-head attention; The formula for calculating MHA is: Each attention i Calculate using the following formula: in, It is the input of the previous layer, W O This represents the output projection matrix. and It is a trainable weight matrix used to generate query, key, and value representations, d k The dimension representing the key; Following MHA, a feedforward neural network module is introduced to capture the temporal dependencies within the layer's output; FFN(H)=ReLU(HW1+b1)W2+b2 Where FFN(H) represents the output after passing through the feedforward neural network module, H represents the output after multi-head attention, and W1, W2, b1 and b2 are network-related adjustable parameters; Finally, layer normalization is applied to normalize the output features of each layer to ensure stability during training. The calculation of layer normalization is as follows: Among them, LN(H (l-1) H indicates that layer normalization is performed on the output of the previous layer. (l-1) μ and σ are the feature representations or outputs of the previous layer. 2 H respectively (l-1) The mean and variance of , parameters γ and β are learnable scaling and shift terms, ⊙ denotes multi-element multiplication, and ∈ is an added small constant.

3. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 2, characterized in that: The formula for calculating the input vector is as follows: H1=Conv2D(Conv2D(X T ,θ T )||X day ·W day ||ReLU(W s ·X T +b s ),θ e )+OR+IF Where H1 represents the input vector, Conv2D(.) refers to the point convolution operation using a filter with a kernel size of 1×1, and X T Represents traffic flow data, θ T X represents the learnable weights associated with the convolutional layer, D represents the dimension of the hidden representation embedded at each time step, and X represents the number of learningable weights associated with the convolutional layer. day W represents the position code. day This represents the learnable parameter matrix associated with the time embedding day. It is a learnable weight matrix. It is the bias vector, ReLU represents the activation function ReLU, PE represents the position embedding, and SE represents the segment embedding.

4. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 3, characterized in that: The formula for calculating the final traffic flow forecast is as follows: Among them, Y S This indicates the final traffic flow forecast. For the feature representation obtained from the sixth Transformer block, W conv b represents the learnable convolutional weights. conv This represents the learnable convolutional bias term.

5. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 4, characterized in that: The formula for calculating the time step embedding is as follows: E T =Conv2D(X T ,i T ) Among them, E T Representing time-step embedding, Conv2D(.) refers to point convolution operation using a filter with a kernel size of 1×1, where X... T Represents traffic flow data, θ T The learnable weights associated with the convolutional layer are represented by , and D represents the dimension of the hidden representation embedded at each time step. The calculation formula for embedded systems is as follows: HAVE BEEN D =X day ·W day Among them: E D Indicates the embedding of X day W represents the position code. day This represents a learnable parameter matrix related to the time embedding day; The formula for calculating spatial embedding is as follows: HAVE BEEN S =ReLU(W s X T +b s ) Among them, E S Indicates spatial embedding, It is a learnable weight matrix. It is the bias vector, and ReLU represents the activation function ReLU.

6. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 5, characterized in that: The calculation formula for tokens encapsulating spatiotemporal patterns in traffic flow data is as follows: E e =Conv2D(Conv2D(X T ,θ T )||X day ·IN day ||ReLU(W s ·X T +b s ),θ e ) Among them, E e E represents a token. T E represents time step embedding. D Indicates embedding, E S Represents spatial embedding, Conv2D(·) represents the convolution operation applied to the three embeddings, || represents concatenation along the channel dimension, and θ e This represents the learnable parameters of the Conv2D convolution operation.

7. A city traffic flow prediction system integrating spatiotemporal features and a large language model, characterized in that, The urban traffic flow prediction method using any one of claims 1-6, which integrates spatiotemporal features and a large language model, includes a spatiotemporal compiler module and a spatiotemporal adapter module. The spatiotemporal compiler module extracts traffic flow features by using time step embedding, time embedding day, and spatial embedding, and performs feature fusion to obtain a token that encapsulates the spatiotemporal pattern of the traffic flow data. The spatiotemporal adapter module freezes the pre-trained BERT model Transformer block, allowing the model to retain its language processing strength while adapting to traffic flow dynamics, and fuses the obtained token with the pre-trained BERT model Transformer block to obtain the final traffic flow prediction.

8. An electronic device, characterized in that: The method includes a processor and a memory, the memory being used to store processor-executable instructions, the processor being configured to invoke the instructions stored in the memory to perform the method of any one of claims 1-6.

Images