Urban traffic flow prediction method and system fusing spatial-temporal characteristics and large language model

Abstract

The invention discloses an urban traffic flow prediction method and system fusing spatio-temporal characteristics and a large language model, and the system comprises a spatio-temporal compiler module and a spatio-temporal adapter module. The space-time compiler module extracts traffic flow features by using time step embedding, time embedding day and space embedding and performs feature fusion to obtain a token in which a space-time mode is packaged about traffic flow data; and the space-time adapter module allows the model to keep the language processing strength while adapting to the dynamic state of the traffic flow by freezing a pre-trained Bert model Transform block, and fuses the obtained token with the pre-trained BERT model Transform block to obtain the final prediction of the traffic flow. According to the method, spatial and temporal features are combined with a large language model, and spatial and temporal modes in traffic data are modeled by using the advanced learning ability of the LLM, so that highly accurate prediction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and large language models, and specifically to a method and system for predicting urban traffic flow by integrating spatiotemporal features with a large language model. Background Art

[0002] Urban traffic flow forecasting is an important component of intelligent transportation systems (ITS) and plays an increasingly important role in optimizing urban traffic management, alleviating congestion, and enhancing public transportation scheduling. The main goal is to accurately predict future urban traffic conditions such as vehicle volume, congestion level, and travel time using historical data. However, the inherent high nonlinearity and dynamic spatiotemporal dependencies of traffic flow make accurate and timely predictions particularly challenging.

[0003] Deep learning models have been widely used for traffic flow prediction due to their powerful ability to learn from large-scale spatiotemporal data. Convolutional neural networks (CNNs) can effectively capture spatial dependencies, while recurrent neural networks (RNNs) model temporal dynamics. However, CNNs struggle with non-Euclidean spatial structures and RNNs face difficulties with long-term dependencies. To address these challenges, graph neural network (GNN)-based traffic flow prediction models have emerged as a more effective approach, demonstrating superior performance in modeling complex spatial correlations within road networks. Notable examples include STGCN, STGM, STGAFormer, and RGDAN. Despite these advances, these models still require carefully designed architectures to handle the complex spatiotemporal nature of traffic flow data and face challenges in generalizing to real-world scenarios with limited labeled data and new or unseen traffic patterns.

[0004] In recent years, pre-trained large language models (LLMs) have recently achieved remarkable breakthroughs in natural language processing and computer vision, providing new approaches for urban traffic flow prediction. Models such as BERT, GPT-4, and LLAMA have shown extraordinary ability in learning from sequence data, effectively capturing complex short-term and long-term dependencies, often surpassing traditional deep learning techniques. These models can handle large amounts of data well and show excellent generalization performance even with limited supervision, which makes them a valuable tool for time series prediction tasks. However, it remains challenging to directly apply LLMs to traffic flow prediction. Although LLMs demonstrate proficiency in temporal modeling, they usually ignore the critical spatial dependencies in traffic data. Traffic flow data not only contains sequential information over time, but also involves complex spatial relationships at different locations, such as interactions between nearby intersections or regional traffic patterns, which are naturally not suitable for text-based architectures and are usually trained on LLMs, so the main challenge is to convert traffic flow data into a form that LLMs can handle while preserving spatial and temporal information. Summary of the invention

[0005] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides an urban traffic flow prediction method that integrates spatiotemporal features with a large language model.

[0006] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] A method for predicting urban traffic flow by integrating spatiotemporal features with a large language model comprises the following steps:

[0008] A spatiotemporal compiler module is established according to the traffic flow data. The spatiotemporal compiler module extracts traffic flow features by using time step embedding, time embedding day and space embedding, and performs feature fusion to obtain tokens encapsulating spatiotemporal patterns of the traffic flow data.

[0009] A spatiotemporal adapter module is established based on the tokens, and the obtained tokens are fused with the pre-trained BERT model Transformer block to obtain the final traffic flow prediction.

[0010] Preferably: the spatiotemporal compiler module includes a time step embedding unit, a time embedding day unit, a space embedding unit and a feature fusion unit, wherein the time step embedding unit is used to generate a time step embedding according to the traffic flow data. The time embedding day unit is used to apply the absolute position coding to the traffic flow data with a day resolution to obtain the position coding, and obtain the day embedding according to the position coding. The space embedding unit is used to obtain the space embedding according to the traffic flow data by adopting an adaptive space embedding method. The feature fusion unit integrates the time step embedding, the day embedding and the space embedding by using a convolution operation to obtain a token that encapsulates the spatiotemporal pattern of the traffic flow data.

[0011] Preferably: the spatiotemporal adapter module includes an input embedding layer, a frozen pre-trained Bert model Transformer unit, and a convolutional output layer, wherein the input embedding layer is used to obtain an input vector based on token embedding, position embedding, and segment embedding. The frozen pre-trained Bert model Transformer unit includes 6 Transformer blocks, which are respectively recorded as the first Transformer block, the second Transformer block, the third Transformer block, the fourth Transformer block, the fifth Transformer block, and the sixth Transformer block. Each Transformer block includes a multi-head attention layer and a feedforward neural network module, wherein the multi-head attention layer is used to identify the relationship and connection between different positions in the input data, and the feedforward neural network module is used to capture the temporal pattern within each token. The multi-head attention layer in the first Transformer block is frozen to retain the learned spatial semantics from pre-training. The remaining 5 Transformer blocks fine-tune the multi-head attention layer, the feedforward neural network module, and the layer normalization to effectively adapt the model to the prediction task. The convolutional output layer is used to convert the output features of the last Transformer block into the final prediction of traffic flow.

[0012] Preferred: For the frozen first Transformer block, the output of the input embedding H1 is processed through a frozen multi-head attention layer, followed by layer normalization, as follows:

[0013]

[0014] in, represents the frozen output of the first Transformer block, FrozenMHA(H1) means applying the frozen multi-head attention component to H1, and LN means normalizing the output of each layer;

[0015] The calculation of FrozenMHA remains frozen, expressed as:

[0016] FrozenMHA(H1)=Concat(FrozenAttention1,...FrozenAttention n )W O ;

[0017] Each FrozenAttention i The calculation is as follows:

[0018]

[0019] Among them, FrozenAttention i represents the i-th frozen attention head, i = 1, 2, ..., n, n represents the number, Concat represents the concatenation operation, E T represents the time step embedding, E D represents day embedding, E S represents the spatial embedding, Conv2D(·) represents the convolution operation applied to the three embeddings, || represents the concatenation along the channel dimension, and θ e represents the learnable parameters of the Conv2D convolution operation, PE represents position embedding, SE represents segment embedding, and represents the trainable weight matrix used to generate query, key and value representations, d k Represents the dimension of the key;

[0020] The remaining five Transformer blocks fine-tune the multi-head attention layer, feed-forward neural network module, and layer normalization to effectively adapt the model to the prediction task;

[0021] The calculation of the lth layer is performed as follows:

[0022]

[0023] in, Represents the output of the previous layer, and the final output is Given,l,=2,.,6,,LN represents layer normalization, FFN represents feed-forward neural network, and MHA represents multi-head attention;

[0024] The calculation formula of MHA is:

[0025]

[0026] Each attention i Use the following formula to calculate:

[0027]

[0028] in, is the input of the previous layer, W O represents the output projection matrix, and is a trainable weight matrix used to generate query, key, and value representations, d k Indicates the dimension of the key.

[0029] After MHA, a feed-forward neural network module is introduced to capture the temporal dependencies within the output of the layer;

[0030] FFN(H)=ReLU(HW1+b1)W2+b2

[0031] Among them, FFN(H) represents the output after the feedforward neural network module, H represents the output after multi-head attention, and W1, W2, b1 and b2 are adjustable parameters related to the network;

[0032] Finally, layer normalization is applied to ensure stability during training by normalizing the output features of each layer. The calculation of layer normalization is:

[0033]

[0034] Among them, LN(H (l-1) ) represents the layer normalization operation on the output of the previous layer, H (l-1) is the feature representation or output of the previous layer, μ and σ 2 Respectively represent H (l-1) The mean and variance of , the parameters γ and β are learnable scale and shift terms, ⊙ represents multi-element multiplication, and ∈ is a small constant added.

[0035] Preferably: the calculation formula of the input vector is as follows:

[0036] H1=Conv2D(Conv2D(X T ,θ T )||X day ·W day ||ReLU(W s ·X T +b s ),θ e )+PE+SE

[0037] Where H1 represents the input vector, Conv2D(.) refers to the point convolution operation using a filter with a kernel size of 1×1, and X T represents traffic flow data, θ T represents the learnable weights associated with the convolutional layer, D represents the dimension of the hidden representation embedded at each time step, and X day represents the position code, W day represents the learnable parameter matrix associated with the temporal embedding day, is a learnable weight matrix, is the bias vector, ReLU represents the activation function ReLU, PE represents position embedding, and SE represents segment embedding.

[0038] Preferred: The final traffic flow prediction calculation formula is as follows:

[0039]

[0040] Among them, Y S represents the final prediction of traffic flow, is the feature representation obtained by the sixth Transformer block, W conv represents the learnable convolution weights, b conv Represents a learnable convolution bias term.

[0041] Preferred: The calculation formula for time step embedding is as follows:

[0042] E T =Conv2D(X T ,θ T )

[0043] Among them, E T represents the time step embedding, Conv2D(.) refers to the point convolution operation using a filter with a kernel size of 1×1, X T represents traffic flow data, θ T represents the learnable weights associated with the convolutional layer, and D represents the dimension of the hidden representation embedded at each time step;

[0044] The calculation formula for day embedding is as follows:

[0045] E D =X day ·W day

[0046] Where: E D represents the day embedding, X day represents the position code, W day represents the learnable parameter matrix associated with the temporal embedding day;

[0047] The calculation formula of spatial embedding is as follows:

[0048] E S =ReLU(W s ·X T +b s )

[0049] Among them, E S represents spatial embedding, is a learnable weight matrix, is the bias vector, and ReLU represents the activation function ReLU.

[0050] Preferably: the calculation formula for the token encapsulating the spatiotemporal pattern of traffic flow data is as follows:

[0051] E e =Conv2D(Conv2D(X T ,θ T )||X day ·W day ||ReLU(W s ·X T +b s ),θ e )

[0052] Among them, E e Represents a token, E T represents the time step embedding, E D represents day embedding, E S represents the spatial embedding, Conv2D(·) represents the convolution operation applied to the three embeddings, || represents the concatenation along the channel dimension, and θ e Represents the learnable parameters of the Conv2D convolution operation.

[0053] Another object of the present invention is to provide an urban traffic flow prediction system that integrates spatiotemporal features and a large language model, and the urban traffic flow prediction method that integrates spatiotemporal features and a large language model includes a spatiotemporal compiler module and a spatiotemporal adapter module. The spatiotemporal compiler module extracts traffic flow features by using time step embedding, time embedding day and space embedding, and performs feature fusion to obtain tokens that encapsulate spatiotemporal patterns about traffic flow data. The spatiotemporal adapter module freezes the pre-trained Bert model Transformer block, allowing the model to retain its language processing strength while adapting to traffic flow dynamics, and fuses the obtained tokens with the pre-trained BERT model Transformer block to obtain the final prediction of traffic flow.

[0054] An electronic device includes a processor and a memory, wherein the memory is used to store processor executable instructions, and the processor is configured to call the instructions stored in the memory to execute the urban traffic flow prediction method that integrates spatiotemporal features with a large language model.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. The present invention combines spatiotemporal features with a large language model (LLM) and uses the advanced learning capabilities of LLM to model spatial and temporal patterns in traffic data, thereby achieving highly accurate predictions.

[0057] 2. The present invention designs a spatiotemporal compiler module to convert raw traffic data into LLM-compatible tokens, and the parser applies time step, time (day) and space embedding to effectively represent spatiotemporal semantics.

[0058] 3. This paper designs a spatiotemporal adapter module to fuse the generated tokens with the pre-trained BERT model. This module freezes the multi-head attention of the first layer while fine-tuning the remaining five layers, allowing the model to retain its language processing strength while adapting to traffic flow dynamics. It provides valuable guidance and decision-making basis for fields such as traffic flow prediction, congestion management, and traffic planning. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 The overall flow chart of the urban traffic flow prediction method and system provided by the present invention that integrates spatiotemporal features and a large language model.

[0060] Figure 2 A flow chart of the spatiotemporal compiler of the urban traffic flow prediction method and system that integrates spatiotemporal features and a large language model provided by the present invention.

[0061] Figure 3 A schematic diagram of a spatiotemporal adapter for the urban traffic flow prediction method and system provided by the present invention that integrates spatiotemporal features with a large language model.

[0062] Figure 4 This is a comparison chart of the MAPE prediction performance of each model for the METR-LA dataset.

[0063] Figure 5 This is a comparison chart of the RMSE prediction performance of each model for the METR-LA dataset.

[0064] Figure 6 This is a comparison chart of the MAE prediction performance of each model for the METR-LA dataset.

[0065] Figure 7 This is a comparison chart of the MAPE prediction performance of the PEMS-BAY dataset in each model.

[0066] Figure 8 This is a comparison chart of the RMSE prediction performance of the PEMS-BAY dataset in each model.

[0067] Fig. 9 This is a comparison chart of the MAE results of the prediction performance of the PEMS-BAY dataset in each model. DETAILED DESCRIPTION

[0068] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, the various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0069] Example 1

[0070] A method for predicting urban traffic flow that integrates spatiotemporal features with a large language model, such as Figure 1-3 As shown, the following steps are included:

[0071] A spatiotemporal compiler module is established according to the traffic flow data. The spatiotemporal compiler module extracts traffic flow features by using time step embedding, time embedding day and space embedding, and performs feature fusion to obtain tokens encapsulating spatiotemporal patterns of the traffic flow data.

[0072] The spatiotemporal compiler module includes a time step embedding unit, a time embedding day unit, a space embedding unit and a feature fusion unit. The spatiotemporal compiler module uses three embedding strategies, namely time step embedding, time embedding (day) and space embedding, to extract traffic flow features, and adopts a feature fusion method to encapsulate complex spatiotemporal patterns.

[0073] It should be noted that the data preprocessing stage includes the following steps:

[0074] A1: Load public traffic datasets such as METR-LA, PEMS-BAY, etc., extract the fields required for research, and delete useless data.

[0075] A2: Use interpolation to process missing data and remove abnormal data caused by external factors.

[0076] ① Delete data that do not match travel time and travel distance.

[0077] ② Delete outlier data through KNN algorithm.

[0078] A3: Normalize the preprocessed data so that they are within a limited range.

[0079] A4: Divide the dataset into 70% training set, 20% test set and 10% validation set.

[0080] B1: Time step embedding

[0081] The time step embedding unit is used to generate time step embeddings according to traffic flow data.

[0082] A time step represents a discrete unit in a traffic flow sequence and contains critical temporal information reflecting the changing traffic patterns at each spatial location. Embedding these time steps preserves their sequential structure, enabling the parser to capture complex temporal dependencies that are critical for accurate prediction. We employ point-wise convolution to extract the temporal dependencies from traffic flow data. Generate time-step embeddings that effectively capture the temporal pattern within each time step. The time-step embedding is calculated as follows:

[0083] E T =Conv2D(X T ,θ T )

[0084] Among them, E T represents the time step embedding, Conv2D(.) refers to a point convolution operation using a filter with a kernel size of 1×1, X T represents traffic flow data, θ T denotes the learnable weights associated with the convolutional layer and D denotes the dimensionality of the embedded hidden representation at each time step.

[0085] B2: Time embedding (days)

[0086] The time embedding day unit is used to apply the absolute position coding to the traffic flow data with day resolution to obtain the position coding, and obtain the day embedding according to the position coding.

[0087] We adopt day embeddings while excluding week embeddings. This choice is based on two main reasons. First, in many traffic flow prediction tasks, the weekly periodic characteristics are stable and can be effectively captured by day-based embeddings. Second, especially in short-term forecasts or with limited data, introducing week embeddings may complicate the model and increase the risk of overfitting. Therefore, focusing only on day embeddings enhances the generalization ability of the model, simplifies the computation, and maintains the prediction accuracy. Following the established practice in temporal data processing, we apply absolute position encoding to traffic flow data with “day” resolution, obtaining the position encoding The calculation formula for day embedding is as follows:

[0088] E D =X day ·W day

[0089] Where: E D Indicates day embedding, X day represents the position code, W day represents the matrix of learnable parameters associated with the temporal embedding day.

[0090] B3: Spatial Embedding

[0091] The spatial embedding unit is used to obtain spatial embedding according to traffic flow data by adopting an adaptive spatial embedding method.

[0092] In order to fully represent the spatial relationship between different locations in the traffic flow network, we use an adaptive spatial embedding method. The calculation formula of spatial embedding is as follows:

[0093] E S =ReLU(W s ·X T +b s )

[0094] Among them, E S represents spatial embedding, is a learnable weight matrix, is the bias vector, and ReLU represents the activation function ReLU. The ReLU activation function is used to introduce nonlinearity, thereby improving the model's expressiveness.

[0095] B4: Feature Fusion

[0096] The feature fusion unit uses convolution operations to integrate time step embedding, day embedding and space embedding to obtain tokens that encapsulate the spatiotemporal patterns of traffic flow data.

[0097] To obtain the final token representation We utilize convolution operations instead of direct connections to integrate time-step embeddings. Time Embedding and spatial embedding This approach captures temporal and spatial interactions more effectively by learning feature correlations in embeddings. The calculation formula for the tokens that encapsulate spatiotemporal patterns for traffic flow data is as follows:

[0098] E e =Conv2D(E T ||E D ||E S ,θ e )

[0099] =Conv2D(Conv2D(X T ,θ T )||X day ·W day ||ReLU(W s ·X T +b s ),θ e )

[0100] Among them, E e Represents a token, E T represents the time step embedding, E D represents day embedding, ES represents the spatial embedding, Conv2D(·) represents the convolution operation applied to the three embeddings, || represents the concatenation along the channel dimension, and θ e Represents the learnable parameters of the Conv2D convolution operation.

[0101] This convolution-based fusion produces a comprehensive token representation that encapsulates complex spatiotemporal patterns, enhancing the model's ability to understand and predict traffic flow dynamics more accurately.

[0102] S2: Design a spatiotemporal adapter module

[0103] A spatiotemporal adapter module is established based on the tokens, and the obtained tokens are fused with the pre-trained BERT model Transformer block to obtain the final traffic flow prediction.

[0104] The spatiotemporal adapter module includes an input embedding layer, a frozen pre-trained Bert model Transformer unit, and a convolutional output layer.

[0105] C1: Input Embedding Layer

[0106] The input embedding layer is used to obtain an input vector according to token embedding, position embedding and segment embedding.

[0107] The input vector is calculated as follows:

[0108] H1=E e +PE+SE

[0109] =Conv2D(E T ||E D ||E S ,θ e )+PE+SE

[0110] =Conv2D(Conv2D(X T ,θ T )||X day ·W day ‖ReLU(W s ·X T +b s ),θ e )+PE+SE

[0111] Among them, H1 represents the input vector, E e represents token, PE represents position embedding, and SE represents segment embedding. PE and SE are fine-tuned parameters. This formula effectively combines position information and segment distinction, promoting the model's understanding of the token context with respect to its position in the sequence.

[0112] C2: Freeze the pre-trained Bert model Transformer unit

[0113] The frozen pre-trained Bert model Transformer unit includes 6 Transformer blocks, which are recorded as the first Transformer block, the second Transformer block, the third Transformer block, the fourth Transformer block, the fifth Transformer block, and the sixth Transformer block. Each Transformer block includes a multi-head attention layer and a feedforward neural network module. The multi-head attention layer is used to identify the relationship and connection between different positions in the input data, and the feedforward neural network module is used to capture the temporal pattern within each token. Freeze the multi-head attention layer in the first Transformer block to retain the learned spatial semantics from pre-training. For the remaining 5 Transformer blocks, fine-tune the multi-head attention layer, feedforward neural network module, and layer normalization to effectively adapt the model to the prediction task.

[0114] In this section, we implemented a strategic approach to fine-tune the BERT Transformer architecture, which consists of six layers. Each Transformer block includes two main components: multi-head attention and feed-forward neural network (FFN) modules. The multi-head attention mechanism contributes to the model's ability to identify spatial dependencies between tokens, enabling it to identify relationships and connections at different locations in the input data. On the other hand, the FFN module captures the temporal patterns within each token, thereby extracting sequential trends and improving the model's ability to analyze spatiotemporal dynamics.

[0115] To preserve the integrity of the pre-trained knowledge, we freeze the multi-head attention layer in the first Transformer block, retaining the learned spatial semantics from pre-training. This selective freezing ensures that the core semantic understanding is preserved while also reducing the number of parameters that need to be tuned. In the remaining five layers, we fine-tune the multi-head attention, FFN, and layer normalization (LN) components to effectively adapt the model to the prediction task.

[0116] For the frozen first Transformer block, the output of the input embedding H1 is processed through a frozen multi-head attention layer, followed by layer normalization (LN), as follows:

[0117]

[0118] in, represents the frozen output of the first Transformer block, FrozenMHA(H1) means applying the frozen multi-head attention component to H1, and LN means normalizing the output of each layer.

[0119] The calculation of FrozenMHA remains frozen, expressed as:

[0120] FrozenMHA(H1)=Concat(FrozenAttention1,...FrozenAttention n )W O .

[0121] Each FrozenAttention i The calculation is as follows:

[0122]

[0123] Among them, FrozenAttention i represents the i-th frozen attention head, i = 1, 2, ..., n, n represents the number, Concat represents concatenation, and represents the trainable weight matrix used to generate query, key and value representations, d k Indicates the dimension of the key.

[0124] While for the remaining 5 Transformer blocks, the multi-head attention layer, feed-forward neural network module, and layer normalization are fine-tuned to effectively adapt the model to the prediction task. The model undergoes fine-tuning not only in the multi-head attention layer (MHA), but also in the feed-forward neural network module (FFN) and layer normalization (LN).

[0125] The calculation of the lth layer is performed as follows:

[0126]

[0127] in, Represents the output of the previous layer, and the final output is Given,l,=2,.,6,,LN denotes layer normalization, FFN denotes feed-forward neural network, and MHA denotes multi-head attention.

[0128] In this equation, Multi-Head Attention (MHA) plays a core role in capturing the spatial dependencies in the data. Specifically, MHA is calculated as:

[0129]

[0130] Each attention iUse the following formula to calculate:

[0131]

[0132] in, is the input of the previous layer, W O represents the output projection matrix, and is a trainable weight matrix used to generate query, key, and value representations, d k represents the dimension of the key, and the softmax function is used to normalize the attention scores and generate weights for each attention head.

[0133] Following MHA, a feed-forward neural network module is introduced to capture the temporal dependencies within the output of a layer, which is defined as:

[0134] FFN(H)=ReLU(HW1+b1)W2+b2

[0135] Among them, FFN(H) represents the output after the feedforward neural network module, H represents the output after multi-head attention, and W1, W2, b1 and b2 are adjustable parameters related to the network.

[0136] This two-layer feed-forward network adds nonlinear transformations to improve the model's ability to capture complex temporal patterns that are critical for accurate predictions.

[0137] Finally, layer normalization (LN) is applied to ensure stability during training by normalizing the output features of each layer. The calculation of layer normalization is defined as:

[0138]

[0139] Among them, LN(H (l-1) ) represents the layer normalization operation on the output of the previous layer, H (l-1) is the feature representation or output of the previous layer, μ and σ 2 Respectively represent H (l-1) The parameters γ and β are learnable scale and shift terms, ⊙ represents multi-element multiplication, and ∈ is a small constant added to avoid division by zero.

[0140] C3: Convolutional output layer

[0141] The convolutional output layer is used to transform the output features of the last Transformer block into the final prediction of traffic flow.

[0142] This layer uses the feature representation obtained from the sixth layer of the pre-trained model And apply the convolution operation to generate the regression output. We calculate the output prediction of traffic flow The final traffic flow prediction calculation formula is as follows:

[0143]

[0144] Among them, Y S represents the final prediction of traffic flow, is the feature representation obtained by the sixth Transformer block, W conv represents the learnable convolution weights, b conv Represents a learnable convolution bias term.

[0145] The convolution operation is used to capture the spatial dependencies within the feature map from the previous layer, thereby enabling effective prediction of traffic flow at subsequent time steps. To introduce nonlinearity and enhance the model’s capabilities, we apply an activation function (such as ReLU) to the predicted output Y S .

[0146] Example 2

[0147] This embodiment provides an urban traffic flow prediction system that integrates spatiotemporal features and a large language model, and adopts the urban traffic flow prediction method that integrates spatiotemporal features and a large language model, including a spatiotemporal compiler module and a spatiotemporal adapter module. The spatiotemporal compiler module extracts traffic flow features by using time step embedding, time embedding day and space embedding, and performs feature fusion to obtain tokens that encapsulate spatiotemporal patterns about traffic flow data. The spatiotemporal adapter module freezes the pre-trained Bert model Transformer block, allowing the model to retain its language processing strength while adapting to traffic flow dynamics, and fuses the obtained tokens with the pre-trained BERT model Transformer block to obtain the final traffic flow prediction.

[0148] Example 3

[0149] This embodiment provides an electronic device, including a processor and a memory, wherein the memory is used to store processor executable instructions, and the processor is configured to call the instructions stored in the memory to execute the urban traffic flow prediction method that integrates spatiotemporal features with a large language model.

[0150] The spatiotemporal compiler module of the present invention converts raw traffic data into LLM-compatible tokens, and the parser applies time steps, time and space embeddings to effectively represent spatiotemporal semantics. The spatiotemporal adapter module fuses the generated tokens with the pre-trained BERT model. This module freezes the multi-head attention of the first layer while fine-tuning the remaining five layers, allowing the model to retain its language processing strength while adapting to traffic flow dynamics. The present invention combines large language models with spatiotemporal features, and uses the advanced learning capabilities of LLM to model spatial and temporal patterns in traffic data, thereby achieving highly accurate predictions. The present invention can provide valuable guidance and decision-making basis for fields such as traffic flow prediction, congestion management and traffic planning.

[0151] Example 4

[0152] Reference Figure 1 , 2 , 3 is an example of the present invention, which provides an urban traffic flow prediction method that integrates spatiotemporal features and a large language model. In order to verify its beneficial effects, the comparison results of two schemes are provided.

[0153] The embodiment of the present invention adopts two public traffic data sets METR-LA and PEMS-BAY.

[0154] The METR-LA dataset includes traffic data from 207 sensors on Los Angeles freeways. It covers the time span from March 1, 2012 to June 30, 2012, with a total of 34,272 time steps, where each time step represents a 5-minute interval. The PEMS-BAY dataset includes data from 325 sensors in the Bay Area of ​​California. It covers the time span from January 1, 2017 to May 31, 2017, with a total of 52,116 time steps, also at 5-minute intervals.

[0155] For both datasets, we follow the traditional pre-processing method, normalizing the 5-minute interval data and splitting them into 70% for training, 10% for validation, and 20% for testing.

[0156] We use our invention to predict traffic flow for the upcoming 12 time steps, using data from the previous 12 time steps as input. The time dimension Td is set to 288, and each time step represents a 5-minute interval. All experiments are performed on an RTX 4090D GPU with 24GB of memory. The model is trained using the Ranger21 optimizer with a learning rate of 0.0001. The LLM architecture has a 6-layer BERT model, where the attention mechanism of the first layer is frozen. The training is run for 25 epochs with a batch size of 32, and the results are averaged across the prediction steps.

[0157] To evaluate the effectiveness of our invention, we rely on three main metrics: mean absolute error (MAE), root mean square error (RMSE), and mean absolute percentage error (MAPE).

[0158] In the experiments, we compare our invention with 8 baseline models, which can be divided into the following categories: statistical methods (HA, ARIMA), graph-based methods (STGCN, DCRNN, GWNet), pre-trained models (STEP), and large language models (GPT2, GATGPT).

[0159] from Figure 4-9 It can be seen from the comparison chart of the prediction performance of each model that the present invention shows excellent performance on both the METR-LA and PEMS-BAY datasets, and the present invention outperforms the eight baseline models in all three prediction ranges (15 minutes, 30 minutes, and 60 minutes). On the METR-LA dataset, the present invention shows excellent performance in all three prediction ranges in terms of MAE, RMSE, and MAPE, demonstrating its advantage in traffic flow prediction. Specifically, for the 15-minute prediction, the MAE of the present invention is 2.34, which is significantly lower than STEP (2.61), and the MAPE is 5.71%. This sustained performance is also verified in the 30-minute and 60-minute predictions, especially in the 60-minute prediction, the present invention maintains a leading position with a MAE of 3.00 and a MAPE of 8.15%. Similarly, in the PEMS-BAY dataset, the present invention similarly demonstrates its excellent prediction ability. It achieves a MAE of 1.05, a RMSE of 2.11, and a MAPE of 2.10%, outperforming other models in all indicators. Similar to the previous datasets, the present invention shows the best performance in 15-minute, 30-minute, and 60-minute predictions, especially in the 15-minute prediction, where its MAE exceeds Gemma2 (1.13). This consistent performance demonstrates the strong generalization and adaptability of the present invention on different datasets. In summary, the prediction accuracy of the present invention on both the METR-LA and PMS-BAY datasets is significantly improved, confirming its effectiveness in the field of traffic flow prediction. This shows that the present invention can effectively capture the complex patterns of traffic flow changes, provide more accurate short-term traffic flow predictions, and is applicable to different traffic data sets, showing a wide range of application potential.

[0160] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for predicting urban traffic flow by integrating spatiotemporal features with a large language model, characterized in that: The following steps are involved: Establishing a spatiotemporal compiler module according to the traffic flow data, wherein the spatiotemporal compiler module extracts traffic flow features by using time step embedding, time embedding day and space embedding and performs feature fusion to obtain tokens encapsulating spatiotemporal patterns of the traffic flow data; A spatiotemporal adapter module is established based on the tokens. The spatiotemporal adapter module allows the model to retain its language processing strength while adapting to the traffic flow dynamics by freezing the pre-trained BERT model Transformer block. The obtained tokens are fused with the pre-trained BERT model Transformer block to obtain the final traffic flow prediction.

2. The urban traffic flow prediction method according to claim 1, which integrates spatiotemporal features and a large language model, is characterized by: The spatiotemporal compiler module includes a time step embedding unit, a time embedding day unit, a space embedding unit and a feature fusion unit, wherein the time step embedding unit is used to generate a time step embedding according to traffic flow data; The time embedding day unit is used to apply the absolute position coding to the traffic flow data with a day resolution to obtain the position coding, and obtain the day embedding according to the position coding; the space embedding unit is used to obtain the space embedding according to the traffic flow data by adopting the adaptive space embedding method; The feature fusion unit uses convolution operations to integrate time step embedding, day embedding and space embedding to obtain tokens that encapsulate the spatiotemporal patterns of traffic flow data.

3. The urban traffic flow prediction method according to claim 2, which integrates spatiotemporal features and a large language model, is characterized by: The spatiotemporal adapter module includes an input embedding layer, a frozen pre-trained Bert model Transformer unit, and a convolutional output layer. The input embedding layer is used to obtain an input vector according to token embedding, position embedding, and segment embedding. The frozen pre-trained Bert model Transformer unit includes 6 Transformer blocks, which are respectively recorded as the first Transformer block, the second Transformer block, the third Transformer block, the fourth Transformer block, the fifth Transformer block, and the sixth Transformer block. Each Transformer block includes a multi-head attention layer and a feedforward neural network module. The multi-head attention layer is used to identify the relationship and connection between different positions in the input data, and the feedforward neural network module is used to capture the time pattern within each token. The multi-head attention layer in the first Transformer block is frozen to retain the learned spatial semantics from pre-training. The remaining 5 Transformer blocks fine-tune the multi-head attention layer, the feedforward neural network module, and the layer normalization to effectively adapt the model to the prediction task. The convolutional output layer is used to convert the output features of the last Transformer block into the final prediction of traffic flow.

4. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 3 is characterized by: For the frozen first Transformer block, the output of the input embedding H1 is processed through a frozen multi-head attention layer, followed by layer normalization, as follows: in, represents the frozen output of the first Transformer block, FrozenMHA(H1) means applying the frozen multi-head attention component to H1, and LN means normalizing the output of each layer; The calculation of FrozenMHA remains frozen, expressed as: FrozenMHA(H1)=Concat(FrozenAttention1,...FrozenAttention n )W O ; Each FrozenAttention i The calculation is as follows: Among them, FrozenAttention i represents the i-th frozen attention head, i = 1, 2, ..., n, n represents the number, Concat represents the concatenation operation, E T represents the time step embedding, E D represents day embedding, E S represents the spatial embedding, Conv2D(·) represents the convolution operation applied to the three embeddings, || represents the concatenation along the channel dimension, and θ e represents the learnable parameters of the Conv2D convolution operation, PE represents position embedding, SE represents segment embedding, and represents the trainable weight matrix used to generate query, key and value representations, d k Represents the dimension of the key; The remaining five Transformer blocks fine-tune the multi-head attention layer, feed-forward neural network module, and layer normalization to effectively adapt the model to the prediction task; The calculation of the lth layer is performed as follows: in, Represents the output of the previous layer, and the final output is Given,l,=2,.,6,,LN represents layer normalization, FFN represents feed-forward neural network, and MHA represents multi-head attention; The calculation formula of MHA is: Each attention i Use the following formula to calculate: in, is the input of the previous layer, W O represents the output projection matrix, and is a trainable weight matrix used to generate query, key, and value representations, d k Represents the dimension of the key; After MHA, a feed-forward neural network module is introduced to capture the temporal dependencies within the output of the layer; FFN(H)=ReLU(HW1+b1)W2+b2 Among them, FFN(H) represents the output after the feedforward neural network module, H represents the output after multi-head attention, and W1, W2, b1 and b2 are adjustable parameters related to the network; Finally, layer normalization is applied to ensure stability during training by normalizing the output features of each layer. The calculation of layer normalization is: Among them, LN(H (l-1) ) represents the layer normalization operation on the output of the previous layer, H (l-1) is the feature representation or output of the previous layer, μ and σ 2 Respectively represent H (l-1) The mean and variance of , the parameters γ and β are learnable scale and shift terms, ⊙ represents multi-element multiplication, and ∈ is a small constant added.

5. The urban traffic flow prediction method according to claim 4, which integrates spatiotemporal features and a large language model, is characterized by: The input vector is calculated as follows: H1=Conv2D(Conv2D(X T ,θ T )||X day ·W day ||ReLU(W s ·X T +b s ),θ e )+OR+IF Where H1 represents the input vector, Conv2D(.) refers to the point convolution operation using a filter with a kernel size of 1×1, and X T represents traffic flow data, θ T represents the learnable weights associated with the convolutional layer, D represents the dimension of the hidden representation embedded at each time step, and X day represents the position code, W day represents the learnable parameter matrix associated with the temporal embedding day, is a learnable weight matrix, is the bias vector, ReLU represents the activation function ReLU, PE represents position embedding, and SE represents segment embedding.

6. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 5 is characterized by: The final traffic flow prediction calculation formula is as follows: Among them, Y S represents the final prediction of traffic flow, is the feature representation obtained by the sixth Transformer block, W conv represents the learnable convolution weights, b conv Represents a learnable convolution bias term.

7. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 6 is characterized by: The calculation formula of time step embedding is as follows: E T =Conv2D(X T ,i T ) Among them, E T represents the time step embedding, Conv2D(.) refers to the point convolution operation using a filter with a kernel size of 1×1, X T represents traffic flow data, θ T represents the learnable weights associated with the convolutional layer, and D represents the dimension of the hidden representation embedded at each time step; The calculation formula for day embedding is as follows: HAVE BEEN D =X day ·W day Where: E D represents the day embedding, X day represents the position code, W day represents the learnable parameter matrix associated with the temporal embedding day; The calculation formula of spatial embedding is as follows: HAVE BEEN S =ReLU(W s X T +b s ) Among them, E S represents spatial embedding, is a learnable weight matrix, is the bias vector, and ReLU represents the activation function ReLU.

8. The urban traffic flow prediction method integrating spatiotemporal features and a large language model according to claim 7 is characterized by: The calculation formula for the tokens that encapsulate the spatiotemporal pattern of traffic flow data is as follows: E e =Conv2D(Conv2D(X T ,θ T )||X day ·IN day ||ReLU(W s ·X T +b s ),θ e ) Among them, E e Represents a token, E T represents the time step embedding, E D represents day embedding, E S represents the spatial embedding, Conv2D(·) represents the convolution operation applied to the three embeddings, || represents the concatenation along the channel dimension, and θ e Represents the learnable parameters of the Conv2D convolution operation.

9. An urban traffic flow prediction system integrating spatiotemporal features and a large language model, characterized in that: The urban traffic flow prediction method of integrating spatiotemporal features with a large language model as described in claims 1-8 includes a spatiotemporal compiler module and a spatiotemporal adapter module. The spatiotemporal compiler module extracts traffic flow features by using time step embedding, time embedding day and space embedding, and performs feature fusion to obtain tokens that encapsulate spatiotemporal patterns about traffic flow data; the spatiotemporal adapter module freezes the pre-trained Bert model Transformer block, allowing the model to retain its language processing strength while adapting to traffic flow dynamics, and fuses the obtained tokens with the pre-trained BERT model Transformer block to obtain the final traffic flow prediction.

10. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory is used to store processor-executable instructions, and the processor is configured to call the instructions stored in the memory to execute any one of the methods described in claims 1-8.

Images