A traffic prediction method based on a distilled large language model

Abstract

This invention discloses a traffic prediction method based on a distilled large language model, belonging to the field of spatiotemporal data mining in intelligent transportation systems. The method includes: constructing an original dataset based on historical traffic data collected by traffic sensors; standardizing the data using the Z-Score method; converting the graph structure data of the traffic network into a sequence input acceptable to the large language model through graph data serialization; embedding and extracting spatiotemporal features from the historical traffic data using a dynamic spatiotemporal DC network; extracting hidden features based on a pre-trained large language model and constructing a distillation module to align the features; transferring the knowledge of the language model to the DSTN by calculating the distillation loss, and optimizing the prediction output based on Huber loss; and outputting the optimized DSTN with traffic state prediction results for future time periods. This invention improves the model's generalization and transfer capabilities under sparse data conditions.

Description

Technical Field

[0001] This invention belongs to the field of spatiotemporal data mining in intelligent transportation systems, and particularly relates to a traffic prediction method based on a distilled large language model. Background Technology

[0002] Traffic forecasting, as the cornerstone of ITS (Information Technology System), directly impacts the efficiency of traffic network scheduling, travel services, and safety management. However, the limited data available in reality severely restricts forecast accuracy, making the development of forecasting methods adapted to such scenarios extremely necessary. From the current data acquisition perspective, the completeness and continuity of traffic data are often compromised by multiple factors. Insufficient coverage of monitoring equipment in remote road sections and old urban areas makes it difficult to collect sufficient flow and speed data; extreme weather, equipment failure, or network outages can lead to data collection interruptions, resulting in a large number of missing values; traffic data from some sensitive areas cannot be publicly shared due to privacy protection or management restrictions, further exacerbating the data scarcity problem. These objectively existing data shortcomings make it difficult to implement deep learning forecasting models that rely on massive amounts of data for training.

[0003] Currently, most advanced traffic prediction methods rely on deep learning models. These models require iterative parameter optimization using large-scale labeled data, making them prone to overfitting and excessive prediction bias in scenarios with sparse or missing data. In light of this, existing technologies have proposed cross-regional traffic prediction methods. These methods train a pre-trained model in a data-rich region (the source region) and then transfer it to a data-limited region (the target region) for fine-tuning, thereby achieving traffic prediction. However, the effectiveness of this method depends on the source and target regions having similar data distributions. When a source region with a similar data distribution cannot be found in the target region, the performance of this method will be affected. Therefore, how to construct a reliable traffic prediction method with only a limited amount of data in the target region is a pressing problem that needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a traffic prediction method based on a distilled large language model, comprising:

[0005] Based on historical traffic data collected by traffic sensors, obtain a raw dataset containing traffic status information such as flow rate and speed;

[0006] Based on the mean and variance of the original dataset, the original data is Z-Score standardized to obtain normalized data.

[0007] Based on the time dimension and sensor dimension of the normalized data, the input batch data of the dynamic spatiotemporal DC network is constructed.

[0008] Based on the batch data, the graph structure data is converted into a sequence form acceptable to the GPT2 model and dimension mapping is completed through graph data serialization.

[0009] Based on the sequence-form data, historical traffic information is input into a dynamic spatiotemporal DC network for embedding and encoding to obtain spatiotemporal features;

[0010] Based on the pre-trained large language model GPT2, the hidden layer features of the corresponding layer are extracted as distillation source features; based on the output features of the dynamic spatiotemporal DC network and the distillation source features of GPT2, the distillation loss is calculated to achieve feature alignment;

[0011] The parameters of the dynamic spatiotemporal DC network are optimized based on the distillation loss, and the deviation between the predicted output and the true value is evaluated using Huber loss.

[0012] The optimized dynamic spatiotemporal DC network outputs traffic state prediction results for future time periods.

[0013] Optionally, the graph data serialization process includes: unfolding the spatial topology of each sensor node into a temporal arrangement based on the adjacency matrix of the traffic network, and combining them into an input sequence according to the time window length.

[0014] Optionally, the input layer of the dynamic spatiotemporal DC network includes a fully connected layer with an activation function, which maps historical traffic data according to weight parameters and bias parameters to obtain hidden layer feature representations.

[0015] Optionally, the dynamic spatiotemporal DC network includes multiple dynamic spatiotemporal modules that aggregate spatial information between sensor nodes according to a spatial aggregation function and aggregate time series information according to a time aggregation function to obtain spatiotemporal fusion features.

[0016] Optionally, the spatial aggregation function calculates the correlation between the query, key, and value vectors based on the node adjacency relationship, and completes spatial feature aggregation through the Hadamard product and activation function.

[0017] Optionally, the time aggregation function generates a weight distribution in the time dimension based on the correlation of each time step in the time window through a gating mechanism and bias mapping, and completes the aggregation of time features.

[0018] Optionally, the distillation process of the GPT2 model includes: misaligning the i-th layer features of the dynamic spatiotemporal DC network with the j-th layer features of GPT2, adjusting the feature dimensions through pooling or mapping to calculate the mean squared error, and realizing knowledge transfer.

[0019] Optionally, the optimization process employs an adaptive moment estimation optimizer, which updates parameters based on a set learning rate and batch size to obtain the final traffic prediction model.

[0020] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0021] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0022] Compared with the prior art, the present invention has the following advantages and technical effects:

[0023] This invention introduces a large language model distillation mechanism to transfer knowledge from the pre-trained model GPT2 to a dynamic spatiotemporal DC network, thereby maintaining stable prediction performance in traffic scenarios with limited data. By designing a graph data serialization method, the non-Euclidean topology data of the traffic network can be adapted to the input structure of the large language model, expanding the application scope of the large language model in spatiotemporal prediction tasks. The spatial and temporal aggregation functions in DSTN respectively realize the multidimensional dependency modeling of traffic data, improving the model's ability to express complex traffic evolution patterns. The misalignment strategy of the distillation module reduces computational resource consumption while ensuring feature consistency, making knowledge transfer more efficient. A two-stage training process realizes an optimized path from knowledge distillation to target region fine-tuning, significantly improving the model's prediction accuracy and robustness under small sample conditions. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0028] Example 1

[0029] like Figure 1 As shown, this embodiment provides a traffic prediction method based on a distilled large language model, including:

[0030] Based on historical traffic data collected by traffic sensors, obtain a raw dataset containing traffic status information such as flow rate and speed;

[0031] Based on the mean and variance of the original dataset, the original data is Z-Score standardized to obtain normalized data.

[0032] Based on the time dimension and sensor dimension of the normalized data, the input batch data of the dynamic spatiotemporal DC network is constructed.

[0033] Based on the batch data, the graph structure data is converted into a sequence form acceptable to the GPT2 model and dimension mapping is completed through graph data serialization.

[0034] Based on the sequence-form data, historical traffic information is input into a dynamic spatiotemporal DC network for embedding and encoding to obtain spatiotemporal features;

[0035] Based on the pre-trained large language model GPT2, the hidden layer features of the corresponding layer are extracted as distillation source features; based on the output features of the dynamic spatiotemporal DC network and the distillation source features of GPT2, the distillation loss is calculated to achieve feature alignment;

[0036] The parameters of the dynamic spatiotemporal DC network are optimized based on the distillation loss, and the deviation between the predicted output and the true value is evaluated using Huber loss.

[0037] The optimized dynamic spatiotemporal DC network outputs traffic state prediction results for future time periods.

[0038] The specific process is as follows:

[0039] Step 1: Dataset and Data Preprocessing;

[0040] This embodiment uses a publicly available traffic dataset, with the dataset having the following dimensions: .in Indicates the number of samples in the dataset; Indicates the number of traffic sensors; This represents the number of traffic states, which include flow rate, speed, etc. To improve training efficiency, this embodiment uses Z-Score to standardize the data, scaling the mean and variance of the dataset to 0 and 1, respectively. Considering that deep learning models typically use stochastic gradient descent for optimization, the dataset is divided into multiple batches, each batch having a dimension of [dimension value missing]. .in, This represents the length of the historical and future time windows. The dataset is divided into training and test sets, with a small batch size for the training set.

[0041] Step 2: Data embedding;

[0042] Using a non-replacement approach, this embodiment randomly samples multiple batches of data from the dataset, where each batch of data is divided into historical data. and future data This embodiment predicts traffic data using a Dynamic Spatiotemporal DC Network (DSTN), a deep learning model commonly used for traffic prediction. Its input layer maps historical data to embedded data through a fully connected layer with an activation function. :

[0043] ;

[0044] in, and Indicates the weighting parameters and biases; Indicates the dimension of the hidden layer.

[0045] Since the large language model GPT2 will be used later, its input layer will be described here. Considering historical data... The dimensions cannot be adapted to the input size of GPT2. This embodiment uses graph data serialization to transform the dimensions. Size conversion Then it is mapped to through a fully connected layer. .in, It is the feature dimension required by GPT2.

[0046] Step 3, Dynamic Spatiotemporal Module:

[0047] The Dynamic Spatiotemporal Module (DSTM) is the core module of the DSTN, consisting of... The first layer is primarily used to aggregate spatiotemporal information from traffic data. The DSTM calculation for the layer is as follows:

[0048] ;

[0049] in, ; Indicates the first The input / output of the DSTM layer; This represents the adjacency matrix, used to represent the associations between sensor nodes; This represents a spatial aggregation function used to aggregate spatial information; This represents a time aggregation function used to aggregate time information. The calculation is as follows:

[0050] ;

[0051] in, Indicates weight, These are mapped to Query, Key, and Value, respectively. Indicates transpose; It is a commonly used activation function; It represents the Hadamardi (or Hadama) stack. The calculation is as follows:

[0052] = ( ;

[0053] in, These are weights, used to describe the correlation between each time step within a time window; the gating mechanism uses these weights. and deviation and Achieve Nonlinear mapping; This represents the activation function.

[0054] Step 4: Multi-head attention and feedforward fully connected network;

[0055] GPT2 is a pre-trained model, and its core module is... The multi-head attention and feedforward fully connected networks of the layers are computed as follows:

[0056] ;

[0057] ;

[0058] in, This represents a multi-head attention function. Indicates a feedforward fully connected network; ; Indicates the first Input / output of multi-head attention layer; Indicates the first Input / output of a layer feedforward fully connected network; The layer normalization function is indicated by the "" at the end of the equation. "and This indicates a residual connection.

[0059] Step 5, distillation module;

[0060] Given the limited training data, this embodiment designs a distillation module to transfer knowledge from GPT2 to DSTN. To achieve this goal, this embodiment designs a distillation loss mechanism. This forces the output features of the two models to be similar.

[0061] ;

[0062] in, This represents the mean squared error; taking into account the number of layers in GPT2. Typically much larger than the number of layers in a DSTM. This embodiment uses a misaligned alignment method for distillation. By aligning the two layers, the outputs of the LLM and DSTM layers are forced to approximate each other, thereby achieving LLM guidance of the DSTM (i.e., distillation). Layer DSTM feature alignment Layer GPT2. Considering and Due to inconsistent sizes, this embodiment utilizes pooling. Mapping Change in the way The size of the GPT2 sensor. DSTM learns from GPT2 by aligning with its features. It's important to note that the GPT2 parameters are frozen and not updated.

[0063] Step 6: Data output;

[0064] This embodiment does not require the use of GPT2 output, therefore there is no need to describe the GPT output layer. Features acquired by DSTM The signals are fed into the output layer to perform traffic prediction. The output layer consists of a fully connected layer with an activation function. Mapped to predicted future data This embodiment uses the Huber loss assessment model to predict values. Compared with the true value distance ;

[0065] ;

[0066] in, It is a threshold used to change the pattern of the loss function.

[0067] Step 7: Model training and parameter setting;

[0068] The model training consists of two phases: Phase 1 distills knowledge from GPT2 and transfers it to DSTN. During this phase, the GPT2 model parameters are frozen throughout and do not participate in optimization updates. Phase 2 fine-tunes DSTN using a small amount of traffic data. DSTN uses adaptive moments as the optimizer with a learning rate of 0.01. Both phases have 100 training iterations, and the batch size for each training iteration is 8. Other hyperparameter settings are as follows: , Determined by the dataset used. , , , , Furthermore, this embodiment uses commonly used regression evaluation metrics to evaluate the model's performance, namely mean absolute error, mean absolute percentage error, and root mean square error.

[0069] On the other hand, this embodiment also provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0070] On the other hand, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0071] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A traffic prediction method based on a distilled large language model, characterized in that, include: Based on historical traffic data collected by traffic sensors, obtain the raw dataset of traffic flow, speed and traffic status information; Based on the mean and variance of the original dataset, the original data is Z-Score standardized to obtain normalized data. Based on the time dimension and sensor dimension of the normalized data, the input batch data of the dynamic spatiotemporal DC network is constructed. Based on the batch data, the graph structure data is converted into a sequence form acceptable to the GPT2 model and dimension mapping is completed through graph data serialization. Based on the sequence-form data, historical traffic information is input into a dynamic spatiotemporal DC network for embedding and encoding to obtain spatiotemporal features; Based on the pre-trained large language model GPT2, the hidden layer features of the corresponding layer are extracted as distillation source features; Based on the output characteristics of the dynamic spatiotemporal DC network and the distillation source characteristics of GPT2, the distillation loss is calculated to achieve feature alignment; The expression for the distillation loss is: ； in, Indicates the mean squared error; For the first A layered DSTM, taking into account the number of layers in GPT2. The number of layers greater than DSTM Distillation is performed using a misaligned alignment method. This alignment forces the outputs of the LLM and DSTM layers to approximate each other, thereby enabling the LLM to guide the DSTM, i.e., the first... Layer DSTM feature alignment Layer GPT2; considering and Due to inconsistent sizes, pooling is used. Mapping Change in the way Dimensions; DSTM learns knowledge of GPT2 by aligning with the features of GPT2; The parameters of the dynamic spatiotemporal DC network are optimized based on the distillation loss, and the deviation between the predicted output and the true value is evaluated using Huber loss. The optimized dynamic spatiotemporal DC network outputs traffic state prediction results for future time periods.

2. The method according to claim 1, characterized in that, The process of graph data serialization includes: unfolding the spatial topology of each sensor node into a temporal arrangement based on the adjacency matrix of the traffic network, and combining them into an input sequence according to the time window length.

3. The method according to claim 1, characterized in that, The input layer of the dynamic spatiotemporal DC network includes a fully connected layer with an activation function, which maps historical traffic data according to weight parameters and bias parameters to obtain hidden layer feature representations.

4. The method according to claim 1, characterized in that, The dynamic spatiotemporal DC network includes multiple dynamic spatiotemporal modules, which aggregate spatial information between sensor nodes according to a spatial aggregation function and aggregate time series information according to a time aggregation function to obtain spatiotemporal fusion features.

5. The method according to claim 4, characterized in that, The spatial aggregation function calculates the correlation between the query, key, and value vectors based on the node adjacency relationship, and completes spatial feature aggregation through the Hadamard product and activation function.

6. The method according to claim 4, characterized in that, The time aggregation function generates a weight distribution in the time dimension based on the correlation of each time step in the time window through a gating mechanism and bias mapping, and completes the aggregation of time features.

7. The method according to claim 1, characterized in that, The distillation process of the GPT2 model includes: misaligning the i-th layer features of the dynamic spatiotemporal DC network with the j-th layer features of GPT2, adjusting the feature dimensions through pooling or mapping to calculate the mean squared error, and thus achieving knowledge transfer.

8. The method according to claim 1, characterized in that, The optimization process employs an adaptive moment estimation optimizer, which updates parameters based on a set learning rate and batch size to obtain the final traffic prediction model.

9. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-8.

Images