Traffic prediction method combining dynamic GCN and fine-tuning GPT2

Abstract

The invention discloses a traffic prediction method combining dynamic GCN and fine-tuning GPT2, and the method comprises the steps: defining a traffic road map structure: employing road segments as nodes, employing the traffic flow upstream and downstream relation between the road segments as edges, constructing a road network map, and representing the space structure of an urban road; namely, if the traffic flow exists between the two road sections, it is defined that a connecting edge exists between the nodes of the two road sections, and an adjacent matrix is designed; preprocessing the input traffic historical data, including missing value filling, abnormal value detection and data standardization; the preprocessed traffic road network data, including historical data and a constructed road network graph structure, are sent to a GCN network, and time-space relevance is learned; fine tuning of a GPT2-based prediction model: combining GCN to perform fine tuning of GPT-2; the dynamic GCN introduced in the method helps the fine tuning GPT2 model pay attention to the spatial relationship of the traffic network graph, and the best prediction precision is obtained.

Description

Technical Field

[0001] The present invention relates to the fields of pattern recognition and deep learning, as well as large language models and time series prediction technology, and in particular to a method for fine-tuning a large prediction model in combination with a dynamic graph neural network and adapting it to traffic flow prediction. Background Art

[0002] In recent years, with the advancement of smart city construction, the expansion and informatization level of urban roads have been significantly improved. Although urban road infrastructure has been rapidly developed, the growing urban population and number of motor vehicles, as well as the resulting traffic demand, have far exceeded the construction speed of urban transportation systems. As a result, many cities are facing problems such as frequent traffic accidents, traffic congestion and traffic pollution, which seriously affect the quality of life of residents and the traffic efficiency of urban vehicles, and become an obstacle to the sustainable development of modern cities. Especially in megacities with a population of more than 10 million, the transportation system is an open and complex system that integrates massive demand management, traffic flow spatiotemporal distribution regulation, comprehensive behavior prediction, and energy and environmental optimization, and faces huge challenges. In order to solve these problems, many researchers have proposed the construction of an intelligent transportation system (ITS), using sensors, computers, artificial intelligence and other technologies to improve the utilization rate of urban road resources. The premise of building an intelligent transportation system and realizing intelligence is the effective use of traffic big data, and traffic prediction is the most important and basic method for the effective use of traffic big data. Traffic forecasting trains network models with historical traffic data to predict future traffic data. The forecast results can provide important data support for optimizing the traffic system, implementing traffic management measures, and warning of traffic congestion. It also provides people with travel route planning and other related assistance. Therefore, traffic forecasting research is one of the hot topics in intelligent transportation research.

[0003] In existing traffic forecasting research, methods based on graph convolutional networks (GCN), long short-term memory networks (LSTM), and transformers are widely used. These methods usually rely on a large amount of reliable data to complete model training. For example, GCN makes predictions by capturing the spatial characteristics of the traffic network, LSTM is good at processing time series data to predict traffic flow, and the transformer model shows excellent performance in traffic forecasting through its powerful sequence modeling capabilities. However, such data-driven methods face challenges in the context of smart city expansion, because many emerging transportation systems lack a large amount of comprehensive historical data. This lack of data makes it difficult for traditional traffic forecasting methods to meet the needs of smart city expansion for traffic forecasting. Therefore, how to make effective traffic forecasts in the absence of sufficient data has become an important direction of current research.

[0004] Existing traffic prediction methods often ignore the possibility of solving this problem through trained large language models (LLMs) when dealing with data-deficient smart city expansion scenarios. LLM-based methods have achieved remarkable results in many fields such as natural language processing, demonstrating their strong generalization and context understanding capabilities. However, directly training a specific large model in a new scenario usually requires a large data set and huge computing resources, which is not feasible in many practical applications. Recent studies have shown that adapting the LLM model trained in the language field to the time series data field through fine-tuning can allow the large model to achieve excellent results in new scenarios. This method utilizes the powerful context-related prior knowledge that already exists in the LLM model, enabling it to better extract and understand the features of time series data. Therefore, for traffic flow prediction scenarios with insufficient data, the present invention proposes an innovative method: adapting the trained LLM model to the traffic prediction task through fine-tuning. In this way, the model can still effectively predict traffic flow in the absence of a large amount of historical data, thereby improving the accuracy and reliability of the prediction. This approach not only reduces the dependence on large-scale data and computing resources, but also provides a new solution for traffic management in smart cities. Summary of the invention

[0005] This paper focuses on the particularity of traffic flow prediction, combines dynamic graph convolutional networks, designs a specific fine-tuning strategy, and proposes a traffic prediction method that combines dynamic GCN and fine-tuned GPT-2. This method aims to solve the difficulty of traffic prediction caused by insufficient data during the expansion of smart cities.

[0006] The technical solution steps of the present invention are as follows:

[0007] Step 1: Define the traffic prediction problem:

[0008] Model the traffic data (traffic flow data or traffic speed data, etc.) of different road sections and different time periods in the city, and set represents the traffic data of road section i at time t, where F represents the characteristic dimension of traffic data, then the traffic data of all road sections at time t can be expressed as Where N represents the total number of road sections in the entire urban road network, the traffic data sequence at time (1, 2, …, τ) can be expressed as 1:τ =[X1,X2,…,X τ ] T ∈R N×F×τ Therefore, given a fixed road network graph G and T before time t P Traffic history data for time frames Input to the trained model In the prediction, T F Traffic data by time frame

[0009] Step 2: Build a traffic network diagram:

[0010] S21 defines the traffic road graph structure: The present invention uses road sections as nodes and the upstream and downstream relationships of traffic between road sections as edges to construct a road network graph, thereby representing the spatial structure of urban roads. That is, if there is traffic between two road sections, it is defined that there is a connecting edge between the nodes of the two road sections. Construct an undirected graph structure of the urban road network G = (V, E, A), where V = {v1, v2, ..., v N} represents a set of N nodes, v i represents the ith node (i.e., the ith observation section); E = {e1, e2, …, e M} represents a set of M edges, i.e., the characteristic correlation between the observed road segments; A = [A ij ]∈R N×N Represents the adjacency matrix of the graph, A ij Represents the weight value of the edge between node i and node j in the graph.

[0011] S22 Design of adjacency matrix: The construction of adjacency matrix A includes static and dynamic parts. The static construction method based on distance can be expressed as

[0012]

[0013] where dist(v i ,v j ) represents node vi and v j The distance between them, σ represents the standard deviation of the distance, and k represents the threshold of the distance mapping to the Gaussian kernel. The adjacency matrix dynamically updated by adaptive graph structure learning can be expressed as

[0014]

[0015] Where E1,E2∈R N×K is a trainable parameter. The final adjacency matrix is ​​a combination of the static and dynamic parts, i.e. A = A o +ΔA. Through this combined approach, the adjacency matrix can not only reflect the static spatial relationship between nodes, but also be dynamically adjusted to capture the complex dynamic characteristics of the transportation network.

[0016] Step 3: Introduce the method of fine-tuning GPT-2 in combination with GCN:

[0017] S31 inputs traffic history data Preprocessing is performed, including missing value filling, outlier detection, and data standardization. First, for each missing value Linear interpolation through adjacent known values Then, the Z-score method is used to detect outliers. Then, for each data point calculate like >threshold, then The data are treated as outliers and replaced with the median. Finally, Min-Max normalization is used to scale the data to the [0,1] interval.

[0018] S32 will pre-process the traffic network data, including historical data And the constructed road network structure G = (V, E, A) is added to the GCN network to learn the spatiotemporal correlation of data. The specific update process of GCN is as follows: Among them, H (l) is the node feature matrix of the lth layer, initially is the normalized adjacency matrix, W (l) is the trainable weight matrix of the lth layer; σ is the activation function ReLU. In this way, GCN can capture the spatiotemporal correlation of traffic data and provide rich feature representation for subsequent GPT-2 fine-tuning.

[0019] S33 fine-tunes the prediction model based on GPT2:

[0020] Block processing: feature matrix H output by GCN (l)Rearrange and pad the sequence to ensure that the length is suitable for subsequent processing. Use the specified stride to split the padded data into multiple blocks, each block represents a subsequence of the input data. The new representation after conversion is H”∈R N×i×p , where p represents the length of each block and the number of blocks

[0021] Embedding and positional encoding: The input H' is converted to a high-dimensional vector h∈R through a linear layer N×i×F At the same time, in order to preserve the order of the sequence, the embedding vector h is combined with the position encoding pos obtained from the pre-trained GPT-2 model to obtain H = h + pos.

[0022] For the design of the prediction model, the present invention uses the Transformer layer of GPT-2 to process the features extracted by GCN. The Transformer layer consists of a multi-head attention mechanism and a feedforward neural network, which can effectively capture complex patterns in sequence data. The multi-head attention layer is used to process different representation subspaces in parallel to enhance the feature extraction capability of the model, while the feedforward layer performs further nonlinear transformations on the output of the attention layer. In order to retain the existing contextual knowledge of the pre-trained GPT-2 model, most of the parameters are frozen during model training to ensure that the model remains stable during fine-tuning. The remaining parameters will be updated based on traffic data to adapt to specific prediction tasks. In this way, the model can flexibly adapt to the specific needs of traffic prediction tasks while retaining the GPT-2 pre-training knowledge. Next, the implementation of the multi-head attention mechanism will be introduced in detail.

[0023] Multi-head attention mechanism: In order to enhance the model's ability to explore information in different representation subspaces, a parallel processing head is introduced to further extract the spatiotemporal relationship of traffic flow data. In the multi-head attention mechanism, the input vector H is first converted into a query Q, a key K, and a value V:

[0024] Q=HW Q ,K=HW K ,V=HW V

[0025] Among them, W Q , W K and W V is the weight matrix of the transformation. Each attention head independently calculates the dot product between the query and the key, scales it by a factor, and normalizes it using the softmax function to obtain the attention score where d k Represents the dimension of the key vector, which is used to scale the dot product result to prevent the gradient vanishing problem. The outputs of all heads are concatenated and then linearly transformed through another fully connected layer to generate the final output M of the multi-head attention.H =Concat(S1,…,S H )W'. Concat represents the connection operation, W'∈R f×f is the weight matrix of the output layer. Through this mechanism, multi-head attention not only enhances the model’s ability to identify relevant features of different dimensions, but also accelerates the calculation through parallel processing.

[0026] Residual connections and normalization layers: Residual connections and normalization layers integrate information from different processing stages and maintain the continuity and consistency of data through multiple levels. It helps prevent overfitting, enhances the model's generalization ability to new data, and accelerates model convergence by reducing internal covariate shift during training. Therefore, it is particularly effective for processing complex time series data that requires the model to understand long-term dependencies. Residual connections and normalization layers are strategically placed before and after the feed-forward layers. Before the feed-forward layer, this module first combines the outputs from the multi-head attention mechanism, the adapter module, and the residual connection through an addition operation, and then normalizes the combined data through layer normalization. The mathematical expression of layer normalization is Where X represents the input to be normalized, μ and σ are the mean and standard deviation calculated across feature dimensions, and γ and β are trainable parameters used to restore the original scale and offset of the normalized data.

[0027] Feedforward layer: The feedforward layer is inherited and frozen from the original GPT2 model, and is used to process data adjusted by the self-attention mechanism and the adapter module. This layer consists of two linear transformations and uses a GELU (Gaussian Error Linear Unit) nonlinear activation function between them. Specifically, the first linear transformation maps the input data to a higher-dimensional space, where the GELU activation function introduces the necessary nonlinearity to capture more complex data relationships. The second linear transformation maps the data back to the original dimension. This "expansion-activation-compression" mode enables the feedforward layer to effectively learn complex data representations.

[0028] Output linear layer: After the L-layer fine-tuned GPT2 Transformer network, the model maps the features to the required output dimension through the linear layer, that is, Where N represents the number of road sections, T F Indicates the length of the forecast period. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a structural diagram of the model proposed in the present invention.

[0030] Figure 2 and Figure 3The proposed method is compared with the latest iTransformer, PatchTST, Timesnet and OFA method with fine-tuning GPT2 on PEMS04 and PEMS08 datasets respectively. The final performance is measured by the mean absolute error (MAE), root mean square error (RMSE) and mean absolute percentage error (MAPE) at different proportions of training data.

[0031] Figure 4 The parameters (Params) of all the compared methods and the model proposed by the present invention are compared. DETAILED DESCRIPTION

[0032] The traffic prediction method of dynamic GCN and fine-tuning GPT2 proposed in the present invention adopts the Pytorch deep learning framework to implement the code and runs on a single Nvidia GeForce RTX 4090 GPU with 24GB of memory. In the training phase, the number of training Epochs is 50, the batch size is 24, the initial learning rate learning rate=0.0001, and the decay rate of the learning rate of each Epoch is configured to be 0.97. At the same time, some hyperparameters need to be configured: the input historical traffic time series length and the prediction time series length are both 12, the time frame is 12 frames, and the sampling frequency is once every 5 minutes, that is, the historical traffic data of the previous hour is used to predict the traffic data within the next hour.

[0033] In order to accurately evaluate the present invention, the experiment used two sets of real road traffic data sets PEMS04 and PEMS08, which came from two regions in California respectively. The data are sampled at a sampling frequency of one sample every 5 minutes, and PEMS04 and PE-MS08 calculate the evaluated traffic flow of the road within the sampling frequency as data. The data set is divided in time order with a ratio of 6:2:2 for training set: validation set: test set. In order to verify the traffic flow prediction effect of the present invention in data-deficient scenarios, 5%, 10%, and 30% of the training set data are used for training in the experiment.

[0034] The experimental accuracy of the PEMS04 and PEMS08 datasets is as follows Figure 2 and Figure 3As shown, the method of the present invention has improved MAE, RMSE and MAPE compared with the latest iTransformer method based on Transformer structure. The feasibility of migrating the LLM model to the field of traffic flow prediction has been verified, indicating that the rich contextual knowledge possessed by the LLM model can provide guiding experience for traffic flow time series prediction. Compared with OFA, the latest method for migrating the LLM large model to the field of time series prediction, our method takes into account the unique spatiotemporal problems faced by the field of traffic flow prediction. The introduced dynamic GCN helps fine-tune the GPT2 model to focus on the spatial relationship of the traffic network diagram, and obtains the best prediction accuracy.

[0035] Figure 4 By comparing the parameters of all the comparison methods and the model proposed in this invention, it can be found that our method achieves the best performance without increasing additional computing consumption.

Claims

1. A traffic prediction method combining dynamic GCN and fine-tuned GPT2, characterized in that: The implementation steps of this method are as follows: Step 1: Construct a traffic network map; S11 defines the traffic road graph structure: using road segments as nodes and the upstream and downstream relationships between road segments as edges to construct a road network graph, thereby representing the spatial structure of urban roads; That is, if there is traffic between two road segments, it is defined that there is a connecting edge between the two road segment nodes; S12 Design of adjacency matrix: The construction of adjacency matrix includes static and dynamic parts; the static construction method based on distance is expressed as: where dist(v i ,v j ) represents node v i and v j The distance between them, σ represents the standard deviation of the distance, and k represents the threshold of the distance mapping to the Gaussian kernel; Step 2: Fine-tune GPT-2 with GCN; S21 inputs traffic history data Perform preprocessing, including missing value filling, outlier detection, and data standardization; S22 will pre-process the traffic network data, including historical data And the constructed road network structure G = (V, E, A) is added to the GCN network to learn the spatiotemporal correlation; S23 fine-tunes the prediction model based on GPT2: Block processing: feature matrix H output by GCN (l) Perform rearrangement and padding operations to ensure that the sequence length is suitable for subsequent processing; split the padded data into multiple blocks using the specified stride, each block represents a subsequence of the input data; the new representation after conversion is H”∈R N×i×p , where p represents the length of each block and the number of blocks Embedding and positional encoding: The input H' is converted to a high-dimensional vector h∈R through a linear layer N×i×F ; At the same time, in order to preserve the order of the sequence, the embedding vector h is combined with the position encoding pos obtained from the pre-trained GPT-2 model to obtain H = h + pos; For the design of the prediction model, the Transformer layer of GPT-2 is used to process the features extracted by GCN; The Transformer layer consists of a multi-head attention mechanism and a feedforward neural network, which can effectively capture complex patterns in sequence data; the multi-head attention layer is used to process different representation subspaces in parallel to enhance the model's feature extraction capability, while the feedforward layer performs further nonlinear transformations on the output of the attention layer; in order to retain the existing contextual knowledge of the pre-trained GPT-2 model, most of the parameters are frozen during model training to ensure that the model remains stable during fine-tuning; The remaining parameters will be updated based on traffic data to suit the specific prediction task.

2. A traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that: Model the traffic data of different road sections and different time periods in the city and set represents the traffic data of road section i at time t, where F represents the characteristic dimension of traffic data, and the traffic data of all road sections at time t is expressed as Where N represents the total number of road sections in the entire urban road network, the traffic data sequence at time (1, 2, …, τ) is represented by X 1:τ =[X1,X2,…,X τ ] T ∈R N×F×τ Expression: Given a road network graph G and time t before T P Traffic history data for time frames Input into the model F that combines GCN and GPT2 to predict T after time t F Traffic data by time frame 3. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that: In S11, an undirected graph structure of the urban road network G = (V, E, A) is constructed, where V = {v1, v2, …, v N } represents a set of N nodes, v i represents the ith node, that is, the ith observation section; E = {e1, e2, …, e M } represents a set of M edges, i.e., the characteristic correlation between the observed road segments; A = [A ij ]∈R N×N Represents the adjacency matrix of the graph, A ij Represents the weight value of the edge between node i and node j in the graph.

4. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that: In S12, the adjacency matrix dynamically updated by adaptive graph structure learning is expressed as: Where E1,E2∈R N×K is a trainable parameter; the final adjacency matrix is ​​a combination of the static and dynamic parts, that is, A = A o +ΔA.

5. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that: In S21, first, for each missing value Linear interpolation through adjacent known values Then the Z-score method is used to detect outliers; then, for each data point calculate like Threshold value, then The data are treated as outliers and replaced with the median; finally, Min-Max normalization is used to scale the data to the [0,1] interval.

6. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that: In S22, the specific update process of GCN is: Among them, H (l) is the node feature matrix of the lth layer, initially is the normalized adjacency matrix, W (l) is the trainable weight matrix of the lth layer; σ is the activation function ReLU; in this way, GCN can capture the spatiotemporal correlation of traffic data and provide rich feature representation for subsequent GPT-2 fine-tuning.

7. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that: The implementation process of the multi-head attention mechanism is as follows: Multi-head attention mechanism: To enhance the model’s ability to explore information in different representation subspaces, a parallel processing head is introduced to extract the spatiotemporal relationship of traffic flow data. In the multi-head attention mechanism, the input vector H is first converted into a query Q, a key K, and a value V: Q=HW Q ,K=HW K ,V=HW V Among them, W Q , W K and W V is the weight matrix of the transformation; each attention head independently calculates the dot product between the query and the key, scales it by a factor, and normalizes it using the softmax function to obtain the attention score where d k Represents the dimension of the key vector, which is used to scale the dot product result to prevent the gradient vanishing problem; the outputs of all heads are concatenated and then linearly transformed through another fully connected layer to generate the final output M of the multi-head attention. H =Concat(S1,…,S H )W'; where Concat represents the connection operation, W'∈R f×f is the weight matrix of the output layer; through this mechanism, multi-head attention not only enhances the model’s ability to identify relevant features of different dimensions, but also accelerates calculations through parallel processing; Residual connection and normalization layer: The residual connection and normalization layer integrates information from different processing stages and maintains the continuity and consistency of data through multiple layers; before the feed-forward layer, the outputs from the multi-head attention mechanism, the adapter module, and the residual connection through the addition operation are first combined, and then the combined data is normalized through layer normalization; the mathematical expression of layer normalization is Where X represents the input to be normalized, μ and σ are the mean and standard deviation calculated across feature dimensions, and γ and β are trainable parameters used to restore the original scale and offset of the normalized data; Feedforward layer: The feedforward layer is inherited and frozen from the original GPT2 model, and is used to process data adjusted by the self-attention mechanism and the adapter module; this layer consists of two linear transformations, and a Gaussian error linear unit GELU nonlinear activation function is used between them; the first linear transformation maps the input data to a higher-dimensional space, where the GELU activation function introduces the necessary nonlinearity to capture more complex data relationships; the second linear transformation maps the data back to the original dimension; Output linear layer: After the L-layer fine-tuned GPT2 Transformer network, the model maps the features to the required output dimension through the linear layer, that is, Where N represents the number of road sections, T F Indicates the length of the forecast period.

Images