A traffic prediction method combining dynamic GCN and fine-tuned GPT2

Abstract

The application discloses a traffic prediction method combining dynamic GCN and fine-tuning GPT2, defines a traffic road graph structure, takes a road section as a node, and takes a traffic flow upstream and downstream relationship between road sections as an edge to construct a road network graph, and further represents a spatial structure of a city road, namely, if there is a traffic flow between two road sections, a connection edge between two road section nodes is defined, and an adjacency matrix is designed; preprocessing is performed on input traffic historical data, including missing value filling, abnormal value detection and data standardization; the preprocessed traffic road network data, including historical data and the constructed road network graph structure, is input into a GCN network to learn a space-time correlation; and a prediction model based on GPT2 is fine-tuned: the GPT-2 is fine-tuned in combination with the GCN; and the dynamic GCN introduced in the method helps the fine-tuned GPT2 model to focus on the spatial relationship of the traffic network graph, and the best prediction precision is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pattern recognition and deep learning, and large language models, time series prediction, in particular to a method for fine-tuning a large prediction model combined with a dynamic graph neural network and adapting it to traffic flow prediction. BACKGROUND

[0002] In recent years, with the advancement of smart city construction, the expansion and informatization level of urban roads have been significantly improved. Although the urban road infrastructure has developed rapidly, the growing urban population and the number of motor vehicles, as well as the resulting traffic demand, far exceed the construction speed of the urban transportation system. Many cities are therefore facing problems such as frequent traffic accidents, traffic congestion and traffic pollution, which seriously affect the quality of life of residents and the traffic efficiency of urban vehicles, and become an obstacle to the sustainable development of modern cities. In particular, in super-large cities with a population of over ten million, the transportation system is an open complex system integrating mass demand management, traffic flow space-time distribution adjustment, comprehensive behavior prediction, and energy environment optimization, and faces enormous challenges. In order to solve these problems, many researchers have proposed the construction of intelligent transportation systems (ITS), which use sensors, computers, artificial intelligence and other technologies to improve the utilization rate of urban road resources. The premise of building an intelligent transportation system is to effectively use traffic big data, and traffic prediction is the most important and basic method of effectively using traffic big data. Traffic prediction trains network models through historical traffic data to predict future traffic data, and the prediction results can provide important data support for the optimization and scheduling of the transportation system, the implementation of traffic management measures, and the early warning of traffic congestion, and also provide people with related help such as travel planning routes. Therefore, traffic prediction research is one of the hotspots of intelligent transportation research.

[0003] In existing traffic prediction research, methods based on Graph Convolutional Network (GCN), Long Short-Term Memory (LSTM), and Transformer are widely used. These methods usually rely on a large amount of reliable data to complete the training of the model. For example, GCN captures the spatial features of the traffic network to make predictions, LSTM is good at processing time series data to predict traffic flow, and the Transformer model exhibits excellent performance in traffic prediction through its powerful sequence modeling capability. However, such data-driven methods face challenges in the context of smart city expansion, as many emerging transportation systems lack a large amount of comprehensive historical data. This lack of data makes it difficult for traditional traffic prediction methods to meet the needs of smart city expansion for traffic prediction. Therefore, how to effectively predict traffic in the case of insufficient data has become an important direction of current research.

[0004] Existing traffic prediction methods often ignore the possibility of using a well-trained Large Language Model (LLM) to solve this problem when dealing with data-deficient smart city expansion scenarios. LLM-based methods have achieved remarkable results in multiple fields such as natural language processing, demonstrating their strong generalization ability and contextual understanding. However, directly training a specific large model in a new scenario often requires a large dataset and enormous computing resources, which is not feasible in many practical applications. Recent research has shown that adapting an LLM model trained in the language field to the time series data field through fine-tuning can enable the large model to achieve excellent results in new scenarios. This method takes advantage of the strong prior knowledge of context association already present in the LLM model, allowing it to better extract and understand the features of time series data. Therefore, for data-deficient traffic flow prediction scenarios, the invention proposes an innovative method: adapting a well-trained LLM model to the traffic prediction task through fine-tuning. In this way, the model can still effectively predict traffic flow in the absence of a large amount of historical data, improving the accuracy and reliability of the prediction. This method not only reduces the dependence on large-scale data and computing resources, but also provides a new solution for traffic management in smart cities. SUMMARY

[0005] The invention focuses on the particularity of traffic flow prediction and combines dynamic graph convolutional networks to design a specific fine-tuning strategy, proposing a traffic prediction method that combines dynamic GCN and fine-tuned GPT-2. This method aims to solve the difficulty of traffic prediction due to data deficiency during the expansion of smart cities.

[0006] The technical solution steps of the present application are as follows:

[0007] Step 1: Define the traffic prediction problem:

[0008] Model the traffic data (traffic flow data or traffic speed data, etc.) of different road segments in the city at different time periods, and set to represent the traffic data of road segment i at time t, where F represents the feature dimension of the traffic data, then the traffic data of all road segments at time t can be represented as where N represents the total number of road segments in the entire city road network, then the traffic data sequence at time (1, 2, …, τ) can be represented as 1:τ = [X1, X2, …, X τ ] T ∈R N×F×τ expression. Therefore, given the road network graph G and the traffic history data of T P time frames before time t input into the model trained for learning , the traffic data of T F time frames after time t

[0009] Step 2: Construct the traffic road network graph:

[0010] S21 defines the traffic road graph structure: the present application constructs the road network graph by taking road segments as nodes and the upstream and downstream relationships of traffic flow between road segments as edges, thereby representing the spatial structure of urban roads. That is, if there is traffic flow between two road segments, a connection edge between the two road segment nodes is defined. The undirected graph structure of the city road network G = (V, E, A) is constructed, where V = {v1, v2, …, v N} represents the set of N nodes, v i represents the i-th node (i.e. the i-th observed road segment); E = {e1, e2, …, e M} represents the set of M edges, i.e. the feature association between observed road segments; A = [A ij ] ∈ R N×N represents the adjacency matrix of the graph, A ij represents the weight value of the edge between node i and node j in the graph.

[0011] S22 designs the adjacency matrix: the construction of the adjacency matrix A includes static and dynamic parts. The static construction method based on distance can be represented as

[0012]

[0013] where dist(v i ,v j ) represents the distance between nodes vi and v j the distance between two nodes, σ represents the distance standard deviation, and k represents the threshold value of the distance mapping to the Gaussian kernel. The dynamically updated adjacency matrix by adaptive graph structure learning can be represented as

[0014]

[0015] where E1, E2∈R N×K is the trainable parameter. The final adjacency matrix is the combination of the static and dynamic parts, i.e., A = A o + ΔA. Through this combination method, the adjacency matrix can not only reflect the static spatial relationship between nodes, but also dynamically adjust to capture the complex dynamic characteristics of the traffic network.

[0016] Step 3: Introduce the method of combining GCN to fine-tune GPT-2:

[0017] S31 preprocess the input traffic historical data , including missing value filling, outlier detection, and data standardization. First, for each missing value , linear interpolation is performed through adjacent known values Then, the Z-score method is used to detect outliers. Then, for each data point , calculate If > threshold, then is considered an outlier and is replaced with the median. Finally, use Min-Max standardization to scale the data to the [0, 1] interval.

[0018] S32 put the preprocessed traffic road network data, including historical data and the constructed road network graph structure G = (V, E, A) into the GCN network to learn the spatio-temporal correlation of the data. The specific update process of GCN is where H (l) is the node feature matrix of the l-th layer, initially is the normalized adjacency matrix, W (l) is the trainable weight matrix of the l-th layer; σ is the activation function ReLU. In this way, GCN can capture the spatio-temporal correlation of traffic data and provide rich feature representation for subsequent GPT-2 fine-tuning.

[0019] S33 fine-tune the GPT2-based prediction model:

[0020] Block processing: process the feature matrix H (l)Reordering and padding operations are performed to ensure the sequence length fits subsequent processing. The padded data is split into multiple chunks using a specified stride, each representing a sub-sequence of the input data. The new representation after transformation is H" ∈ R N×i×p where p denotes the length of each chunk, and the number of chunks

[0021] Embedding and position encoding: The input H" is transformed into a high-dimensional vector h ∈ R N×i×F by a linear layer. Meanwhile, to preserve the order of the sequence, the embedding vector h is combined with the position encoding pos obtained from a pre-trained GPT-2 model, resulting in H = h + pos.

[0022] For the design of the prediction model, the invention uses the Transformer layer of GPT-2 to process the features extracted by GCN. The Transformer layer consists of multi-head attention mechanism and feed-forward neural network, which can effectively capture complex patterns in sequence data. The multi-head attention layer is used to process different representation subspaces in parallel, enhancing the model's feature extraction capability, while the feed-forward layer further nonlinearly transforms the output of the attention layer. To preserve the existing context knowledge of the pre-trained GPT-2 model, during model training, most parameters are frozen to ensure the stability of the model during fine-tuning. The remaining parameters will be updated according to the traffic data to adapt to specific prediction tasks. In this way, the model can preserve the pre-training knowledge of GPT-2 while flexibly adapting to the specific needs of traffic prediction tasks. Next, the implementation of the multi-head attention mechanism will be detailed.

[0023] Multi-head attention mechanism: To enhance the model's ability to explore information in different representation subspaces, parallel processing heads are introduced to further extract the spatio-temporal relationships of traffic flow data. In the multi-head attention mechanism, the input vector H is first transformed into query Q, key K, and value V:

[0024] Q = HW Q , K = HW K , V = HW V

[0025] where W Q , W K , and W V are the transformation weight matrices. Each attention head independently calculates the dot product between the query and the key, and after scaling by a factor, the softmax function is used for normalization to obtain the attention score where d k represents the dimension of the key vector, used to scale the dot product result to prevent gradient vanishing problems. The outputs of all heads are concatenated and then linearly transformed by another fully connected layer to generate the final output M of multi-head attentionH = Concat(S1,...,S H )W'. Where Concat denotes the concatenation operation, W' ∈ R f×f is the weight matrix of the output layer. Through this mechanism, multi-head attention not only enhances the model's ability to identify different dimensionally dependent features, but also speeds up the calculation through parallel processing.

[0026] Residual connection and normalization layer: The residual connection and normalization layer integrates information from different processing stages and maintains the continuity and consistency of data through multiple levels. It helps to prevent overfitting, enhances the model's generalization ability to new data, and speeds up model convergence by reducing internal covariate shift during training. Therefore, it is particularly effective for processing complex time series data that requires the model to understand long-term dependencies. The residual connection and normalization layer is strategically placed before and after the feedforward layer. Before the feedforward layer, this module first combines the outputs from the multi-head attention mechanism, the adapter module, and the residual connection through an additive operation, and then normalizes the combined data through layer normalization. The mathematical expression of layer normalization is where X represents the input that needs to be normalized, μ and σ are the mean and standard deviation calculated across the feature dimension, γ and β are trainable parameters used to restore the original scale and offset of the normalized data.

[0027] Feedforward layer: The feedforward layer inherits and freezes from the original GPT2 model to process data adjusted by the self-attention mechanism and the adapter module. This layer consists of two linear transformations and uses the GELU (Gaussian Error Linear Unit) nonlinear activation function between them. Specifically, the first linear transformation maps the input data to a higher dimensional space, where the GELU activation function introduces the necessary nonlinearity to capture more complex data relationships. The second linear transformation then maps the data back to the original dimension. This "expansion-activation-compression" pattern enables the feedforward layer to effectively learn complex data representations.

[0028] Output linear layer: After the L-layer fine-tuning of the GPT2 Transformer network, the model maps the features to the required output dimension through a linear layer, i.e. where N represents the number of road segments, T F represents the predicted time length. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The model structure diagram proposed by the present application.

[0030] Figure 2 and Figure 3The final performance relies on the mean absolute error (MAE), root mean square error (RMSE) and mean absolute percentage error (MAPE) of different training data proportions.

[0031] Figure 4 The parameter quantity (Params) of all comparative methods and the proposed model is compared. DETAILED DESCRIPTION

[0032] The traffic prediction method of the proposed dynamic GCN and fine-tuned GPT2 is implemented using the Pytorch deep learning framework. The code is run on a single Nvidia GeForce RTX 4090 GPU with 24 GB of memory. In the training phase, the number of training epochs is 50, the batch size is 24, the initial learning rate is 0.0001, and the decay rate of the learning rate of each epoch is configured to be 0.97. Some hyperparameters also need to be configured: the input historical traffic time series length and the prediction time series length are both 12, the time frame is 12 frames, and the sampling frequency is 5 minutes per sample, i.e., the future one hour of traffic data is predicted based on the previous one hour of historical traffic data.

[0033] In order to accurately evaluate the present application, two groups of real road traffic data sets PEMS04 and PEMS08 are used in the experiment, which come from two regions in California. The data is sampled at a sampling frequency of 5 minutes per sample, and the PEMS04 and PEMS08 calculate the evaluation traffic flow of the road within the sampling frequency as data. The data set is divided into training set: validation set: test set in the ratio of 6:2:2 by time sequence. In order to verify the traffic flow prediction effect of the present application in the data insufficient scene, the training set 5%, 10%, 30% data is used for training in the experiment.

[0034] The experimental accuracy of the PEMS04 data set and the PEMS08 data set is as follows: Figure 2 and Figure 3As shown, the method of the application has improved MAE, RMSE and MAPE compared with the latest iTransformer method based on the Transformer structure. The feasibility of migrating the LLM model to the traffic flow prediction field is verified, which shows that the rich context knowledge possessed by the LLM model can provide guidance experience for traffic flow time series prediction. Compared with the latest OFA method of migrating the LLM large model to the time series prediction field, the method considers the unique space-time problems faced by the traffic flow prediction field, and the introduced dynamic GCN helps the GPT2 model to focus on the spatial relationship of the traffic network graph, and the best prediction accuracy is obtained.

[0035] Figure 4 For all the comparative methods and the parameters of the model proposed by the application, it can be found that the best performance is obtained without increasing additional calculation consumption.

Claims

1. A traffic prediction method combining dynamic GCN and fine-tuned GPT2, characterized in that, The implementation steps of this method are as follows: Step 1: Construct a traffic network map; S11 defines the traffic road map structure: using road segments as nodes and the upstream and downstream traffic flow relationships between road segments as edges to construct a road network map, thereby representing the spatial structure of urban roads; That is, if there is traffic flow between two road segments, then a connecting edge is defined between the nodes of the two road segments; S12 Adjacency Matrix Design: The construction of the adjacency matrix includes both static and dynamic parts; the distance-based static construction method is expressed as follows: Where dist(v i ,v j ) represents node v i and v j The distance between them, where σ represents the standard deviation of the distance and k represents the threshold for mapping the distance to the Gaussian kernel; Step 2: Fine-tune GPT-2 using GCN; S21 inputs historical traffic data Preprocessing is performed, including missing value imputation, outlier detection, and data standardization; S22 will process the pre-processed traffic network data, including historical data. The constructed road network structure G=(V,E,A) is input into the GCN network to learn spatiotemporal correlations; S23 fine-tunes the GPT2-based prediction model: Block processing: The feature matrix H output by GCN... (l) Perform rearrangement and padding operations to ensure the sequence length is suitable for subsequent processing; divide the padded data into multiple blocks using a specified stride, each block representing a subsequence of the input data; the transformed new representation is H”∈R N×i×p Where p represents the length of each block and the number of blocks. Embedding and positional encoding: Input J” is transformed into a high-dimensional vector h∈R through a linear layer. N×i×F Meanwhile, in order to preserve the sequence order, the embedding vector h is combined with the position encoding pos obtained from the pre-trained GPT-2 model to obtain H = h + pos; For the design of the prediction model, the Transformer layer of GPT-2 is used to process the features extracted by GCN; The Transformer layer consists of a multi-head attention mechanism and a feedforward neural network, which can effectively capture complex patterns in sequential data. The multi-head attention layer is used to process different representation subspaces in parallel, enhancing the model's feature extraction capabilities, while the feedforward layer performs further nonlinear transformations on the output of the attention layer. In order to preserve the existing contextual knowledge of the pre-trained GPT-2 model, most parameters are frozen during model training to ensure that the model remains stable during fine-tuning. The remaining parameters will be updated based on traffic data to adapt to specific forecasting tasks.

2. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that, Modeling is performed on traffic data from different road sections and time periods in the city, and settings are defined. Let F represent the traffic data for road segment i at time t, where F represents the feature dimension of the traffic data. The traffic data for all road segments at time t are represented as follows: Where N represents the total number of road segments in the entire city's road network, then the traffic data sequence at time (1,2,…,τ) is represented by X. 1:τ =[X1,X2,…,X τ ] T ∈R N×F×τ Expression; given the road network map G and time T before time t. P Traffic history data in time frames Input into a model combining GCN and GPT2 In the middle, predicting T after time t F Traffic data in time frames 3. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that, In S11, an undirected graph structure G = (V, E, A) for the urban road network is constructed, where V = {v1, v2, ..., v}. N } represents a set of N nodes, v i This represents the i-th node, i.e., the i-th observed road segment; E = {e1, e2, ..., e...} M } represents the set of M edges, i.e., the characteristic correlations between observed road segments; A = [A ij ]∈R N×N Let A represent the adjacency matrix of the graph. ij This represents the weight value of the edge between node i and node j in the graph.

4. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that, In S12, the adjacency matrix learned through the adaptive graph structure and dynamically updated is represented as: Where E1, E2 ∈ R N×K These are trainable parameters; the final adjacency matrix is ​​a combination of the static and dynamic components, i.e., A = A o +ΔA.

5. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that, In S21, firstly, for each missing value... Linear interpolation is performed using adjacent known values. Then, the Z-score method is used to detect outliers; then, for each data point calculate like The threshold will then These are considered outliers and replaced with the median; finally, the data is scaled to the [0,1] range using Min-Max normalization.

6. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that, In S22, the specific update process of GCN is as follows: Among them, H (l) It is the node feature matrix of the l-th layer, initially set to... It is a normalized adjacency matrix. W (l) σ is the trainable weight matrix of the l-th layer; σ is the activation function ReLU; in this way, GCN can capture the spatiotemporal correlation of traffic data, providing rich feature representations for subsequent GPT-2 fine-tuning.

7. The traffic prediction method combining dynamic GCN and fine-tuned GPT2 according to claim 1, characterized in that, The implementation process of the multi-head attention mechanism is as follows: Multi-head attention mechanism: To enhance the model's ability to explore information in different representation subspaces, parallel processing heads are introduced to extract the spatiotemporal relationships of traffic flow data. In the multi-head attention mechanism, the input vector H is first converted into a query Q, a key K, and a value V. Q=HW Q ,K=HW K ,V=HW V Among them, W Q W K and W K This is the weight matrix of the transformation; each attention head independently calculates the dot product between the query and the key, scales it by a factor, and then normalizes it using the softmax function to obtain the attention score. Where d k The dimension of the key vector is used to scale the dot product result to prevent the vanishing gradient problem; the outputs of all heads are concatenated and then linearly transformed through another fully connected layer to generate the final output M of the multi-head attention. H =Concat(S1,…,S H )W'; where Concat represents the join operation, W'∈R f×f It is the weight matrix of the output layer; through this mechanism, multi-head attention not only enhances the model's ability to identify relevant features in different dimensions, but also accelerates computation through parallel processing; Residual Connectivity and Normalization Layer: The residual connectivity and normalization layer integrates information from different processing stages and maintains data continuity and consistency through multiple layers. Before the feedforward layer, the outputs from the multi-head attention mechanism, the adapter module, and the residual connectivity through addition are first combined, and then the combined data is normalized through layer normalization. The mathematical expression for layer normalization is: Where X represents the input that needs to be normalized, μ and σ are the mean and standard deviation calculated across the feature dimensions, and γ and β are trainable parameters used to recover the original scale and offset of the normalized data; Feedforward layer: The feedforward layer inherits from and is frozen from the original GPT2 model to process data that has been tuned by the self-attention mechanism and adapter module; this layer consists of two linear transformations with a Gaussian error linear unit (GELU) nonlinear activation function between them; the first linear transformation maps the input data to a higher-dimensional space, where the GELU activation function introduces the necessary nonlinearity to capture more complex data relationships; the second linear transformation maps the data back to the original dimension; Output Linear Layer: After L layers of fine-tuning of the GPT2 Transformer network, the model maps features to the desired output dimension through linear layers, i.e. Where N represents the number of road segments, T F Indicates the predicted time period.

Images