Large-scale highway network traffic flow map fine-tuning large model prediction method

Abstract

The application discloses a large-scale highway network traffic flow graph fine-tuning large model prediction method, comprising the following steps: S1: constructing a heterogeneous graph of highway network traffic flow, the nodes in the heterogeneous graph being toll station nodes and intersection nodes, and the edges in the heterogeneous graph being the traffic flow between nodes; S2: using a graph convolution network to aggregate neighbor information of the nodes, capturing the spatial characteristics of the highway network traffic flow, and calculating the spatial weight of the edges between the nodes; S3: adding time information in the heterogeneous graph, using a time convolution network combined with Chebyshev filtering and dilated convolution to capture the time characteristics of the highway network traffic flow, and calculating the time weight of the edges between the nodes; S4: splicing the spatial characteristics and the time characteristics to generate a unified sentence vector; and S5: inputting the unified sentence vector into a large language model, and performing traffic flow prediction through instruction fine-tuning and a partial attention layer freezing strategy.

Description

Technical Field

[0001] This invention relates to the field of traffic flow prediction, specifically to a large-scale model prediction method for fine-tuning traffic flow maps of highway networks. Background Technology

[0002] Highway traffic flow prediction primarily relies on traditional traffic flow detection devices and prediction models based on historical data. Traditional traffic flow monitoring devices include geomagnetic sensors, radar speedometers, laser speed measuring devices, cameras, and floating car data. Geomagnetic sensors calculate vehicle flow and speed by sensing the interference of vehicles' metal on the magnetic field; radar speedometers and laser speed measuring devices measure vehicle speed by emitting radio frequency signals or laser beams and estimate flow based on vehicle travel time and distance; cameras mainly collect traffic flow data in real time through image monitoring and use image processing technology to extract information such as traffic density and vehicle speed; floating car data (FCD) collects traffic flow information by monitoring the GPS location of vehicles in real time, and then performs flow prediction.

[0003] While these traditional traffic flow monitoring devices and methods can provide traffic flow information to some extent, they still face several problems. First, these methods often rely on the deployment of physical sensors, and in special locations such as highway tunnel complexes and tunnel entrances / exits, traditional devices face data scarcity. Second, most traditional traffic flow prediction methods are based on statistical analysis of historical traffic data, employing methods such as time series analysis and regression models. These methods typically rely on relatively simple statistical models, ignoring the complex spatiotemporal dynamics of traffic flow and failing to fully consider the influence of complex environmental factors. Furthermore, existing traffic flow prediction methods mainly rely on macroscopic traffic flow parameters (such as flow rate, speed, and occupancy rate) for modeling, but these macroscopic parameters have significant limitations in practical applications.

[0004] In practical predictive applications, most existing methods rely on local data for forecasting, failing to analyze large-scale data. This significantly limits the models, making it difficult to effectively capture the spatiotemporal dependencies between different highway segments within a road network. Furthermore, these methods depend on statistical models and traditional machine learning algorithms, which cannot handle complex traffic flow patterns over long periods, especially on highways where frequent changes occur. Traditional models struggle to adapt to rapidly shifting traffic flow patterns. Therefore, while existing methods perform well in certain small-scale, static environments, they cannot respond in real-time to rapid changes in traffic flow when dealing with large-scale, complex traffic flow prediction tasks, thus limiting their application.

[0005] Therefore, although traditional traffic flow prediction methods are still effective for small-scale road network data, they have significant shortcomings in large-scale road network applications. In complex and large-scale environments, how to effectively handle spatiotemporal dependencies, how to obtain complete and real-time traffic flow data, and how to improve the generalization ability of the model are the core problems that current methods need to solve. This also provides a strong demand for the proposal and development of new traffic flow prediction methods, especially advanced models that can combine large-scale traffic data and have strong adaptability and real-time performance.

[0006] In recent years, with the rapid development of deep learning technology, more and more research has attempted to apply deep learning to traffic flow prediction. These methods mainly construct complex deep neural network models, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and graph neural networks (GNNs), to model and predict time-series data. In particular, models like LSTMs and GNNs can effectively capture complex dependencies in time-series data, exhibiting strong modeling and generalization capabilities. However, these methods still have certain limitations. While LSTMs can handle long-term dependencies in time-series data, their performance is not ideal when dealing with spatially dependent data. Although graph neural networks (GNNs) can handle spatially structured data to some extent, they are typically limited in extracting spatial information from the data and cannot effectively incorporate complex environmental factors for modeling. Most existing deep learning models rely on large-scale historical data for training; when data is scarce or unavailable, the performance of these models degrades. Summary of the Invention

[0007] The purpose of this invention is to provide a large-scale prediction method for fine-tuning traffic flow maps of highway networks, in order to solve the problem that current deep learning models need to rely on large-scale historical data for training to maintain performance.

[0008] To address the aforementioned technical problems, this invention provides a large-scale highway network traffic flow map fine-tuning prediction method using a large model, comprising the following steps:

[0009] S1: Construct a heterogeneous graph of traffic flow in the highway network. The nodes in the heterogeneous graph are toll station nodes and intersection nodes, and the edges in the heterogeneous graph are the traffic flow between each node.

[0010] S2: Utilize Graph Convolutional Network (GCN) to aggregate the neighbor information of nodes, capture the spatial characteristics of traffic flow in the highway network, and calculate the spatial weights of edges between nodes;

[0011] S3: Add time information to the heterogeneous graph, use the Temporal Convolutional Network (TCN) combined with Chebyshev filtering and dilated convolution to capture the temporal characteristics of traffic flow in the highway network, and calculate the temporal weights of edges between nodes.

[0012] S4: Concatenate the spatial and temporal features to generate a unified sentence vector;

[0013] S5: Input the unified sentence vector into a large language model LLM, and perform traffic flow prediction through instruction fine-tuning and partial attention layer freezing strategies.

[0014] Furthermore, the heterogeneous graph is represented as:

[0015] G = (V, E)

[0016] Wherein, V is the node set, including the toll station node U and the intersection node I; E is the edge set, including the traffic flow between the toll station node and the intersection node, the traffic flow between toll station nodes, and the traffic flow between intersection nodes.

[0017] Furthermore, the spatial features output by each graph convolutional layer of the graph convolutional network GCN are represented as follows:

[0018]

[0019] Where σ is the activation function. Let h be the set of neighboring nodes of node u, and Wuv be the spatial weight of the edge between node u and node v; (l) This is the input for the layer l graph convolution.

[0020] Furthermore, the temporal features of the output of each convolutional layer of the Temporal Convolutional Network (TCN) are represented as follows:

[0021]

[0022] in: Let be the time weight of the edge between node u and node v. The vector is represented by dilated convolution. For the |Tth t |-d l Traffic flow prediction scores at time steps t, where yt is the Chebyshev filter representation vector;

[0023] The dilated convolution representation vector Represented as:

[0024]

[0025] Where: x t Let d represent the traffic flow characteristics at time step t. lThe expansion rate of the l-th layer; is the kernel function; Tt is the vectorized representation of the traffic flow of the highway network at time t. For the |Tt|-d l The input at t time steps represents the time steps skipped by the convolution kernel during the dilated convolution;

[0026] The Temporal Convolutional Network (TCN) uses a Type I Chebyshev filter for tensor filtering. The transfer function of the Type I Chebyshev filter is expressed as:

[0027]

[0028] Where: ε is the ripple coefficient of the filter, used to control the gain fluctuation in the passband; Let ω be the nth order Chebyshev polynomial. c Here, is the cutoff frequency of the filter, and s is the input parameter of the Chebyshev filter;

[0029] By passing parameters, H(s) is the graph convolution calculation result H(G). i,j );

[0030] For C i The signal sequence input to ρ layers of temporal convolution on each channel The weights of the convolutional layer are w i The output after Chebyshev filtering is:

[0031]

[0032] G i,j This is graph convolution;

[0033] Traffic flow prediction score Represented as:

[0034]

[0035] Where: α is the traffic flow decay factor in the time dimension; Let time step t and |Tt|-d l The time difference between t; η is the parameter controlling time decay; χ is the time weight threshold.

[0036] Furthermore, the unified sentence vector uf is represented as:

[0037] uf = [gf; tf]

[0038] Where gf is the spatial feature h (l+1) The vector representation of tf; where tf is the time feature The vector representation of ; [;] represents the vector concatenation operation.

[0039] Furthermore, the method also includes embedding the tokens through pointwise convolution; for spatial vectors, adaptive vector embedding is employed.

[0040]

[0041] Where σ represents the activation function, It is a learnable parameter, Par S It is a spatial vector embedding;

[0042] For the time vector, a linear layer is used to encode the input data into the number of days in January and the number of days in a week. These two embeddings are then made independent and summed to obtain the time embedding representation.

[0043] Embedded number of days in January for:

[0044]

[0045] Embedded number of days in a week

[0046]

[0047] in, and To generate location information by encoding the absolute location of each piece of provincial traffic data at monthly and weekly resolutions; and It is a learnable time embedding used for the number of days in a week and the number of days in a month.

[0048] Furthermore, the method also includes using an adaptive preference gate bidirectional gated recurrent unit to process the input uf sequence u1, u2, ..., u from front to back and from back to front, respectively. t ,…,u T After processing, the forward sequence output is:

[0049]

[0050] The reverse sequence output is:

[0051]

[0052] The hidden state is obtained by concatenating the hidden states of the forward and reverse sequence outputs. for:

[0053]

[0054] Then, the degree of attention given to different features is dynamically adjusted, and an additional gating mechanism is introduced. At time t, the update process of the preference gate is as follows:

[0055] g t =σ(W g ·u t +U g ·h t-1 )

[0056] Among them, g t For the current moment, W g and U g For learnable weights, x t This is the input for the current moment.

[0057] Furthermore, the large language model LLM adopts an encoder and decoder architecture;

[0058] The encoder converts the input sequence X into a contextual representation through instruction fine-tuning:

[0059] Henc = Encoder(X,θ) enc )

[0060] Among them, H enc For the encoded context-hidden representation, θ enc For encoder parameters;

[0061] X = Concat(P, H)

[0062] Where P is a matrix composed of time vectors, denoted as P = {W1|W2|…|W t |…|W T H is a historical flow data matrix H = {u1|u2|…|u}, composed of column vectors from the uf sequence. t |…|u T};

[0063] The decoder predicts the result based on the encoder's output:

[0064] Y = Decoder(Henc, Y <t ;θ dec )

[0065] Where Y is the output sequence of the decoder, Y <t θ is the output of the decoder before time step t. dec These are decoder parameters.

[0066] Furthermore, during the training of the large language model LLM, the multi-head attention layer and feedforward network of the large language model LLM are frozen during training, freezing the first F layers of the large language model LLM and unfreezing the parameters of the last U-layer multi-head attention layer.

[0067] Furthermore, the loss function for large-scale language models (LLMs) is:

[0068]

[0069]

[0070] Wherein, the regularization term λ∑κ∈θκ 2 For L2 regularization, θ is the learnable parameter of the model, λ is the regularization hyperparameter, and y is the true value of the training data. Let be the smoothed true value of the v-th predicted target. Let σ be the predicted probability of the v-th target, σ be the label smoothing factor, |V| be the number of points in the graph set, and ω be the coefficient for adjusting the overall loss function and "penalizing" negative sample pairs.

[0071]

[0072] Where: cos(·,·) is the distance function, and the formulas for calculating the u-th and i-th predicted targets are:

[0073]

[0074] Let u be the indicator function. The formulas for calculating the true values ​​of the smoothed u-th and i-th predicted targets are:

[0075]

[0076] Where, N r yes Random sampling pairs in Quantity, It is the point representation of the heterogeneous graph in the last hidden layer. As an indicator function, when When, it means and Different labels; when When, it means and Same tags.

[0077] The beneficial effects of this invention are as follows: by combining a space-time graph structure and a deep learning model, the accuracy and efficiency of prediction are improved, solving the problem that traditional techniques cannot capture the space-time dependencies in traffic flow data well; at the same time, based on this, an adaptive preference gate-based unified sentence vector large language model traffic flow prediction method is designed, which significantly improves the adaptability and generalization ability of the model on various traffic flow datasets, and has good scalability and universality, and can adapt to different spatiotemporal data prediction tasks. Attached Figure Description

[0078] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, use the same reference numerals to denote the same or similar parts. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0079] Figure 1 This is a flowchart of one embodiment of the present invention;

[0080] Figure 2 Adaptive preference gate principle diagram;

[0081] Figure 3 Diagram of door reset, door update, and hidden state. Detailed Implementation

[0082] like Figure 1 The large-scale highway network traffic flow map fine-tuning prediction method shown includes the following steps:

[0083] S1: Construct a heterogeneous graph of traffic flow in the highway network, where the nodes in the heterogeneous graph are toll station nodes and intersection nodes, and the edges in the heterogeneous graph represent the traffic flow between each node; the heterogeneous graph is represented as follows:

[0084] G = (V, E)

[0085] Wherein, V is the node set, including the toll station node U and the intersection node I; E is the edge set, including the traffic flow between the toll station node and the intersection node, the traffic flow between toll station nodes, and the traffic flow between intersection nodes.

[0086] The edge adjacency matrix of a heterogeneous graph is used to represent the edge relationships in the graph, where there are N... u One toll station node and N i For each intersection node, define the following adjacency matrix:

[0087] Intersection-Intersection Adjacency Matrix

[0088] Tollbooth-Intersection Adjacency Matrix Where aui =1 indicates that there is traffic flow between toll station u and intersection i.

[0089] Toll station - Toll station adjacency matrix

[0090] S2: Utilize graph convolutional network (GCN) to aggregate the neighbor information of nodes, capture the spatial characteristics of traffic flow in the highway network, and calculate the spatial weights of edges between nodes.

[0091] This step, based on the edge relationships of these heterogeneous graphs, uses a Graph Convolutional Network (GCN) to aggregate neighbor information and update the node representations. The node representation update can be achieved through graph convolutional layers, and the update rules for node representations (i.e., spatial features) are as follows:

[0092]

[0093] Where σ is the activation function. Let W be the set of neighboring nodes of node u. uv h represents the spatial weights of the edges between nodes u and v. Different types of edges have different weights, which are learned through the spatial graph. (l) This is the input for the layer l graph convolution.

[0094] S3: Add time information to the heterogeneous graph, use the Temporal Convolutional Network (TCN) combined with Chebyshev filtering and dilated convolution to capture the temporal characteristics of traffic flow in the highway network, and calculate the temporal weights of edges between nodes.

[0095] Existing traffic flow prediction models typically use data that is dynamic only within a small area, meaning that all interactions between toll stations change only locally and remain equally important over long periods. However, road network traffic is dynamic, and the time factor plays a crucial role in traffic forecasting. Traffic flow can vary significantly over long periods, therefore the time sensitivity of historical road network traffic data needs to be considered.

[0096] This invention incorporates time information (i.e., timestamps) into the graph structure to enhance prediction accuracy. The goal is to predict traffic flow at the next time step and over a future period, given traffic flow data from the past M time steps.

[0097] Input data

[0098] X = {x1, x2, ..., x} t …,x M}

[0099] Each of the X sequences For one-dimensional input, C i x is the input size of the time feature map. tIt represents the traffic flow characteristics at time step t, which includes different types of vehicle flow data.

[0100] Output data

[0101] Y = {y1, y2, ..., y} t …y M}

[0102] In multidimensional tensors C o The output size of the time feature map, where each y t It is the predicted traffic flow for the corresponding time step.

[0103] Next, we use TCN (Temporal Convolutional Network) to capture the temporal dependencies in traffic flow data through convolution operations, and then perform graph convolution G... α When applied to multidimensional tensor inputs, ρ layers of temporal convolutions are performed for each t across M time steps, enabling parallel computation.

[0104] For nodes u and v, the graph convolution G u,v The calculation method is as follows:

[0105]

[0106] Where: the softmax function is an activation function, applied to an element z in tensor Z. i and z j The formula for the softmax function is:

[0107]

[0108] X = {x1, x2, ..., x} t …,x M} represents the input data. Let σ be the adjacency matrix of the heterogeneous graph at the corresponding time step, and σ be the activation function (using ReLU as described above).

[0109] Let the edge weights be those of the l-th layer. The edge weights for the (l+1)th layer are:

[0110]

[0111] TCN performs convolution operations on this multidimensional tensor. This model uses dilated convolution to capture time dependencies, with each convolution kernel having a size of k and an dilation rate of d. A Chebyshev filter with good spectral characteristics is first used, allowing for tensor filtering under strict frequency response requirements.

[0112] Chebyshev filters control the filter's gain characteristics and minimize ripple in the frequency response by appropriately selecting Chebyshev coefficients. Chebyshev polynomials are a class of orthogonal polynomials commonly used for approximating functions and constructing filters.

[0113] In general, the first kind of Chebyshev polynomial T n The recurrence relation for (x) is:

[0114] T0(x)=1

[0115] T1(x)=x

[0116] T n (x)=2xT n-1 (x)-T n-2 (x), n≥2

[0117] In filter design, Chebyshev polynomials are used to construct the frequency response, particularly to control the "ripple" phenomenon in graphical convolutional filtering, i.e., gain fluctuations in the signal passband. For TCN, a Chebyshev Type I filter is used, and its transfer function is expressed as:

[0118]

[0119] Where: ε is the ripple coefficient of the filter, which controls the gain fluctuation in the passband. It is the nth order Chebyshev polynomial, ω c is the cutoff frequency of the filter, and s is the input parameter of the Chebyshev filter.

[0120] In this model, through parameter passing, H(s) is the graph convolution calculation result H(G) as described below. i,j ).

[0121] For C i The signal sequence input to ρ layers of temporal convolution on each channel The weights of the convolutional layer are w i The output of the (learnable parameters) after Chebyshev filtering is:

[0122]

[0123] G i,j The graph convolution G generated above corresponds to the graph convolution mentioned above. u,v x i and y i These are the constituent parameters of xt and yt.

[0124] To capture long-term dependencies, the TCN is stacked with multiple convolutional layers, each using a different dilation rate. Let the dilation rate of the l-th layer be d. l The kernel function is If the time series of highway traffic flow in N is represented by vectorized time as T0, T1, ..., TN, then the operator It can be represented as:

[0125]

[0126] The result of T after being generated by the F operator.

[0127] Perform a dilated convolution operation on each element Tt:

[0128]

[0129] It is the |Tt|-d l The input at t time steps represents the time steps skipped by the convolution kernel during dilated convolution. After performing dilated convolution, the corresponding dilated convolution prediction flow at each time step is obtained. The calculation was obtained, namely:

[0130]

[0131] Then the time information will be combined with the representation of the point-edge interaction. As time weight, Let h be the dilated convolution representation vector of the h-th layer. Let be the Chebyshev filter representation vector of the h-th layer. As shown in the formula:

[0132]

[0133] Different time series behaviors have varying impacts on prediction; therefore, time-weighted weighting is used to weight traffic flow. The closer the traffic flow time is to the present, the more representative it is of the true value for predicting the future period. This is achieved by introducing a decay factor, as shown in the equation:

[0134]

[0135] This represents the traffic flow prediction score, used to measure the correlation between specific time steps Tm and Tn. Intuitively, it represents the degree of influence of the traffic flow state at a historical time step m on the future time step n. Since the impact of proximity on prediction varies, a time decay factor α is used to adjust this weight.

[0136] Where α is the attenuation factor, representing the time elapsed between the traffic flow and the current time. This method uses exponential attenuation, i.e.:

[0137]

[0138] α m,n This represents the attenuation factor of traffic flow over time.

[0139] δ mn This represents the time difference (i.e., the absolute value of the time interval) between time steps m and n;

[0140] η represents a parameter that controls time decay, similar to the role of standard deviation. A larger value indicates slower time decay, meaning historical data still has influence over a longer period. A smaller value indicates that earlier historical data has less impact, and the model focuses more on more recent data.

[0141] χ represents the time weight threshold, setting a minimum weight threshold to filter irrelevant time steps. Only when... A value greater than or equal to χ is considered sufficient to indicate a significant influence between time steps i and j; otherwise, the influence is set to 0. This is equivalent to introducing a pruning mechanism, reducing the impact of irrelevant historical data on the current prediction and improving computational efficiency. (Updated) As shown in the formula:

[0142]

[0143] In this way, the time weight of traffic flow can be combined with the spatial weight, enabling this method to make accurate predictions that change over time.

[0144] S4: Concatenate the spatial features and temporal features to generate a spatial vector; concatenate the spatial weights and temporal weights to generate a temporal vector; then concatenate the spatial vector and temporal vector to construct a unified sentence vector; the unified sentence vector, Unified Sentence Vector (USV), is the framework used in this model to fuse spatiotemporal graph embedding vectors. After being reset and updated by an adaptive preference-gate bidirectional Bi-GRU (bidirectional gated recurrent unit), it is fine-tuned by Seq-to-Seq instructions to generate a sentence vector with predictive information.

[0145] The unified sentence vector uf is represented as:

[0146] uf = [gf; tf]

[0147] Where gf is a spatial vector, that is, after multi-layer spatial heterogeneous graph processing, the spatial weights Wuv and the spatial feature representation h of the last layer are obtained. (l+1)After passing through an adaptive recurrent gate, the spatial vector gf in the USV concatenation is obtained; the tf is the time vector, that is, after multiple layers of temporal convolution processing, the time weights are obtained. and time feature representation After passing through the adaptive loop gate, the time vector tf in the USV concatenation is obtained; [;] represents the vector concatenation operation.

[0148] Bidirectional adaptive recurrent gated recurrent unit (Bi-GRU) is a commonly used recurrent neural network (RNN) structure in natural language processing, capable of capturing forward and backward dependencies in sequences. Compared to the traditional GRU (gated recurrent unit), the adaptive Bi-GRU in this method enhances the model's memory capacity by running GRU units in both directions (forward and backward).

[0149] The adaptive preference gate consists of three gating mechanisms: a reset gate, an update gate, and a candidate hidden state. The hidden state h... t The update formula is:

[0150]

[0151] Where: z t It is the renew gate, which controls the combination of the hidden state from the previous time step with the current candidate hidden state. t It's the reset gate, which controls how the hidden state from the previous moment is reset. t These are the candidate hidden states, calculated using a non-linear activation function (tanh). w represents the learnable parameters of the Bi-GRU, obtained during training.

[0152] To unify the time-space vectors and generate a format usable by deep learning models, tokens need to be embedded using pointwise convolutions. In this model, the token embedding principle is: [The text abruptly shifts to a different topic] i Convert to Embedded

[0153] Emb i =Cov Point (x i ,θ i )

[0154] Among them, Emb i Represents the embedded token. Cov Point This indicates a pointwise convolution operation using a 1×1 convolution kernel. iThe input data (i.e., the input vector of pointwise convolution, as described above in terms of space and time), C i It is the input hidden dimension (compared to the input size C of the temporal feature map mentioned earlier). i (Same size), θ i This represents the learnable parameters at each point of the pointwise convolution.

[0155] For spatial correlation, this model uses adaptive vector embedding.

[0156]

[0157] Where σ represents the activation function. It is a learnable parameter, Par S It is a spatial vector embedding, that is, the embedded representation of the spatial vectors generated corresponding to the spatial weights in the above text (it is an embedded representation, the order has been changed).

[0158] To preserve the temporal information in the unified sentence vector, a linear layer is used to encode the input data into the number of days in a month and the number of days in a week, with these two encodings performed independently. Absolute location encoding is then applied to each provincial traffic data set at both monthly and weekly resolutions, generating location information as follows: and Embedded number of days in January Embedded with the number of days in a week Calculate using the following formula:

[0159]

[0160] in and This is a learnable temporal embedding for the number of days in a week and a month, which is the embedded representation of the time vectors generated corresponding to the time weights mentioned above. By adding these two embeddings, we obtain the temporal embedding representation.

[0161] S5: Input the unified sentence vector into a large language model LLM, and perform traffic flow prediction through instruction fine-tuning and partial attention layer freezing strategies.

[0162] Fine-tuning a large language model, LLM, requires splitting the input into two parts:

[0163] Spatiotemporal encoding part: Spatiotemporal weight matrix P (i.e., composed of the weights generated from the time-space graph above), P = {W1|W2|…|W t |…|W T}

[0164] Context: The representation of USV, uf, is a sequence u1, u2, ..., u t ,…,u T This sequence is arranged into a historical traffic data matrix H = {u1|u2|…|u...} by column vectors. t |…|u T}

[0165] The final USV input sequence can be expressed as:

[0166] X = Concat(P, H)

[0167] Concat(·) represents the matrix concatenation operation.

[0168] In order to adapt the already trained large language model (LLM) to the traffic prediction task, this model defines the time step of each location of the traffic flow data in the above method as a token, so that the spatial-temporal embedding layer converts these tokens into a spatial-temporal representation aligned with the LLM.

[0169] This model instruction fine-tuning uses LLM to guide the model to perform specific traffic flow prediction tasks, which can enhance the generalization ability and task understanding of large models and optimize the performance of traffic flow prediction.

[0170] LLM employs an encoder-decoder architecture. The goal of the instruction fine-tuning stage is to adjust the model parameters to optimize its understanding of spatiotemporal encoding and context. The encoder transforms the input sequence X into a contextual representation, as shown in the equation:

[0171] H enc =Encoder(X,θ) enc )

[0172] Among them, H enc It is the encoded context-hidden representation, θ enc These are encoder parameters.

[0173] The decoder generates the prediction result based on the encoder output, as shown in the following formula:

[0174] Y = Decoder(H) enc ,Y <t ;θ dec )

[0175] Where Y is the output sequence generated by the decoder, Y <t It is the decoder's output before time step t, θ dec These are decoder parameters.

[0176] This method introduces FusionConv to project traffic features to the dimensions required by the LLM. FusionConv integrates spatial and temporal embeddings to provide a unified representation of USV:

[0177] K F =FConv(Y; θ) f )

[0178] in K F This indicates that the unified sentence vectors are fused and convolved, and the resulting representation is used as input to MHA and LN, θ f The learnable parameters representing FConv.

[0179] This model employs a partially frozen attention-based Large Language Model (LLM), specifically designed for parameter freezing to improve traffic prediction accuracy. The multi-head attention layer (MHA) and feed-forward network (FFN) are frozen during training because these layers contain the most important parts learned in the LLM. The model keeps the first F layers frozen and unfrozes the parameters of the final U-layer multi-head attention layer, as these attention layers effectively handle spatial-temporal dependencies in the data. Therefore, the parameter-frozen LLM can adapt to traffic flow prediction tasks while preserving the foundational knowledge gained during pre-training.

[0180] Generally, layer normalization (LN) is placed at the input of each submodule. Compared to pre-activation in residual networks, LN is more suitable for vector inputs with uniform representations. This model adds additional layer normalization after the final multi-head attention layer.

[0181]

[0182] μ and σ represent the mean and standard deviation, respectively, ⊙ represents element-wise multiplication, and γ and β are learnable scaling and translation parameters. It is the output after the first LN, then... Sending MHA bullish attention:

[0183]

[0184] Each head i Operations based on attention mechanisms:

[0185]

[0186] W is a parameter of the attention mechanism, which is learned during training.

[0187] Attention(·) is calculated as follows:

[0188]

[0189] right Through the second LN, we obtain

[0190]

[0191] Next Entering the feedforward network FFN:

[0192]

[0193] In the first F layer of the frozen parameter LLM, the multi-head attention and feedforward layers are frozen. First, the USV is calculated using MHA, then the fused USV is added.

[0194]

[0195] Then, after normalization and FFN, plus the already frozen MHA... Get the output K to the next layer i+1 :

[0196]

[0197] Where i∈{1,…,F-1}, and K i+1 =[K F +PosEmb], where PosEmb represents a learnable positional code. This represents the intermediate representation of the i-th layer after applying frozen multi-head attention (MHA) and the first unfrozen layer normalization (LN). K i This represents the final representation after applying the unfrozen LN and the frozen feedforward network (FFN).

[0198] Similarly, the MHA is unfrozen in the final U layer of the LLM, allowing the model to capture the spatial-temporal dependencies of traffic flow data:

[0199]

[0200] in This represents the F+U-1 layer between the thawed MHA and the second frozen LN, where K is represented. F+U This represents the final output after applying the unfrozen LN and frozen FFN.

[0201] 2.3 Learning and Optimization Methods

[0202] This model designs a new loss function as follows:

[0203]

[0204] Where L soft As shown in equation (1-22):

[0205]

[0206] Regularization term λ∑ k∈θκ 2 L2 regularization penalizes model parameters to prevent overfitting. This controls model complexity, limits parameter size, effectively avoids overfitting on the training set, enhances generalization ability, makes the model more robust on the test set, and improves its predictive ability on unseen data. A Gaussian distribution function is chosen as the soft-label optimization objective, as shown in the equation:

[0207]

[0208] Where θ represents the learnable parameters of the model, λ represents the regularization hyperparameter, and y represents the true values ​​of the training data. It is the smoothed true value of the v-th predicted target. σ is the predicted probability of the v-th target, σ is the label smoothing factor, and |V| is the number of points in the graph. ω is the coefficient that adjusts the overall loss function and "penalizes" negative sample pairs.

[0209] Label smoothing avoids overfitting caused by the model's over-reliance on "hard labels" or "absolute labels." This model smooths the true labels y using a Gaussian distribution factor σ. In previous prediction tasks, labels were hard, but Gaussian label smoothing transforms these hard labels into a distribution, so the model no longer completely depends on the precise labels, but rather on the soft labels y. S Through this smoothing process, the model's predictions become more "fuzzy" and robust, reducing its reliance on hard labels. Especially when dealing with noisy data, the model can generalize better when faced with incomplete or sparse data.

[0210] σ controls the degree of smoothing. A smaller σ value indicates less smoothing of the labels, with the actual labels closer to hard labels; a larger σ value indicates more "loose" labels, allowing the model to accept a wider range of labels. This prevents the model from relying too heavily on extreme labels during training, avoiding overfitting. To reduce overfitting to noise, since traffic flow data is often sparse and may not be entirely accurate or contain noise, label smoothing introduces a certain degree of uncertainty into each label, reducing the model's over-reliance on noisy labels and thus improving the model's generalization ability.

[0211] L sam The loss function is as follows:

[0212]

[0213] Where: cos(·,·) is the distance function, and the formulas for calculating the u-th and i-th predicted targets are as follows:

[0214]

[0215] Cosine similarity is not sensitive to the length of vectors, but only focuses on direction. Therefore, it is suitable for vectorized representation in traffic flow prediction, that is, the similarity of embedded vectors in high-dimensional space. As the indicator function, the formulas for calculating the true values ​​of the smoothed u-th and i-th predicted targets are as follows:

[0216]

[0217] In L sam In the loss function, N r yes Random sampling pairs in Quantity, It is the point representation of the heterogeneous graph in the last hidden layer. As an indicator function, when Explanation and Different labels, when Then explain and Same tags

[0218] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A large-scale highway network traffic flow map fine-tuning prediction method using a large-scale model, characterized in that, Including the following steps: S1: Construct a heterogeneous graph of traffic flow in the highway network. The nodes in the heterogeneous graph are toll station nodes and intersection nodes, and the edges in the heterogeneous graph are the traffic flow between each node. S2: Utilize Graph Convolutional Network (GCN) to aggregate the neighbor information of nodes, capture the spatial characteristics of traffic flow in the highway network, and calculate the spatial weights of edges between nodes; S3: Add time information to the heterogeneous graph. Utilize a Temporal Convolutional Network (TCN) combined with Chebyshev filtering and dilated convolution to capture the temporal characteristics of traffic flow in the highway network and calculate the temporal weights of edges between nodes. The closer the traffic flow time is to the present, the more representative it is of the true value predicted for the subsequent period. This is achieved by introducing a decay factor, as shown in the equation: in, This represents the traffic flow prediction score, used to measure a specific time step. and The correlation between them; As the decay factor, exponential decay is adopted; S4: Concatenate the spatial and temporal features to generate a unified sentence vector; S5: Input the unified sentence vector into a large language model LLM, and perform traffic flow prediction through instruction fine-tuning and partial attention layer freezing strategies.

2. The large-scale highway network traffic flow map fine-tuning prediction method according to claim 1, characterized in that, The heterogeneous graph is represented as follows: in, The node set includes the toll station nodes. and intersection nodes ; It is an edge set, which includes the traffic flow between toll station nodes and intersection nodes, the traffic flow between toll station nodes, and the traffic flow between intersection nodes.

3. The large-scale highway network traffic flow map fine-tuning prediction method according to claim 2, characterized in that, The spatial features output by each graph convolutional layer of the graph convolutional network GCN are represented as follows: in: For activation function, For nodes The set of neighboring nodes, For nodes and nodes Spatial weights of the edges between them; For layer Input to graph convolution.

4. The large-scale highway network traffic flow map fine-tuning prediction method according to claim 3, characterized in that, The temporal features of the output of each convolutional layer of the Temporal Convolutional Network (TCN) are represented as follows: in: For nodes and nodes The time weight of the edges between them The vector is represented by dilated convolution. For the first Traffic flow prediction score at each time step The Chebyshev filter representation vector; The dilated convolution representation vector Represented as: in: For the first Traffic flow characteristics at each time step; For the first The expansion rate of the layer; For kernel functions; In order to be in Vectorized representation of traffic flow in the highway network over time in a time series; For the first The input at each time step represents the time steps skipped by the convolution kernel during the dilated convolution; The Temporal Convolutional Network (TCN) uses a Type I Chebyshev filter for tensor filtering. The transfer function of the Type I Chebyshev filter is expressed as: in: This is the ripple coefficient of the filter, used to control gain fluctuations in the passband; For the first Chebyshev polynomials Here, is the cutoff frequency of the filter, and s is the input parameter of the Chebyshev filter; By passing parameters This is the result of graph convolution calculation. ; For in On each channel Layer-time convolution input signal sequence The weights of the convolutional layer are The output after Chebyshev filtering is: This is graph convolution; Traffic flow prediction score Represented as: in: This is the attenuation factor of traffic flow over time. For time step and The time difference between them; Parameters used to control time decay; This is the time-weighted threshold.

5. The large-scale highway network traffic flow map fine-tuning prediction method according to claim 4, characterized in that, The unified sentence vector Represented as: in, for The vector representation; the For time characteristics Vector representation of; This represents the vector concatenation operation.

6. The large-scale highway network traffic flow map fine-tuning large model prediction method according to claim 5, characterized in that, The method also includes embedding the tokens via pointwise convolution; for spatial vectors, adaptive vector embedding is used. : in, This represents the activation function. , These are learnable parameters. It is a spatial vector embedding; For the time vector, a linear layer is used to encode the input data into the number of days in January and the number of days in a week. These two embeddings are then made independent and summed to obtain the time embedding representation. ; Embedded number of days in January for: Embedded number of days in a week in, and To generate location information by encoding the absolute location of each piece of provincial traffic data at "monthly" and "weekly" resolutions; and It is a learnable time embedding used for the number of days in a week and the number of days in a month.

7. The large-scale highway network traffic flow map fine-tuning prediction method according to claim 6, characterized in that, The method also includes using an adaptive preference gate bidirectional gated recurrent unit to process the input from front to back and from back to front, respectively. sequence After processing, the forward sequence output is: The reverse sequence output is: The hidden state is obtained by concatenating the hidden states of the forward and reverse sequence outputs. for: Then, the degree of attention given to different features is dynamically adjusted, and an additional gating mechanism is introduced in the first... At any given time, the update process of the preference gate is as follows: in, For the current moment, and For learnable weights, This is the input for the current moment.

8. The large-scale highway network traffic flow map fine-tuning prediction method according to claim 7, characterized in that, The large language model LLM adopts an encoder and decoder architecture; The encoder fine-tunes the input sequence via instructions. Convert to context representation: in, For the encoded context-hidden representation, For encoder parameters; in, The matrix composed of time vectors is represented as follows: ; for The sequence is arranged into a historical flow data matrix by column vectors. ; The decoder predicts the result based on the encoder's output: in, This is the output sequence of the decoder. For the decoder at time step The previous output, These are decoder parameters.

9. The large-scale highway network traffic flow map fine-tuning prediction method according to claim 8, characterized in that, During the training of the large language model LLM, the multi-head attention layer and feedforward network of the large language model LLM are frozen during training, thus preventing the large language model LLM from being overloaded. Layer freeze, thaw the last one Multi-head attention layer parameters.

10. The large-scale highway network traffic flow map fine-tuning large model prediction method according to claim 9, characterized in that, The loss function for large language model LLM is: Among them, the regularization term For L2 regularization, These are the learnable parameters of the model. For regularization hyperparameters, For the true values ​​of the training data, For the first The smoothed true value of a predicted target. For the first The predicted probability of an object. For label smoothing factor, The number of points in the graph; To adjust the overall loss function, the coefficients of the negative sample pairs are "penalized"; in: Let be the distance function, the first... The and the first The formula for calculating each prediction target is: As an indicator function, the smoothed first... The and the first The formula for calculating the true value of each predicted target is: in, yes Random sampling pairs in Quantity, It is the point representation of the heterogeneous graph in the last hidden layer. As an indicator function, when When, it means and Different labels; when When, it means and Same tags.

Images