Traffic flow prediction method and device based on model lightening and medium

Abstract

This invention relates to a traffic flow prediction method, apparatus, and medium based on model lightweighting. The method includes: Step S1: Obtaining a traffic flow prediction model to be optimized; Step S2: Calculating the hierarchical importance of all intermediate layers in the trained traffic flow prediction model; Step S3: Obtaining the normalized importance of each intermediate layer; Step S4: Selecting and retaining several intermediate layers with high normalized importance using cumulative contribution ratio constraints and absolute ratio constraints to obtain a temporary traffic flow prediction model; Step S5: Determining whether the model accuracy meets the requirements. If yes, proceed to Step S6; otherwise, proceed to Step S7; Step S6: Using the temporary traffic flow prediction model as the current traffic flow prediction model and returning to Step S2; Step S7: Outputting the current traffic flow prediction model as the lightweighted result. Compared with existing technologies, this invention achieves accurate traffic flow prediction on a low-computing-power computing platform.

Description

Technical Field

[0001] This invention relates to the field of traffic flow prediction, and in particular to a model-based lightweight traffic flow prediction method, apparatus, and medium. Background Technology

[0002] Traffic flow forecasting is a crucial reference for transportation planning. As a type of time-series data, current technologies typically employ time series forecasting. In recent years, inspired by the success of large-scale models in natural language processing, time series forecasting research has primarily developed two technical approaches: one is Large Language Models for Time Series (LM4TS), which embeds and aligns time-series features into text modalities, leveraging knowledge transfer from existing language models for prediction; the other is Time Series Foundation Models (TSFMs), which directly trains models from scratch based on massive amounts of time-series data to learn universally applicable time-series representations. Although the implementation paths of these two paradigms differ, they generally rely on the same assumption—that increasing model size and data volume can improve prediction performance. However, this assumption does not always hold true in time series modeling.

[0003] Existing research often treats these phenomena as "random occurrences" or "experimental anomalies," lacking systematic theoretical explanations and empirical analyses. This leads to model design still following a "blind expansion" approach, ignoring the actual differences in the contributions of the model's internal structure. Existing large-scale time series models exhibit the "expansion paradox," meaning that as the model size increases, predictive performance decreases rather than increases.

[0004] In response, while some existing technologies have introduced methods to prune models using layer importance scores to achieve model lightweighting—for example, Chinese patent CN120748376A discloses obtaining the importance score of each convolutional layer by calculating the proportion of its own trained weight parameters and pruning convolutional layers with lower importance scores—this method has the following drawbacks when applied to time series models:

[0005] 1. The inter-layer interactions in time series models are complex (e.g., through attention mechanisms). The weight monitoring methods mentioned above cannot effectively capture the cosine similarity or attention weight changes between layers, which may lead to the erroneous pruning of layers that are key to time dynamics. This is especially true for pure data such as traffic flow data that lacks semantic data, where the weight parameters of the convolutional layer after training cannot truly reflect its layer contribution.

[0006] 2. Removing multiple layers at once may disrupt the sequence flow of the time series model, leading to an irreversible decline in prediction performance. Time series models need to be dynamically adjusted according to changes in data distribution. The static pruning method described above cannot verify accuracy through iteration and may cause the model to overfit or underfit.

[0007] 3. The dependency network in the above methods may not be able to accurately model the topology of the time series model. In the time dimension, the dependency network does not consider the sequence step size or time window dependency in the time series, and the time continuity may be destroyed during pruning. In terms of attention layer processing, residual connections or multi-head attention in the Transformer layer may complicate the dependency relationship. The simple directed graph of the above methods may not be able to identify the critical path, resulting in network breakage after pruning.

[0008] 4. Time series data is sensitive to numerical precision. Quantization may reduce the model's ability to detect sudden patterns or outliers. The methods mentioned above are not optimized for the numerical distribution of time series data. In addition, they are not sensitive to attention mechanisms. Specifically, the attention heads in time series models may have high importance but low weight norms. The methods mentioned above may misclassify them as redundant layers, leading to a decrease in model performance.

[0009] Ultimately, when the above methods are applied to the lightweighting of time series models, they cannot guarantee the reduction of more intermediate layers without reducing model accuracy, thus failing to achieve accurate traffic flow prediction on computing platforms with lower computing power. Summary of the Invention

[0010] The purpose of this invention is to provide a model-based lightweight traffic flow prediction method, device, and medium that can achieve accurate traffic flow prediction on a computing platform with lower computing power.

[0011] The objective of this invention can be achieved through the following technical solutions:

[0012] A model-lightweight traffic flow prediction method includes:

[0013] Step S1: Obtain the traffic flow prediction model to be optimized, and one or more training sets, wherein the traffic flow prediction model is a time series model, the input is the historical traffic flow sequence and time features, and the output is the future traffic flow sequence;

[0014] Step S2: Train the current traffic flow prediction model using any training set, and calculate the hierarchical importance of all intermediate layers of the trained traffic flow prediction model, wherein the intermediate layers are all layers except the feature input layer and the prediction output layer;

[0015] Step S3: Normalize the hierarchical importance of all intermediate layers to obtain the normalized importance of each intermediate layer;

[0016] Step S4: Based on the normalized importance of each intermediate layer, the cumulative contribution ratio constraint and the absolute ratio constraint are used to select and retain a number of intermediate layers with high normalized importance to obtain a temporary traffic flow prediction model.

[0017] Step S5: Train the current temporary traffic flow prediction model using any training set, and check the accuracy of the trained model. Determine whether the model accuracy meets the requirements. If yes, proceed to step S6; otherwise, proceed to step S7.

[0018] Step S6: Use the temporary traffic flow prediction model as the current traffic flow prediction model and return to step S2;

[0019] Step S7: Output the current traffic flow prediction model as the lightweight result;

[0020] Step S8: Use the lightweight traffic flow prediction model to predict traffic flow.

[0021] In step S2, different training sets are used to train the current traffic flow prediction model in each iteration.

[0022] Step S2 includes:

[0023] Step S2-1: Train the current traffic flow prediction model using any training set;

[0024] Step S2-2: Select any sample as the input of the trained traffic flow prediction model, and calculate the cosine similarity between the output of each intermediate layer and the output of other intermediate layers based on the input and output of each intermediate layer, as well as the attention weight of each attention head of each intermediate layer.

[0025] Step S2-3: Based on the average attention weight of each attention head in each intermediate layer, further calculate the average similarity of all attention head pairs;

[0026] Step S2-4: Calculate the attenuation metric of the influence of the precursor layer based on the cosine similarity of the inputs and outputs of each intermediate layer;

[0027] Step S2-5: Calculate the hierarchy importance based on the precursor layer influence decay metric and the average similarity of all attention head pairs.

[0028] The importance of the hierarchy is as follows:

[0029] ,

[0030] in: Il For the first l The importance of each intermediate layer R l For the first l The precursor layer of each intermediate layer affects the attenuation metric. For the first l The average attention weight of all attention heads in the intermediate layer.

[0031] The specific effect of the precursor layer on the attenuation metric is as follows:

[0032] ,

[0033] ,

[0034] Where: K is the number of leading layers. w k Let K be the strength of the effect of the k-th leading layer on the following layer. α i Let be the attenuation factor of the i-th leading layer. α k H is the attenuation factor of the k-th leader layer. l For the first l The output of the intermediate layer, H l-k For the first l - k The output of the intermediate layer Sim (·) represents the cosine similarity.

[0035] Step S2-2 includes:

[0036] Step S2-2-1: Select any sample as the input to the trained traffic flow prediction model, and calculate the input and output of each intermediate layer based on the input and output of each intermediate layer respectively;

[0037] Step S2-2-2: Calculate the Euclidean distance between the input and output of each intermediate layer, and determine whether there is an intermediate layer whose own input and output Euclidean distance is less than the pre-configured first threshold distance. If so, remove this intermediate layer to obtain the updated current traffic flow prediction model and return to step S2-1. Otherwise, execute step S2-2-3.

[0038] Step S2-2-3: Calculate the cosine similarity between the output of each intermediate layer and the output of other intermediate layers, as well as the attention weight of each attention head of each intermediate layer.

[0039] Step S4 includes:

[0040] Step S4-1: Sort the normalized importance of all intermediate layers from largest to smallest;

[0041] Step S4-2: Based on the sorted normalized importance sequence, starting from the first element in the normalized importance sequence, calculate the cumulative contribution ratio of each element in the normalized importance sequence.

[0042] Step S4-3: Select the first normalized importance whose cumulative contribution ratio exceeds the first set ratio as the target element. Determine whether the proportion of the number from the first element in the normalized importance sequence to the target element exceeds the proportion of the number of intermediate layers. If yes, proceed to step S4-4; otherwise, proceed to step S4-5.

[0043] Step S4-4: Retain the intermediate layers corresponding to the target element and the elements preceding it in the normalized importance sequence, and delete other intermediate layers to obtain the temporary traffic flow prediction model;

[0044] Step S4-5: Retain the intermediate layers corresponding to the elements with the second-highest set proportion in the normalized importance sequence, and delete the other intermediate layers to obtain the temporary traffic flow prediction model.

[0045] The traffic flow prediction model is either a traffic flow prediction model based on a large language model or a time series basic model.

[0046] A model-based lightweight traffic flow prediction device includes a memory, a processor, and a program stored in the memory, wherein the processor executes the program to implement the method described above.

[0047] A storage medium having a program stored thereon, which, when executed, implements the method described above.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. By using cumulative contribution ratio constraints and absolute ratio constraints, a temporary traffic flow prediction model is obtained by selecting and retaining several intermediate layers with high normalization importance. Combined with iterative optimization, the accuracy drop caused by a single pruning is avoided. Thus, the most lightweight and effective network structure can be found by successive approximations. Accurate traffic flow prediction can be achieved on computing platforms with lower computing power, thus adapting to traffic flow prediction models with arbitrary input features.

[0050] 2. Layer importance is calculated based on the influence attenuation metric of the preceding layer and the average similarity of all attention head pairs, rather than based on a single variable of weight parameters. Instead, layer importance is considered based on the cosine similarity of the input and output of each intermediate layer, which can effectively capture the cosine similarity or attention weight changes between layers, effectively improve the sensitivity weight of different layers in processing time series data, and effectively discover the layers that truly play a key role in the time series feature extraction process, thereby obtaining the key layers related to traffic flow data.

[0051] 3. Based on the inputs and outputs of each intermediate layer, intermediate layers with overly similar inputs and outputs can be directly removed, which can reduce noise during the calculation of hierarchical importance.

[0052] 4. By identifying and retaining the key layers that truly play a role in the prediction process, this invention significantly reduces the number of model parameters and computational burden, improving inference speed and resource utilization. The proposed layer contribution evaluation and dynamic selection mechanism has good universality and scalability. This method can be applied simultaneously to time series models with different architectures, such as the language model-based LLM4TS and the pure time series pre-trained model TSFM, and is compatible with various network forms such as encoder-only, decoder-only, and encoder-decoder. This mechanism can effectively suppress the training instability problem caused by inter-layer redundancy, enhance the model's adaptability to different data distributions and generalization performance, thus providing a new solution for the lightweighting, interpretability, and cross-domain application of large time series models. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the main steps of the method of the present invention;

[0054] Figure 2 This is a schematic diagram showing the calculation results of hierarchical importance. Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0056] This application addresses the "extension paradox" problem in existing time series models by proposing an efficient simplification method for large time series models based on layer importance identification. First, hierarchical feature analysis is performed on the trained time series model. By constructing a layer contribution metric, the actual performance of each layer in the prediction task is quantitatively evaluated, thereby identifying the "key execution layers" that truly play a dominant role in the prediction process. Based on this, the original model is structurally restructured by retaining key layers and eliminating redundant layers. This significantly reduces computational load and parameter size while maintaining overall architectural consistency. This approach effectively eliminates ineffective layers within the model, improves parameter utilization, and alleviates overfitting, thus achieving higher prediction accuracy and inference speed without relying on increasing model size or data volume.

[0057] like Figure 1 As shown, a model-based lightweight traffic flow prediction method includes:

[0058] Step S1: Obtain the traffic flow prediction model to be optimized, and one or more training sets;

[0059] The traffic flow prediction model is a time series model, either LLM4TS or TSFM. The input to the traffic flow prediction model is historical traffic flow sequences and time features, and the output is future traffic flow sequences. The time features may include:

[0060] Date tags can be categorized as: weekdays, rest days, and public holidays.

[0061] Seasons can be divided into: spring, summer, autumn, and winter;

[0062] Geographical location, expressed in latitude and longitude.

[0063] The different types of time features selected will cause changes in the importance of different layers, and these changes are not entirely consistent with the weight parameters.

[0064] Step S2: Train the current traffic flow prediction model using any training set, and calculate the hierarchical importance of all intermediate layers in the trained traffic flow prediction model. The intermediate layers are all layers except the feature input layer and the prediction output layer, including:

[0065] Step S2-1: Train the current traffic flow prediction model using any training set;

[0066] Step S2-2: Select any sample as the input of the trained traffic flow prediction model, and calculate the cosine similarity between the output of each intermediate layer and the output of other intermediate layers based on the input and output of each intermediate layer, as well as the attention weight of each attention head of each intermediate layer.

[0067] Among them, the l The output of the intermediate layer and the first... l -1 intermediate layer output (i.e., the -1th) l The cosine similarity of the inputs from the intermediate layers is:

[0068] ,

[0069] Where <·> represents vector dot product, and ||·||2 represents Euclidean distance.

[0070] Furthermore, in this embodiment, step S2-2 includes:

[0071] Step S2-2-1: Select any sample as the input to the trained traffic flow prediction model, and calculate the input and output of each intermediate layer based on the input and output of each intermediate layer respectively;

[0072] Step S2-2-2: Calculate the Euclidean distance between the input and output of each intermediate layer, and determine whether there is an intermediate layer whose own input and output Euclidean distance is less than the pre-configured first threshold distance. If so, remove this intermediate layer to obtain the updated current traffic flow prediction model and return to step S2-1. Otherwise, execute step S2-2-3.

[0073] By directly eliminating intermediate layers whose inputs and outputs are too similar, noise can be reduced when calculating the importance of a hierarchy.

[0074] Step S2-2-3: Calculate the cosine similarity between the output of each intermediate layer and the output of other intermediate layers, as well as the attention weight of each attention head of each intermediate layer.

[0075] Step S2-3: Based on the average attention weight of each attention head in each intermediate layer, further calculate the average similarity of all attention head pairs:

[0076] ,

[0077] in: N The number of attention heads in a single intermediate layer. For the first l The first intermediate layer i Attention weights for each attention head For the first l The first intermediate layer j Attention weights for each attention head.

[0078] Each attention head can be viewed as an independent relation learner, which is parameterized by different projection matrices.

[0079] Step S2-4: Calculate the attenuation metric of the precursor layer's influence based on the cosine similarity of the inputs and outputs of each intermediate layer.

[0080] ,

[0081] ,

[0082] Where: K represents the number of leading layers. w k Let K be the strength of the effect of the k-th leading layer on the following layer. α i Let be the attenuation factor of the i-th leading layer. α k H is the attenuation factor of the k-th leader layer. l For the first l The output of the intermediate layer, H l-k For the first l - k The output of the intermediate layer Sim (·) represents the cosine similarity.

[0083] Step S2-5: Calculate the hierarchical importance based on the precursor layer's influence attenuation metric and the average similarity of all attention head pairs, where the hierarchical importance is:

[0084] ,

[0085] in: I l For the first l The importance of each intermediate layer R l For the first l The precursor layer of each intermediate layer affects the attenuation metric. For the first l The average attention weight of all attention heads in the intermediate layer.

[0086] In addition, in this embodiment, in step S2, different training sets are used to train the current traffic flow prediction model in each iteration.

[0087] Step S3: Normalize the hierarchical importance of all intermediate layers to obtain the normalized importance of each intermediate layer;

[0088] Step S4: Based on the normalized importance of each intermediate layer, using cumulative contribution ratio constraints and absolute ratio constraints, select and retain several intermediate layers with high normalized importance to obtain a temporary traffic flow prediction model, including:

[0089] Step S4-1: Sort the normalized importance of all intermediate layers from largest to smallest;

[0090] Step S4-2: Based on the sorted normalized importance sequence, starting from the first element in the normalized importance sequence, calculate the cumulative contribution ratio of each element in the normalized importance sequence.

[0091] Step S4-3: Select the first normalized importance whose cumulative contribution ratio exceeds the first set ratio as the target element. Determine whether the proportion of the number from the first element in the normalized importance sequence to the target element exceeds the proportion of the number of intermediate layers. If yes, proceed to step S4-4; otherwise, proceed to step S4-5.

[0092] Step S4-4: Retain the intermediate layers corresponding to the target element and the elements preceding it in the normalized importance sequence, and delete other intermediate layers to obtain the temporary traffic flow prediction model;

[0093] Step S4-5: Retain the intermediate layers corresponding to the elements with the second-highest set proportion in the normalized importance sequence, and delete the other intermediate layers to obtain the temporary traffic flow prediction model.

[0094] Step S5: Train the current temporary traffic flow prediction model using any training set, and check the accuracy of the trained model. Determine whether the model accuracy meets the requirements. If yes, proceed to step S6; otherwise, proceed to step S7.

[0095] Step S6: Use the temporary traffic flow prediction model as the current traffic flow prediction model and return to step S2;

[0096] Step S7: Output the current traffic flow prediction model as the lightweight result;

[0097] Step S8: Use the lightweight traffic flow prediction model to predict traffic flow.

[0098] This application visualizes the average distribution of importance scores for each layer on the validation sets of different datasets, such as... Figure 2 As shown, the three hyperparameters are: the first set ratio, the decay factor α of the leading layer influence, and the cumulative score ratio β%, set to 80%, 50%, and 85%, respectively. Based on importance ranking, redundant layers are pruned, and the pruned model is fine-tuned on the training set to realign with the data distribution. We comprehensively evaluate the performance of the pruned model and the original model in in-domain and out-of-domain prediction scenarios, considering both prediction error and inference efficiency.

[0099] Figure 2In the diagram, Small, Tiny, and Large represent different model sizes, while LLM4TS (large language model for time series) and TSFMs (time series foundation models) represent two paradigms of large-scale time series forecasting models. The darker the color, the more important it is.

[0100] For the four LLM4TS models, in this embodiment, the second setting ratio is chosen to be one-third, ultimately achieving a 1.5x inference speedup. This pruning strategy outperforms the traditional baseline truncation method in over 80% of scenarios, with only minor performance degradation (average less than 1%) in a very few cases. For the four TSFMs models, we retain an average of only about 26.9% of the layers, achieving a speedup of over 2.7x. By retaining only the key layers, model performance is almost unaffected. On various mainstream LLM4TS and TSFMs models in the time series domain, our method demonstrates that retaining only the key layers not only significantly reduces model size but also maintains or even improves prediction accuracy.

[0101] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A traffic flow prediction method based on model lightweighting, characterized in that, include: Step S1: Obtain the traffic flow prediction model to be optimized, and one or more training sets, wherein the traffic flow prediction model is a time series model, the input is the historical traffic flow sequence and time features, and the output is the future traffic flow sequence; Step S2: Train the current traffic flow prediction model using any training set, and calculate the hierarchical importance of all intermediate layers of the trained traffic flow prediction model, wherein the intermediate layers are all layers except the feature input layer and the prediction output layer; Step S3: Normalize the hierarchical importance of all intermediate layers to obtain the normalized importance of each intermediate layer; Step S4: Based on the normalized importance of each intermediate layer, the cumulative contribution ratio constraint and the absolute ratio constraint are used to select and retain a number of intermediate layers with high normalized importance to obtain a temporary traffic flow prediction model. Step S5: Train the current temporary traffic flow prediction model using any training set, and check the accuracy of the trained model. Determine whether the model accuracy meets the requirements. If yes, proceed to step S6; otherwise, proceed to step S7. Step S6: Use the temporary traffic flow prediction model as the current traffic flow prediction model and return to step S2; Step S7: Output the current traffic flow prediction model as the lightweight result; Step S8: Use the lightweight traffic flow prediction model to predict traffic flow. Step S2 includes: Step S2-1: Train the current traffic flow prediction model using any training set; Step S2-2: Select any sample as the input of the trained traffic flow prediction model, and calculate the cosine similarity between the output of each intermediate layer and the output of other intermediate layers based on the input and output of each intermediate layer, as well as the attention weight of each attention head of each intermediate layer. Step S2-3: Based on the average attention weight of each attention head in each intermediate layer, further calculate the average similarity of all attention head pairs; Step S2-4: Calculate the attenuation metric of the influence of the precursor layer based on the cosine similarity of the inputs and outputs of each intermediate layer; Step S2-5: Calculate the hierarchical importance based on the precursor layer's influence attenuation metric and the average similarity of all attention head pairs; The importance of the hierarchy is as follows: in: I l For the first l The importance of each intermediate layer R l For the first l The precursor layer of each intermediate layer affects the attenuation metric. For the first l The average attention weight of all attention heads in the intermediate layer.

2. The traffic flow prediction method based on model lightweighting according to claim 1, characterized in that, In step S2, different training sets are used to train the current traffic flow prediction model in each iteration.

3. The traffic flow prediction method based on model lightweighting according to claim 1, characterized in that, The specific effect of the precursor layer on the attenuation metric is as follows: Where: K is the number of leading layers. w k Let K be the strength of the effect of the k-th leading layer on the following layer. α i Let be the attenuation factor of the i-th leading layer. α k H is the attenuation factor of the k-th leader layer. l For the first l The output of the intermediate layer, H l-k For the first l - k The output of the intermediate layer Sim (·) represents the cosine similarity.

4. The traffic flow prediction method based on model lightweighting according to claim 1, characterized in that, Step S2-2 includes: Step S2-2-1: Select any sample as the input to the trained traffic flow prediction model, and calculate the input and output of each intermediate layer based on the input and output of each intermediate layer respectively; Step S2-2-2: Calculate the Euclidean distance between the input and output of each intermediate layer, and determine whether there is an intermediate layer whose own input and output Euclidean distance is less than the pre-configured first threshold distance. If so, remove this intermediate layer to obtain the updated current traffic flow prediction model and return to step S2-1. Otherwise, execute step S2-2-3. Step S2-2-3: Calculate the cosine similarity between the output of each intermediate layer and the output of other intermediate layers, as well as the attention weight of each attention head of each intermediate layer.

5. The traffic flow prediction method based on model lightweighting according to claim 1, characterized in that, Step S4 includes: Step S4-1: Sort the normalized importance of all intermediate layers from largest to smallest; Step S4-2: Based on the sorted normalized importance sequence, starting from the first element in the normalized importance sequence, calculate the cumulative contribution ratio of each element in the normalized importance sequence. Step S4-3: Select the first normalized importance whose cumulative contribution ratio exceeds the first set ratio as the target element. Determine whether the proportion of the number from the first element in the normalized importance sequence to the target element exceeds the proportion of the number of intermediate layers. If yes, proceed to step S4-4; otherwise, proceed to step S4-5. Step S4-4: Retain the intermediate layers corresponding to the target element and the elements preceding it in the normalized importance sequence, and delete other intermediate layers to obtain the temporary traffic flow prediction model; Step S4-5: Retain the intermediate layers corresponding to the elements with the second-highest set proportion in the normalized importance sequence, and delete the other intermediate layers to obtain the temporary traffic flow prediction model.

6. The traffic flow prediction method based on model lightweighting according to claim 1, characterized in that, The traffic flow prediction model is either a traffic flow prediction model based on a large language model or a time series basic model.

7. A model-based lightweight traffic flow prediction device, comprising a memory, a processor, and a program stored in the memory, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-6.

8. A storage medium having a program stored thereon, characterized in that, When the program is executed, it implements the method as described in any one of claims 1-6.

Images