Virtual time series data generation method based on large language model fine tuning

Abstract

The invention relates to a virtual time series data generation method based on fine tuning of a large language model, which comprises the following steps of: firstly, carrying out standardization processing on data by adopting instance Normalization so as to retain statistical characteristics of a time series, and then, aggregating sequences of adjacent time steps into sequence blocks through a partitioning operation; then, fast Fourier transform (FFT) is utilized to extract a low-frequency trend component in the time sequence as a Prompt, and trend information is injected into the model to enhance the capability of capturing the long-term mode of the time sequence. In a model processing stage, a parameter budget is adaptively allocated between activated parameters and layers according to importance scores of a weight matrix, so that the model can effectively utilize pre-training knowledge of a large model, and meanwhile, the calculation overhead is reduced. Finally, the framework verifies the effect on a plurality of reference data sets (ETTh1, ETTm1, ETTm2 and the like).

Description

TECHNICAL FIELD

[0001] The application relates to a virtual time series data generation method based on large language model fine-tuning, and belongs to the technical field of pattern recognition in computer intelligence science. BACKGROUND

[0002] In today's era of accelerating intelligence and automation, time series data, as a key data form describing the state of a system changing over time, is widely used in power dispatching, weather monitoring, financial transactions, medical diagnosis, industrial manufacturing, and many other fields. Many key systems in the real world evolve around time as the core axis, and their core characteristics can often be expressed as one-dimensional or multi-dimensional time series, such as load variation curves in power grids, power output time series of wind farms, price fluctuation sequences in stock markets, and patient physiological monitoring data. By modeling and analyzing these time series data, not only can the dynamic evolution mechanism of the system be understood, but also intelligent decision-making such as prediction, control, and early warning can be achieved, thereby improving operational efficiency and safety levels.

[0003] However, due to problems such as data acquisition costs, scene scarcity, privacy compliance, and distribution drift, there is still a significant shortage of high-quality time series data available for training and verification in the real world. Under this background, developing a virtual time series data generation method with high fidelity and high semantic consistency has become one of the key technologies to promote the research and landing of intelligent systems. By simulating historical laws and potential evolution patterns to automatically generate time series samples, not only can the data scale be effectively expanded and the model robustness be enhanced, but also it can serve engineering testing tasks such as fault injection and boundary analysis.

[0004] Traditional time series generation methods mainly rely on recurrent neural networks (RNN), long short-term memory networks (LSTM), variational autoencoders (VAE), and Transformer structures based on self-attention mechanisms. These models usually rely on time-dependent modeling and sampling mechanisms to simulate the target distribution. However, there are still many challenges in practical applications: first, these methods are prone to generate data jumps, abnormal fluctuations, and discontinuities when facing high complexity long-term dependencies and non-stationary processes, resulting in a decline in overall data quality; second, there is a lack of explicit modeling means for long-term trends and periodicity, which easily ignores the macro evolution laws in the underlying time series structure; in addition, traditional models often have weak understanding of domain features, making it difficult to reflect semantic consistency and rationality of industry background in the generated data, affecting their application effect in high-demand scenarios. SUMMARY

[0005] To overcome the aforementioned problems, this invention proposes a virtual time-series data generation method based on fine-tuning of a large language model. Specifically, firstly, the original time-series data is decomposed in the frequency domain using Fourier Transform to extract low-frequency signals reflecting long-term evolutionary trends. These trend components are then used to construct semantically informative input prompts, guiding the language model to generate time series with long-term consistency. Secondly, an adaptive fine-tuning mechanism is designed to adaptively allocate parameter budgets among activated parameters and layers based on the importance scores of the weight matrix. This allows the model to effectively utilize the pre-training knowledge of a large model while understanding complex temporal correlations, thereby generating higher-quality time-series data.

[0006] To achieve the above objectives, this invention proposes a virtual time-series data generation method based on fine-tuning of a large language model. This method leverages the capabilities of a large language model to address the problems of severe jumps and poor data quality inherent in existing time-series generation methods. Based on a combination of Fourier transform and a large language model, this invention proposes a novel virtual time-series data generation method with a reasonable structure and excellent performance. It provides a practical solution to the problems of severe jumps and insufficient trend modeling in existing time-series generation methods, and offers a reference for exploring the application of language models in the field of time-series modeling.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for generating virtual time-series data based on fine-tuning of a large language model, the method comprising the following steps:

[0009] Step 1: Treat the original time series as the sequence to be processed, X∈R. T×N×d , where X t ={x1,x2,…,x N} represents an input time series of length N at time step t∈T, given the original sequence X, and the mask set. This method aims to predict the masked value X. mask For each time series data to be processed, the method first performs preprocessing on the data, including instance normalization and sequence segmentation. The entire process includes the following sub-steps:

[0010] Step 1-1: First, apply Reverse Instance Normalization to X, calculate the mean μ and standard deviation σ of the sequence using the formula... Standardization is performed to eliminate dimensional differences and preserve the statistical properties of the series.

[0011] Steps 1-2, followed by the normalized sequence The sequence is divided into fixed-length time patches. Each patch aggregates P adjacent time steps to form a labeled sequence. in This step reduces sequence redundancy and expands the input time range, thereby effectively capturing local patterns in the time series.

[0012] Step 2 involves performing a Fourier transform on the labeled sequence and obtaining the periodic signal by submitting high-frequency information. The specific extraction process is as follows:

[0013] Step 2-1, for the labeled sequence Perform a Fast Fourier Transform (FFT) to obtain its frequency domain representation. The mathematical formula for FFT is:

[0014]

[0015] in It is the complex frequency domain component corresponding to frequency k at time step t, where j represents the imaginary unit, satisfying j 2 =-1, It is a complex exponential function.

[0016] Step 2-2, for frequency domain representation Consider the top-k largest high-frequency components, denoted as X'[k], and use the inverse Fourier transform to obtain the periodic signal:

[0017]

[0018] Step 3, through the marked sequence blocks Additive decomposition is performed to obtain trend components with time-series characteristics. Based on these trend components, prompts are generated and fed into the language model along with the time-series data for processing. The specific process is as follows:

[0019] Step 3-1, use the formula For the labeled sequence Decompose, where S t T represents a periodic component. t Indicating the trend component, δ t Represents noise components.

[0020] Step 3-2, next use a multilayer perceptron (MLP) to analyze T. t Perform a linear transformation to compress its dimensions and generate a low-dimensional Prompt, making the trend component T... t It can be better integrated into the input of large language models:

[0021] X Prompt =MLP(T t (3)

[0022] Step 3-3, X Prompt With serialized tag blocks The spliced ​​data is then fed into a larger model for representation:

[0023]

[0024] Step 4: Fine-tune the large language model for the task using adaptive pruning. The specific process is as follows:

[0025] Step 4-1: Fine-tune the large model parameters using incremental updates of the pre-trained weight matrix in the form of singular value decomposition.

[0026] W = W + Δ = W + PΛQ(5)

[0027] in and These represent the left and right eigenvectors of Δ, and the diagonal matrix, respectively. Includes singular values ​​{λ i} 1≤i≤r And r < <min(d1,d2),

[0028] Step 4-2: P and Q are initialized using random Gaussian initialization. Furthermore, to ensure the orthogonality of P and Q, i.e. Add the following regular expression:

[0029]

[0030] in Represents the identity matrix.

[0031] Step 4-3: Apply the adaptive process based on Singular Value Decomposition (SVD) (Equation 5) to the weight matrix W of each large model transformer layer. q W k W v W f1 and W f2 And use k to index the increment matrix, i.e., Δ. k =P k Λ k Q k Where k = 1, ..., n, n represents the number of adaptation weight matrices, and the matrix includes the k-th increment matrix Δ. k The triplet of the i-th eigenvalue and eigenvector is represented as G. k,i ={P k,i* ,λ k,i Q k,i* And its corresponding importance score is denoted as S. k,i The parameter set is represented as Training costs recorded Ultimately, the overall training objective is expressed as:

[0032]

[0033] Where γ>0 represents the regularization coefficient.

[0034] Step 4-4, update using stochastic gradient descent in step t. and Where k = 1, ..., n, for the parameter Its update process is represented as follows:

[0035]

[0036] Where the learning rate η>0,

[0037] Steps 4-5, next, based on importance scores Singular values ​​can be pruned as follows:

[0038]

[0039] in It includes importance scores for all triples, where k is an optional parameter. Simultaneously, it uses the magnitude of singular values ​​to quantify the importance of each triple, i.e., S. k,i =|λ k,i The above methods can be used to train the model and generate time series data.

[0040] A computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the steps of the virtual time-series data generation method based on large language model fine-tuning.

[0041] An electronic device, characterized in that it comprises:

[0042] processor;

[0043] A memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the virtual time-series data generation method based on large language model fine-tuning.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] (1) Introducing frequency domain trends as generation guidance, the time-series data is transformed to the frequency domain through Fourier transform, allowing explicit control over the smoothness and continuity of the generated sequence, thus enhancing its continuity and rationality. Traditional time-series generation models are prone to numerical abrupt changes due to local optimization. Frequency domain guidance, through global constraints, avoids point-to-point local overfitting, thereby reducing non-physical jumps. For example...Figure 3 As shown, the generation method that incorporates frequency domain trends generates data that is closer to the true values, while other time series generation models are prone to jump phenomena, generating data that does not match the true values.

[0046] (2) Leveraging the semantic modeling and contextual association capabilities of pre-trained large-scale models, the generated time-series data exhibits greater consistency and realism in numerical distribution and evolutionary logic. Pre-trained large-scale models learn general patterns from massive amounts of data, such as trend continuity and responses to sudden events, which can be transferred to specific time-series generation tasks, ensuring consistency in numerical distribution and realism in evolutionary logic. Through attention mechanisms, large language models can model long-term dependencies, thereby making reasonable inferences about the data to be generated based on richer observation samples. Compared to statistical models relying on linear assumptions and end-to-end models lacking prior knowledge from pre-training, methods based on pre-trained large-scale models can generate data that more closely resembles real-world data patterns. Figure 4 As shown, the method based on pre-trained large models can quickly and proactively respond to data changes, while other comparative methods cannot generate reasonable data.

[0047] (3) In the time series data generation task, this model achieved excellent interpolation results, and the specific experimental results and related indicators can be obtained from Table 1. By combining a pre-trained large model and a frequency domain decomposition-based analysis method, this model provides a new solution for virtual time series data generation. Compared with existing methods, the frequency domain-based Top-k high-frequency component extraction can adaptively process multi-periodic factors in the data pattern and actively focus on significant periodic patterns in the time series, providing a basis for subsequent data processing. The frequency domain analysis given by this method and the self-attention mechanism of the pre-trained large model can form a positive coupling, thereby generating excellent virtual time series data.

[0048] (4) The generated data can be widely used in downstream tasks such as model training, anomaly detection, and long-term prediction, and has good engineering scalability and research value. Based on the pre-trained large model method used in this model, relatively reasonable imputation data can be generated, and the data generated by this model can be compared with the real values ​​that may produce anomalies. In addition, since the self-attention calculation method used in this model can consider the long-term correlation in the data, it is only necessary to change the mask matrix and the training comparison target to be consistent with the prediction task, and this model can be directly applied to the prediction task. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method of the present invention.

[0050] Figure 2A schematic diagram of the virtual time-series data generation method based on large language model fine-tuning provided by the present invention.

[0051] Figure 3 Visualization of the timing generation effect provided by this invention Figure 1 .

[0052] Figure 4 Visualization of the timing generation effect provided by this invention Figure 2 . Detailed Implementation

[0053] The technical solutions provided by the present invention will be described in detail below with reference to specific implementations. It should be understood that the following specific implementations are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0054] Example:

[0055] Experimental conditions: All experiments were implemented using PyTorch on an NVIDIA RTX A6000 48G GPU.

[0056] Experimental parameter settings: During training, the historical window size was set to 96. The sampling interval was set to 2, and the kernel size of the moving average was set to 3. The model was trained using L2 loss and optimized using the Adam optimizer. The initial learning rate was 0.05, the batch size was 32, the number of training epochs was 10, and an early stopping strategy was used.

[0057] This paper conducts generative experiments on the Power Transformer Dataset (ETT) and compares the performance of our model with seven state-of-the-art benchmark models from 2021 to 2024. The results are shown in Table 1. In the table, the input length of all datasets is I with values ​​of {96, 192, 336, 720} and a missing value rate of 10%. Missing parts of the input are generated. Mean squared error (MSE) and mean absolute error (MAE) are used as evaluation metrics; lower values ​​indicate better model generation performance. The best results are highlighted in bold, and the second-best results are highlighted with an underline. As shown in Table 1, our model outperforms almost all Transformer-based models on this dataset.

[0058] Table 1 Comparison of Model Generation Performance

[0059]

[0060]

[0061] like Figure 3 As shown, at the time steps requiring interpolation, the data generated by this method is more accurate than that of other methods, and there are no abnormal phenomena such as abrupt changes in interpolated values ​​and large deviations from the true values.

[0062] like Figure 4 As shown, during the process of data evolving from a gradual to abrupt changes, this method can respond to data changes in a timely manner and provide more reasonable generation results, while the interpolation of other models fits the peak of the changes poorly.

[0063] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A method for generating virtual time-series data based on fine-tuning of a large language model, characterized in that, The method includes the following steps: Step 1, given time series X t ={x1,x2,…,x N Alignment is performed for instance normalization to eliminate dimensional differences and preserve the statistical properties of the sequence, and labeled sequence blocks are formed through block operations. Step 2: Perform a Fourier transform on the marked sequence block and extract the high-frequency components. Then, use an inverse Fourier transform to form the trend periodic components. Step 3, through the marked sequence blocks Additive decomposition is performed to obtain trend components with time-series characteristics. Based on these trend components, prompts are generated and fed into the language model along with the time-series data for further processing. Step 4: Fine-tune the large language model for the task using adaptive pruning.

2. The virtual time-series data generation method based on large language model fine-tuning according to claim 1, characterized in that, In step 1, the data processing procedure is as follows: Step 1: Treat the original time series as the sequence to be processed, X∈R. T×N×d , where X t ={x1,x2,…,x N } represents an input time series of length N at time step t∈T, given the original sequence X, and the mask set. Aimed at predicting the value X of the mask. mask For each time series data to be processed, the method first performs preprocessing on the data, including instance normalization and sequence segmentation. The entire process includes the following sub-steps: Step 1-1: First, apply Reverse Instance Normalization to X, calculate the mean μ and standard deviation σ of the sequence using the formula... Standardization is performed to eliminate dimensional differences and preserve the statistical properties of the series. Steps 1-2, followed by the normalized sequence The sequence is divided into fixed-length time patches. Each patch aggregates P adjacent time steps to form a labeled sequence. in This step reduces sequence redundancy and expands the input time range, thereby effectively capturing local patterns in the time series.

3. The virtual time-series data generation method based on large language model fine-tuning according to claim 1, characterized in that, In step 2, the periodic signal extraction process is as follows: Step 2-1, for the labeled sequence Perform a Fast Fourier Transform (FFT) to obtain its frequency domain representation. The mathematical formula for FFT is: in It is the complex frequency domain component corresponding to frequency k at time step t, where j represents the imaginary unit, satisfying j 2 =-1, It is a complex exponential function. Step 2-2, for frequency domain representation Consider the top-k largest high-frequency components, denoted as X'[k], and use the inverse Fourier transform to obtain the periodic signal:

4. The virtual time-series data generation method based on large language model fine-tuning according to claim 1, characterized in that, In step 3, the process of constructing the Prompt and large model input is as follows: Step 3-1, use the formula For the labeled sequence Decompose, where S t T represents a periodic component. t Indicating the trend component, δ t Represents noise components. Step 3-2, next use a multilayer perceptron (MLP) to analyze T. t Perform a linear transformation to compress its dimensions and generate a low-dimensional Prompt, making the trend component T... t Better integration with the input of the large language model: X Prompt =MLP(T t ) (3) Step 3-3, X Prompt With serialized tag blocks The spliced ​​data is then fed into a larger model for representation:

5. The virtual time-series data generation method based on large language model fine-tuning according to claim 1, characterized in that, In step 4, the serialized features are input into the large model for alignment and task fine-tuning. The specific process is as follows: Step 4-1: Fine-tune the large model parameters using incremental updates of the pre-trained weight matrix in the form of singular value decomposition. W = W + Δ = W + PΛQ (5) and These represent the left and right eigenvectors of Δ, and the diagonal matrix, respectively. Includes singular values ​​{λ i } 1≤i≤r And r << min(d1, d2), Step 4-2: P and Q are initialized using random Gaussian initialization. Furthermore, to ensure the orthogonality of P and Q, i.e. Add the following regular expression: in Represents the identity matrix. Step 4-3: Apply the adaptive process based on Singular Value Decomposition (SVD) (Equation 5) to the weight matrix W of each large model transformer layer. q W k W v W f1 and W f2 And use k to index the increment matrix, i.e., Δ. k =P k Λ k Q k Where k = 1, ..., n, n represents the number of adaptation weight matrices, and the matrix includes the k-th increment matrix Δ. k The triplet of the i-th eigenvalue and eigenvector is represented as G. k,i ={P k,i* ,λ k,i Q k,i* And its corresponding importance score is denoted as S. k,i The parameter set is represented as Training costs recorded Ultimately, the overall training objective is expressed as: Where γ>0 represents the regularization coefficient. Step 4-4, update using stochastic gradient descent in step t. and Where k = 1, ..., n, for the parameter Its update process is represented as follows: Where the learning rate η>0, Steps 4-5, next, based on importance scores Singular values ​​can be pruned as follows: in It includes importance scores for all triples, where k is an optional parameter. Simultaneously, it uses the magnitude of singular values ​​to quantify the importance of each triple, i.e., S. k,i =|λ k,i The above methods can be used to train the model and generate time series data.

6. A computer-readable storage medium storing computer instructions thereon, characterized in that: When executed by a processor, the computer instructions implement the steps of the virtual time-series data generation method based on large language model fine-tuning as described in any one of claims 1-5.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the virtual time-series data generation method based on large language model fine-tuning as described in any one of claims 1 to 5.

Images