A dataset distillation method suitable for time series base model fine-tuning
By employing a dataset distillation method based on multi-scale trend and periodic term decomposition, the computational overhead and privacy protection issues in fine-tuning of time series baseline models are addressed. The resulting synthetic dataset is suitable for edge devices, reducing resource requirements while ensuring performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2026-03-03
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies suffer from high computational overhead, large resource requirements, difficulty in privacy protection, and inapplicability to traditional dataset distillation methods in fine-tuning of time series basic models, especially on edge devices.
A dataset distillation method using multi-scale trend and periodic term decomposition is employed, performing gradient matching only at the linear output layer to generate a synthetic dataset for fine-tuning the time series base model, reducing computational overhead and protecting privacy.
The generated synthetic dataset can effectively reduce computational overhead, ensure performance, is suitable for edge devices, and provides good privacy protection.
Smart Images
Figure CN122175040A_ABST
Abstract
Description
Technical Field
[0001] This invention is a dataset distillation method suitable for fine-tuning time series basic models, and relates to the fields of deep learning technology and data processing technology. Background Technology
[0002] In the fields of natural language processing and computer vision research, many foundational models have learned general feature representation capabilities through pre-training on massive amounts of data, demonstrating strong generalization performance and breaking the limitation of traditional deep learning where a new model needs to be retrained for each scenario. This development direction has also profoundly influenced the analysis and processing of time series data in recent years, giving rise to a number of foundational time series models. These models, by learning general time series features, can effectively solve various downstream tasks such as prediction, classification, interpolation, and anomaly detection. However, when applying these foundational models with large numbers of parameters to specific vertical fields, such as medical electrocardiogram monitoring, traffic prediction, and financial analysis, fine-tuning on real datasets in the relevant fields is still necessary to achieve better performance. This presents two challenges. On the one hand, although traditional fine-tuning methods only focus on updating the final linear layer, they still require a complete real dataset for training. In many fields, high-quality labeled data is extremely scarce, and the data often involves strict privacy issues, making it difficult to fully utilize. On the other hand, the number of parameters in foundational models is usually in the hundreds of millions, and fine-tuning on real datasets requires extremely high computing power and storage, making it difficult to deploy on edge devices or resource-constrained servers.
[0003] Dataset distillation is a data processing method designed to reduce model training overhead. Its core objective is to synthesize a small dataset so that the model's performance after training solely on the synthetic dataset is comparable to its performance on the original dataset, while significantly reducing training costs. Furthermore, since the synthetic dataset in dataset distillation is entirely synthesized, there are no privacy concerns. Currently, dataset distillation methods primarily aim to match the impact of the original and synthetic datasets on model training as closely as possible, such as generating similar gradients or mapping the model to similar vector spaces. However, directly applying dataset distillation to fine-tuning time-series models presents three problems: First, these distillation methods themselves have enormous computational overhead; for example, parameter matching methods require matching on hundreds of millions of model parameters, which is non-negligible in many scenarios. Second, dataset distillation methods are mainly designed for training from scratch, therefore, they are not entirely suitable for fine-tuning basic models. Finally, current research on dataset distillation primarily focuses on computer vision, and transferring this technique to time-series models presents numerous technical challenges.
[0004] Chinese Patent Application No. 202511113194.0 discloses "A Contrastive Distribution Matching Dataset Distillation Method for Time-Series Electricity Data," which aims to address the problem of poor analytical performance of traditional dataset distillation methods for time-series electricity data during the training of new models. However, its application scenario and the contrastive learning and distribution matching techniques employed differ from this method, and it is not a solution for fine-tuning the basic model.
[0005] Chinese Patent Application No. 202411326206.3 discloses "a method, electronic device and medium for data distillation", which uses the kernel ridge regression method to capture complex relationships in data. This method is not a solution for time series data, nor does it aim to fine-tune the basic model to synthesize the dataset.
[0006] Chinese Patent Application No. 202510985923.5 discloses a "data distillation method, device and electronic device for photovoltaic power generation prediction". The method determines the matching loss function based on the multi-scale time domain features and frequency domain features corresponding to the original dataset and the synthetic dataset. This method is aimed at the prediction task, rather than the basic model and its fine-tuning task. Summary of the Invention
[0007] Objective of the Invention: To address the problems and shortcomings of existing technologies, the objective of this invention is to provide a dataset distillation method suitable for fine-tuning time series base models. This method decomposes the time series in a synthetic dataset into multi-scale trend and periodic terms, thereby better extracting features from the original dataset and improving the optimization effect on the synthetic dataset. Furthermore, this method does not attempt to perform matched distillation on the complete base model, but only on the final linear output layer. This not only adapts to the fine-tuning of the base model but also significantly reduces the computational overhead during the distillation process. The synthetic dataset obtained through this method can significantly reduce the fine-tuning cost of migrating the time series base model to downstream task models, while ensuring data privacy.
[0008] Technical Solution: To achieve the above-mentioned objectives, the technical solution adopted by this invention is a dataset distillation method suitable for fine-tuning time series basic models, comprising the following steps:
[0009] (1.1) Load the original time series dataset and the first time series basic model of the target domain, and initialize the first multi-scale parameter set; (1.2) Construct a linear layer and form a second time series model with the feature extractor extracted from the first time series basic model. Calculate the first multi-scale parameter set to obtain a first synthetic dataset composed of time series data consisting of a mixture of trend terms and periodic terms at different scales. Then use the original time series dataset and the first synthetic dataset as inputs to the second time series model to obtain preliminary output results. (1.3) Calculate the loss value based on the preliminary output results, optimize the first multi-scale parameter set based on the gradient backpropagation of the loss value, and calculate based on the optimized second multi-scale parameter set to obtain the final second synthetic dataset. After that, the second synthetic dataset can be used to fine-tune the time series basic model.
[0010] Further, step (1.1) includes the following steps: (2.1) Load the original time series dataset of the target domain; then normalize the original time series dataset; (2.2) Load the pre-trained first time series base model; then extract the feature extractor from the first time series base model; (2.3) Initialize the first multiscale parameter set based on the set scale, where each multiscale parameter consists of two parts: a randomly generated trend term and a period term; then initialize the scale level of each multiscale parameter based on the set scale range.
[0011] Further, step (1.2) includes the following steps: (3.1) Freeze the parameter weights of the feature extractor; then construct a linear layer using random parameter weights to receive the output of the feature extractor and produce results, thereby forming a second time series model together with the feature extractor; (3.2) Calculate each parameter in the first multi-scale parameter set, mix the multi-scale trend term and periodic term to obtain the corresponding time series data; the time series data corresponding to all parameters are combined to form the first synthetic dataset; (3.3) Perform the same differentiable data augmentation operation on the time series data in the original time series dataset and the first synthetic dataset respectively; input the augmented data into the second time series model to obtain the preliminary output results.
[0012] Further, step (1.3) includes the following steps: (4.1) Calculate the gradients of the initial output results of the original time series dataset and the first synthetic dataset with respect to the linear layer parameter weights, respectively; calculate the loss value based on the first gradient generated from the original time series dataset and the second gradient generated from the first synthetic dataset; (4.2) Calculate the meta-gradient of the loss value with respect to the first multi-scale parameter set; and optimize the first multi-scale parameter set using the meta-gradient; (4.3) Based on the optimized second multi-scale parameter set, each parameter is calculated to obtain the corresponding time series data; the time series data corresponding to all parameters are combined to form the final second synthetic dataset, which can then be used to fine-tune the training of the time series basic model.
[0013] Beneficial effects: (1) This invention uses multi-scale trend and periodic terms as generation constraints for time series in the synthetic dataset, which can better preserve and extract the features of the time series in the original dataset; (2) This invention only uses the gradient of the last linear output layer as the matching object, avoiding parameter matching on the entire base model, which greatly reduces the computational cost; (3) The synthetic dataset obtained by this invention does not come directly from the original dataset, but from the optimization process, which can better protect privacy attributes; (4) The synthetic dataset obtained by this invention can guarantee that it has performance comparable to the original dataset when fine-tuning the base model, and the required computational cost is much less than that of the original dataset. Using the technical solution of this invention, engineers can easily implement the relevant software. Attached Figure Description
[0014] Figure 1 A flowchart of a dataset distillation method suitable for fine-tuning a basic time series model; Figure 2 This is a schematic diagram of the multi-scale trend term-period term decomposition of a time series. Detailed Implementation
[0015] This invention provides a dataset distillation method suitable for fine-tuning time series basic models. This method solves the computational overhead problem in fine-tuning time series basic models by using multi-scale trend-period term decomposition and linear layer gradient matching strategy.
[0016] The complete process of this invention is as follows: Figure 1 As shown, it includes three steps: First, obtain the original time series dataset of the downstream scene and the pre-trained first time series base model, and initialize the first multi-scale parameter set; then, optimize the first multi-scale parameter set based on the feature extractor of the first time series base model and the original dataset to obtain the second multi-scale parameter set; finally, calculate the second synthetic dataset based on the second multi-scale parameter set and use it for fine-tuning the base model.
[0017] The following will illustrate the implementation of the present invention step by step with examples: 1. Obtain the original time series dataset and the first time series basic model, and initialize the first multi-scale parameter set.
[0018] 1.1 Loading the original time-series dataset from the downstream scenario Suppose that it contains 10 time series data points, each data point The format satisfies ,in The length of the time series. Let be the number of channels in the time series data; then, perform Z-Standard normalization on the dataset so that the mean of the numerical distribution in the dataset is 0 and the variance is 1.
[0019] 1.2 Load the pre-trained first time series base model and obtain its feature extractor. For any input time series This feature extractor can output a general vector representation of the data. .
[0020] 1.3. Initialize the first multiscale parameter set.
[0021] 1.3.1 Let the size of the first multi-scale parameter set be set as follows: Then the first multi-scale parameter set It can be represented as Each of the multiscale parameters All of them correspond to time series data in a first synthetic dataset.
[0022] 1.3.2 To ensure that the data in the distilled synthetic dataset better preserves the time series features of the original dataset, each multi-scale parameter... What is stored is not the time series itself, but the trend term. and periodic terms ,like Figure 2 As shown, therefore the first multi-scale parameter set It can be further expressed as By adopting this decomposition method, the trend term and the periodic term can be optimized separately, and the time series calculated by combining the trend term and the periodic term will be more consistent with the characteristics of time series.
[0023] 1.3.3 Initialize the scale hierarchy of each multi-scale parameter according to the set scale range. Specifically, let the maximum scale hierarchy be set to... For each of its layers The current layer's scale is In other words, the span of each point in the time series at the current layer is equivalent to the length of the complete time series. A point, such as Figure 2 As shown. For example, in , In this case, the scales of each layer are 1, 2, 4, and 8, respectively, corresponding to time series resolutions of 512, 256, 128, and 64. Therefore, the earlier the layer, the lower the scale and the higher the resolution, which can better capture the specific details in the time series; the later the layer, the higher the scale and the lower the resolution, which can better capture the coarse structural information in the time series.
[0024] 2. Construct a second time series model. Calculate the first synthetic dataset based on the first multi-scale parameter set. Use both the first synthetic dataset and the original dataset as inputs to the second time series model to obtain preliminary output results.
[0025] 2.1 Constructing a second time series model based on the feature extractor of the pre-trained first time series base model. Specifically, the feature extractor... Input time series data Mapping to a universal vector representation Then set up a linear layer. Used to map the vector representation of data to results Its parameter weights are derived from a standard normal distribution. Randomly generated, i.e. .
[0026] 2.2 For each parameter in the first multi-scale parameter set Through calculation, all its scales Trend items below Periodic terms Restore to the corresponding time series This is so that it can be used as input to the second time series model. The relevant formula is:
[0027] in The activation function is used to non-linearly map the numerical values of a time series to... Within the range; For upsampling operations, interpolation can be performed on time series that are at a higher scale, lower resolution, and less detailed to map them to a full-length time series.
[0028] Finally, all multi-scale parameters Corresponding time series Combined together, we obtain the first synthetic dataset. .
[0029] 2.3 To prevent the synthetic dataset from easily overfitting, data augmentation, a common improvement measure, needs to be introduced. Specifically, when performing data augmentation on the original dataset and the first synthetic dataset, it should be ensured that the same data augmentation operation is used on both. This allows the synthetic dataset to learn features from the original dataset from more perspectives, rather than fitting to the augmented data. In addition, to ensure that the meta-gradient can be backpropagated to the first multi-scale parameter set through the data augmentation layer, the data augmentation operation used here must be differentiable.
[0030] Currently, various differentiable data augmentation operations can be used on time series data, such as simple data perturbation, like adding Gaussian noise to the time series; or simple data scaling, like multiplying the values in the time series by a random scaling factor; in addition, perturbation can also be performed in the frequency domain. First, the time series data is converted to frequency domain data through Fast Fourier Transform, and then perturbation is performed in the frequency domain, such as randomly masking some frequency components, and then the time series data is restored through Inverse Fast Fourier Transform.
[0031] Let the original dataset after differentiable data augmentation be... The corresponding expected output is The first synthetic dataset is... The corresponding expected output is .
[0032] 3. Calculate the loss value, optimize the first multi-scale parameter set, and obtain the final second synthetic dataset.
[0033] 3.1 Setting the loss function This is used to calculate the preliminary output results of the second time series model. With data reality label Loss values between Then, the gradients of the loss values of the original dataset and the first synthetic dataset relative to the linear layer are calculated respectively; that is, the gradients are backpropagated to the linear layer using the loss values. The relevant formula is:
[0034]
[0035] in The gradient of the result of the original dataset with respect to the linear layer. This represents the gradient of the result for the first synthetic dataset relative to the linear layer. In time series classification tasks, the loss function can be cross-entropy loss; in time series prediction or interpolation tasks, the loss function can be mean squared error loss.
[0036] 3.2 Gradients generated from the original dataset and gradients generated from the first synthetic dataset The loss value used to update the first multi-scale parameter set is calculated using cosine distance. The relevant formula is:
[0037] Based on this loss value and the meta-gradient of the first multi-scale parameter set. The first multi-scale parameter set is updated using gradient descent. For the second multi-scale parameter set The relevant formula is:
[0038] in The specified learning rate.
[0039] 3.3. Based on the second multi-scale parameter set obtained through optimization For each of the parameters The corresponding time series was calculated using the same method as described above. These time series data were combined to form the final second synthetic dataset. When fine-tuning the underlying time series model, a second synthetic dataset is used instead of the original dataset.
[0040] The above description details a dataset distillation method suitable for fine-tuning time series fundamental models, which will be understood and implemented by those skilled in the art. Various modifications in form and detail can be made to this method without departing from the scope of the invention. Therefore, all suggested but not limited modifications described above are within the scope defined by the appended claims.
Claims
1. A dataset distillation method suitable for fine-tuning a time series basic model, characterized in that, Includes the following steps: (1.1) Load the original time series dataset and the first time series basic model of the target domain, and initialize the first multi-scale parameter set; (1.2) Construct a linear layer and form a second time series model with the feature extractor extracted from the first time series basic model. Calculate the first multi-scale parameter set to obtain a first synthetic dataset composed of time series data consisting of a mixture of trend terms and periodic terms at different scales. Then use the original time series dataset and the first synthetic dataset as inputs to the second time series model to obtain preliminary output results. (1.3) Calculate the loss value based on the preliminary output results, optimize the first multi-scale parameter set based on the gradient backpropagation of the loss value, and calculate based on the optimized second multi-scale parameter set to obtain the final second synthetic dataset. After that, the second synthetic dataset can be used to fine-tune the time series basic model.
2. The dataset distillation method for fine-tuning time series basic models according to claim 1, characterized in that, Step (1.1) includes the following steps: (2.1) Load the original time series dataset of the target domain; then normalize the original time series dataset; (2.2) Load the pre-trained first time series base model; then extract the feature extractor from the first time series base model; (2.3) Initialize the first multiscale parameter set based on the set scale, where each multiscale parameter consists of two parts: a randomly generated trend term and a period term; then initialize the scale level of each multiscale parameter based on the set scale range.
3. The dataset distillation method for fine-tuning time series basic models according to claim 1, characterized in that, Step (1.2) includes the following steps: (3.1) Freeze the parameter weights of the feature extractor; then construct a linear layer using random parameter weights to receive the output of the feature extractor and produce results, thereby forming a second time series model together with the feature extractor; (3.2) Calculate each parameter in the first multi-scale parameter set, mix the multi-scale trend term and periodic term to obtain the corresponding time series data; the time series data corresponding to all parameters are combined to form the first synthetic dataset; (3.3) Perform the same differentiable data augmentation operation on the time series data in the original time series dataset and the first synthetic dataset respectively; input the augmented data into the second time series model to obtain the preliminary output results.
4. The dataset distillation method for fine-tuning time series basic models according to claim 1, characterized in that, Step (1.3) includes the following steps: (4.1) Calculate the gradients of the initial output results of the original time series dataset and the first synthetic dataset with respect to the linear layer parameter weights, respectively; calculate the loss value based on the first gradient generated from the original time series dataset and the second gradient generated from the first synthetic dataset; (4.2) Calculate the meta-gradient of the loss value with respect to the first multi-scale parameter set; and optimize the first multi-scale parameter set using the meta-gradient; (4.3) Based on the optimized second multi-scale parameter set, each parameter is calculated to obtain the corresponding time series data; the time series data corresponding to all parameters are combined to form the final second synthetic dataset, which can then be used to fine-tune the training of the time series basic model.