Large time series prediction model for continuous learning

Through fine-tuning of multi-classification model and LoRA module, combined with weighted classification losses, the prediction difficulties and forgetting problems of time series large models in complex data environments are solved, and the continuous accumulation and rapid adaptation of knowledge are achieved, and the adaptability of the model is improved. Error tolerance.

CN119004018AInactive Publication Date: 2024-11-22SHANGHAI DINGMAO INFORMATION TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411030914.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing time series models are difficult to achieve effective predictions when facing complex and diverse data, and traditional continuous learning methods will lead to catastrophic forgetting of historical knowledge and cannot quickly adapt to new data changes.

Method used

The multi-classification model is used to classify the newly added data, fine-tune the LoRA module of the corresponding category, and align it with the time series data through the frozen large model, use the LoRA module with the highest similarity to make predictions, and design weighted classification losses to alleviate the impact of classification errors. Only update the classifier without retraining the entire model.

Benefits of technology

It realizes continuous accumulation of knowledge and improves adaptability, reduces computing resources and time costs, avoids forgetting problems, and improves the fault tolerance and prediction performance of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119004018A_ABST
    Figure CN119004018A_ABST
Patent Text Reader

Abstract

The invention provides a continuous learning time series prediction large model. The continuous learning time series prediction large model comprises the following steps: step 1, a classification stage; 2, a prediction stage; for each category, performing fine tuning on the time sequence large model by using the category data to obtain a LoRA module of each category, performing incremental training on a newly added similar data set in the corresponding category, performing reasoning prediction, and then re-initializing one LoRA module for a data set which does not belong to any known category for training; or; after the large model is aligned with time sequence data, potential representation of an input time sequence is obtained by using the frozen large model, each LoRA matrix is represented by using a vector, the representation is matched with LoRA representation, and the LoRA with the highest similarity is selected for a subsequent prediction task; the knowledge can be continuously accumulated, both new data and historical data can be fully utilized and optimized in the model, and the problem that a traditional continuous learning method generates disastrous forgetting on historical knowledge is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of continuous learning, and more specifically, particularly relates to a continuous learning time series prediction large model. Background Art

[0002] With the emergence of large models, time series prediction has entered a new stage. At present, the combination of large models and time series data has become a hot topic of research. Current time series prediction methods usually require rich domain knowledge. In contrast, LLM has been proven to have significant capabilities in few-shot and zero-shot transfer learning. It does not require the design and training of a model for each type of data set and each type of task, making time series prediction more universal and efficient. The current mainstream approach is PEFT, which fine-tunes the pre-trained LLM. In the existing PEFT technology, LoRA can significantly reduce the amount of saved parameters and can build different LoRA blocks for different tasks. By replacing different LoRAs, LLM can adapt to different tasks. Therefore, LoRA is usually used to achieve alignment with time series tasks. However, these methods are still studying how LLM can be effectively combined with time series data and fine-tuning and verifying a single data set. However, in actual scenarios, the types of time series data are complex and diverse, and it is impractical to fine-tune a model for each data set. Therefore, it is crucial to enable the model to have the ability to learn continuously.

[0003] Due to the diversity and complexity of data, time series data from different fields and sources vary in frequency, periodicity, trend, and noise characteristics, making it extremely difficult to design a universal prediction model. Although large models have certain zero-sample capabilities, it is difficult to fully perform well in the face of actual data of varying types. In zero-sample scenarios, they can only achieve limited improvements over ordinary deep learning models.

[0004] In addition, in practical applications, time series data is continuously generated, and retraining the large model requires a lot of computing resources and time. Therefore, the large model needs to adapt to new data changes and continue to learn, rather than just training on static historical data sets. This requires the large model to be able to dynamically learn from new data and adapt quickly, while avoiding forgetting old knowledge.

[0005] Existing continuous learning methods, such as O-LoRA, can only provide limited relief but cannot avoid forgetting when applied to large time series model scenarios.

[0006] Due to the above challenges, existing large time series models are difficult to effectively predict time series data in different scenarios. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a large model of time series prediction for continuous learning to solve the problem in the prior art that traditional continuous learning methods will cause catastrophic forgetting of historical knowledge. O-LoRA uses the orthogonal method of spatial vectors to alleviate catastrophic forgetting, but adds restrictions to the parameter search space of new tasks, resulting in the inability to achieve optimal performance on new tasks.

[0008] The purpose and efficacy of the continuous learning time series prediction large model of the present invention are achieved by the following specific technical means:

[0009] A time series prediction model with continuous learning includes the following steps:

[0010] Step 1: Classification stage;

[0011] Use the existing data set to train a multi-classification model, classify the newly added data set, determine the category it belongs to, and use the existing data set and the new data set to train a new classification model for those that do not belong to the existing known categories, and integrate it with the old classification model;

[0012] Step 2: Prediction stage;

[0013] For each category, the time series large model is fine-tuned using the data of that category to obtain the LoRA module of each category. The newly added similar data sets are incrementally trained in the corresponding category for inference prediction, and then the data sets that do not belong to any known category are used to reinitialize a LoRA module for training;

[0014] Alternatively, after aligning the large model with the time series data, the frozen large model is used to obtain the potential representation of the input time series. At the same time, each LoRA matrix is ​​represented by a vector, which is matched with the LoRA representation, and the LoRA with the highest similarity is selected for subsequent prediction tasks.

[0015] In a preferred embodiment, continuous learning is achieved by pre-classifying the module and training or incrementally training a LoRA module for each category, and a new weighted classification loss is designed to alleviate the impact of the accuracy of the classification stage on subsequent prediction tasks.

[0016] In a preferred embodiment, the classification model can adopt statistical methods, machine learning or deep learning classification methods, train multiple classification models based on existing data, determine the category of the new data set, and after the training is completed, it is manifested as an integration of multiple classification models.

[0017] In a preferred embodiment, in the weighted classification loss, the penalty for similar class classification errors is small, and the penalty for dissimilar class classification errors is large, thereby reducing the impact of the two-stage cumulative error on the prediction result.

[0018]

[0019] Where w represents the weight value corresponding to the similarity class, and its value range is (0,1).

[0020] In a preferred embodiment, in step 2, the inference prediction process is: first classify the input data, and then select the corresponding LoRA module and combine it with the time series large model to perform the prediction task.

[0021] In a preferred embodiment, the prediction stage includes a prediction model, which uses LLM as the backbone network. During pre-training, the prediction head layer and embedding layer of LLM are aligned with the time series. Subsequent training only uses LoRA to fine-tune the attention layer, while freezing other modules to ensure prediction capabilities and reduce storage costs.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] 1. Through the pre-classification module and the training or incremental training of the LoRA module for each category, continuous learning is achieved so that knowledge can be accumulated continuously. Both new data and historical data can be fully utilized and optimized in the model, solving the problem that traditional continuous learning methods will cause catastrophic forgetting of historical knowledge, thereby improving the overall performance and adaptability of the model.

[0024] 2. In addition to the LoRA parameter matrix used for fine-tuning, the model only needs to update the classifier during the continuous learning process, without retraining the entire model. This solves the problem of difficult and costly model update and maintenance in traditional continuous learning methods as data continues to grow and change. It not only greatly saves computing resources and time costs, but also improves the efficiency and flexibility of model updates, allowing the model to quickly adapt to new data patterns and structures and maintain good predictive capabilities.

[0025] 3. Through the innovative weighted classification loss strategy, the impact of data misclassification on subsequent prediction performance is effectively reduced. The weighted classification loss can reduce the impact of misclassification. When data is misclassified, it is more likely to be classified into similar categories, thereby reducing the negative impact of the first stage classification accuracy on the second stage in the two-stage process, thereby improving the fault tolerance of the model, so that the model can still maintain stable prediction performance when facing complex and changing data environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a large model of time series prediction with continuous learning according to the present invention;

[0027] Figure 2It is a principle diagram of a classification model in a large model of time series prediction for continuous learning of the present invention;

[0028] Figure 3 It is a principle diagram of a prediction model in a large time series prediction model for continuous learning of the present invention. DETAILED DESCRIPTION

[0029] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with the accompanying drawings and examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0030] Example:

[0031] Please refer to Figures 1 to 3 As shown, the present invention provides a time series prediction large model for continuous learning, including the following steps:

[0032] Step 1: Classification stage;

[0033] First, use the existing data set to train a multi-classification model. Then, input the newly added data set into the trained classification model to determine the category to which each new sample belongs. For those new samples that do not belong to the existing known categories, use them together with the existing data set to retrain an updated classification model. This new classification model not only contains the original category knowledge, but also absorbs the category information in the newly added data set. Finally, integrate this updated classification model with the previous old classification model to form a more comprehensive and robust classification system, paving the way for the subsequent prediction stage. In this way, the method can continuously absorb new category information, continuously expand the model's cognitive ability, and provide a richer and more accurate foundation for subsequent prediction tasks.

[0034] Step 2: Prediction stage;

[0035] For each identified category, the time series model is fine-tuned using the data set of that category to obtain a LoRA module specific to that category. Then, the newly added similar data set is input into the corresponding LoRA module for incremental training, so that the model can perform effective time series prediction for the new data set. For those data sets that do not belong to any known category, a LoRA module needs to be reinitialized and trained independently. The purpose of this is to allow the model to continuously absorb new knowledge and expand its prediction capabilities without being overly restricted by existing category information. During the entire prediction process, the model can use the corresponding LoRA module to perform targeted time series predictions based on the characteristics of different categories, thereby improving the overall prediction accuracy, enabling the model to better cope with dynamically changing time series data and continuously improve the performance of the model.

[0036] Alternatively, by aligning a large language model with time series data, the model can fully tap into the potential feature representations contained in the input time series; specifically, the framework first uses the pre-trained LLM to extract the potential representation of the input data, and represents each LoRA fine-tuning module with a vector. Next, the model will select the LoRA module with the highest similarity to the feature representation of the input data for subsequent prediction tasks. This strategy based on similarity matching allows the model to only update the classifier part during continuous learning without retraining the entire network. This not only greatly reduces computing resources and time costs, but also greatly improves the efficiency and flexibility of model updates, enabling it to quickly adapt to new data patterns and structures and maintain good prediction performance. Compared with traditional continuous learning methods, this solution based on LLM and LoRA has significant advantages in coping with changing data environments, fully demonstrating the adaptability of the model.

[0037] This continuous learning method achieves effective knowledge absorption and transfer by training the LoRA module in the classification module and for each category or incrementally training the LoRA module; at the same time, this method also designs a new weighted classification loss function to alleviate the negative impact of the accuracy of the classification stage on subsequent prediction tasks. The original intention of designing this weighted classification loss function is that even if misclassification occurs in the classification stage, these erroneous samples can be more easily classified into similar categories, thereby reducing the negative connection between the two stages; that is, even if the classification accuracy of the first stage is not ideal, through this weighted loss design, it can also reduce its impact on the prediction performance of the second stage to a certain extent, thereby improving the fault tolerance of the model, so that even in the face of complex and changing data environments, the model can still maintain stable prediction performance.

[0038] In this continuous learning framework, the classification module used can be a classification model based on statistical methods, machine learning or deep learning. By training these classification models with existing data sets, an efficient multi-classifier can be obtained; in practical applications, when encountering a new data set, this classification module can judge it and determine the category to which the data set belongs. It is worth noting that this classification module is essentially an integration of multiple classification models, each of which learns and predicts different categories. This integration method can not only improve the accuracy of classification, but also enhance the generalization ability of the model, enabling it to better adapt to complex and changing data environments.

[0039] In the above continuous learning framework, the designed weighted classification loss function can effectively alleviate the negative impact of errors in the classification stage on subsequent prediction tasks; specifically, the weighted loss function will penalize classification errors to varying degrees according to the similarity of the categories to which the samples belong. For samples that are misclassified into similar categories, the loss penalty will be relatively small; while for samples that are misclassified into completely different categories, the loss penalty will be larger; this weighted mechanism is conducive to reducing the impact of the accumulated errors between the two stages, so that the model can better adapt to the complex and changing data environment, thereby improving the final prediction performance;

[0040]

[0041] In this weighted classification loss function, the weight parameter w represents the similarity between similar categories, and its value range is (0,1). Adjusting this parameter can flexibly control the penalty intensity of misclassification of different categories and further optimize the robustness and fault tolerance of the model.

[0042] In the second step, the process of inference and prediction reflects the organic combination of classification and prediction. First, the model will classify the input data and use the previously trained multi-classification model to determine the category to which the data belongs. Then, based on the classification results, the model will select the corresponding LoRA fine-tuning module and combine it with the pre-trained large time series model to complete the final prediction task. This process of classification first and then prediction makes full use of the capabilities of the classification model and can select appropriate LoRA modules for fine-tuning and reasoning for different types of data, greatly improving the adaptability and generalization ability of the model. At the same time, this division of labor and cooperation method also reduces the learning pressure of the large time series model and helps to improve its prediction accuracy.

[0043] The prediction stage includes a prediction model, which uses LLM as the backbone network. During pre-training, the prediction head layer and embedding layer of LLM are aligned with the time series. In subsequent training, only LoRA is used to fine-tune the attention layer, while freezing other modules to ensure prediction capabilities and reduce storage costs;

[0044] In the prediction stage of the continuous learning framework, a prediction model is adopted. The model uses a large language model (LLM) as the backbone network. By pre-training the prediction head layer and embedding layer of the LLM, effective alignment with the time series data is achieved. In the subsequent training process, the model only needs to use the LoRA technology to fine-tune the attention layer, while freezing other modules unchanged; it can not only give full play to the powerful feature extraction ability of LLM, but also maximize the prediction performance of the model, while also greatly reducing the model storage and computing overhead; through this selective fine-tuning strategy, the prediction model can quickly adapt to the new data environment, while ensuring the stability and reliability of the overall prediction ability; the prediction model based on LLM provides strong support for achieving the high-precision and low-cost goals of intelligent prediction.

[0045] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A large model for time series prediction with continuous learning, characterized by: The following steps are included: Step 1: Classification stage; Use the existing data set to train a multi-classification model, classify the newly added data set, determine the category it belongs to, and use the existing data set and the new data set to train a new classification model for those that do not belong to the existing known categories, and integrate it with the old classification model; Step 2: Prediction stage; For each category, the time series large model is fine-tuned using the data of that category to obtain the LoRA module of each category. The newly added similar data sets are incrementally trained in the corresponding category for inference prediction, and then the data sets that do not belong to any known category are used to reinitialize a LoRA module for training; Or; after aligning the large model with the time series data, use the frozen large model to obtain the potential representation of the input time series. At the same time, each LoRA matrix is ​​represented by a vector, and the representation is matched with the LoRA representation, and the LoRA with the highest similarity is selected for subsequent prediction tasks.

2. The time series prediction model of continuous learning according to claim 1 is characterized by: Through the pre-classification module, and training or incrementally training a LoRA module for each category, continuous learning is achieved, and a new weighted classification loss is designed to alleviate the impact of the accuracy of the classification stage on subsequent prediction tasks.

3. The time series prediction large model of continuous learning according to claim 2 is characterized by: The classification model can adopt statistical methods, machine learning or deep learning classification methods, train multiple classification models based on existing data, determine the category of the new data set, and after the training is completed, it will be expressed as an integration of multiple classification models.

4. The time series prediction large model of continuous learning according to claim 3 is characterized by: In the weighted classification loss, the penalty for similar classification errors is small, while the penalty for dissimilar classification errors is large, which reduces the impact of the two-stage cumulative error on the prediction results. Where w represents the weight value corresponding to the similarity class, and its value range is (0,1).

5. The time series prediction large model of continuous learning according to claim 1 is characterized by: In step 2, the inference prediction process is: first classify the input data, then select the corresponding LoRA module and combine it with the time series large model to perform the prediction task.

6. The time series prediction large model of continuous learning according to claim 5 is characterized by: The prediction stage includes a prediction model, which uses LLM as the backbone network. During pre-training, the prediction head layer and embedding layer of LLM are aligned with the time series. Subsequent training only uses LoRA to fine-tune the attention layer, while freezing other modules to ensure prediction capabilities and reduce storage costs.

Citation Information

Patent Citations

  • Loss value acquisition method and device for classification model

    CN108764283A

  • Image classification method based on improved cross entropy loss function

    CN116310542A

  • Model parameter adjustment method, device and equipment and computer readable storage medium

    CN116644791A

  • Medical service method, apparatus and device based on LLM intelligent agent architecture, and medium

    CN117112759A

  • Image classification pre-training model continuous learning method based on low-rank adaptive combination

    CN117611913A

Cited By

  • Battery state prediction method, device, equipment, medium and program product

    CN119199570A

  • Cardiovascular disease assessment management method and system based on big data

    CN119623479A

  • Natural gas demand prediction method based on GPT driving mode perception segmentation matching

    CN120146262A