Dual descent for time series prediction

Abstract

A "modern institution" training framework for a deep learning-based time series prediction model is described herein. Overall, the "Modern Institute" training framework includes configuring hyper-parameters of the training process in a manner that significantly increases the complexity of the time series prediction model beyond conventional criteria and enables the time series prediction model to achieve "dual descent" during the training process. Substantially, when the performance of the time series prediction model is allowed to deteriorate significantly after initial improvement and reaching local maximum performance and then improved again to reach performance beyond the initial local maximum performance, a "dual drop" occurs.

Description

Technical Field

[0001] The apparatus and methods disclosed herein relate to machine learning, and more particularly to training models for time series forecasting. Background Art

[0002] Unless otherwise indicated herein, the materials described in this section are not admitted to be prior art by inclusion in this section.

[0003] Deep learning models (especially Transformers) have achieved remarkable performance in many fields, including natural language processing (NLP), computer vision (CV), and time series forecasting. While existing research on time series forecasting has mainly focused on architectural modifications and data augmentation, recent advances in the NLP space have demonstrated the importance of innovating the training paradigm of deep learning models (i.e., innovating on how to train the model rather than how to mathematically construct the model).

[0004] Traditionally, it is believed that as model complexity increases, models become prone to overfitting, and as a result, generalization performance degrades. This can be referred to as the "conventional regime". However, deep double descent, a phenomenon that has recently regained popularity in deep learning, challenges the conventional understanding of the relationship between model complexity, training epochs, and generalization performance. In particular, the deep double descent phenomenon reveals more subtle patterns, where the generalization error first decreases, then increases, and then potentially decreases again as the model size and / or training epochs grow. The deep double descent phenomenon by epoch shows that extending the training process can lead to improved generalization performance even after the model appears to have begun to overfit the training dataset. This highlights the importance of revisiting the conventional understanding of generalization in deep learning and encourages further research into the role of model size, training epochs, and implicit regularization in improving the performance of deep learning models.

[0005] The deep double descent phenomenon has been the basis for many innovations in the NLP and CV fields and can be attributed to being a key factor in the trend towards larger and larger models. Training frameworks based on the double descent theory can be called "modern regimes". However, "modern regimes" have not yet been designed and applied to time series forecasting. Summary of the invention

[0006] A method for training a deep learning-based time series prediction model is described. The method includes: storing in a memory (i) a training data set including a first plurality of time series data and (ii) a test data set including a second plurality of time series data. The method also includes: using a processor to train the time series prediction model using the training data set. The time series prediction model has a plurality of parameters learned during training. The method also includes: periodically during training, using a processor to evaluate the performance of the time series prediction model on the test data set. The training is performed for a sufficient number of training epochs so that after initial improvement and reaching a local maximum performance, the performance of the time series prediction model is allowed to deteriorate, and then improved again to reach a performance that exceeds the local maximum performance.

[0007] A non-transitory computer-readable medium storing program instructions is disclosed, wherein the program instructions are used to train a time series prediction model based on deep learning. The program instructions, when executed by a processor, cause the processor to receive (i) a training data set including a first plurality of time series data and (ii) a test data set including a second plurality of time series data. The program instructions, when executed by the processor, cause the processor to use the training data set to train the time series prediction model. The time series prediction model has a plurality of parameters learned during training. The program instructions, when executed by the processor, also cause the processor to periodically evaluate the performance of the time series prediction model for the test data set during training. The training is performed for a sufficient number of training epochs so that after initial improvement and reaching a local maximum performance, the performance of the time series prediction model is allowed to deteriorate, and then improved again to reach a performance that exceeds the local maximum performance.

[0008] A system for training a time series prediction model based on deep learning is disclosed. The system includes a memory configured to store (i) a training data set including a first plurality of time series data and (ii) a test data set including a second plurality of time series data. The system also includes a processor. The processor is configured to train the time series prediction model using the training data set. The time series prediction model has a plurality of parameters learned during training. The processor is also configured to periodically evaluate the performance of the time series prediction model for the test data set during training. The training is performed for a sufficient number of training epochs so that after initial improvement and reaching a local maximum performance, the performance of the time series prediction model is allowed to deteriorate, and then improved again to reach a performance that exceeds the local maximum performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing aspects and other features of the methods and systems are explained in the following description taken in conjunction with the accompanying drawings.

[0010] Figure 1 is a graph showing model performance in training and testing as a function of model complexity.

[0011] Figure 2 An exemplary embodiment of a computing device that can be used to train a time series forecasting model is shown.

[0012] Figure 3 A flowchart showing a method for training time series forecasts using the Modern Regime training framework.

[0013] Figure 4 A table comparing the results of applying the "modern regime" training framework with the results of applying the original training framework is shown. DETAILED DESCRIPTION

[0014] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments shown in the accompanying drawings and described in the following written specification. It should be understood that this is not meant to limit the scope of the present disclosure. It should also be understood that the present disclosure includes any changes and modifications to the illustrated embodiments, and includes further applications of the principles of the present disclosure that would normally occur to a person skilled in the art to which the present disclosure belongs.

[0015] Overview This article describes a "modern regime" training framework for deep learning based time series forecasting models. It should be understood that the current state of the art models for time series forecasting are under-trained based on the double descent theory and therefore may result in suboptimal time series forecasts. To address this issue, the training framework described in this article advantageously applies the "modern regime" to time series forecasting and is shown to improve the performance of time series forecasting models.

[0016] The time series prediction model 30 using the "modern system" training framework can adopt any of a variety of deep learning-based model architectures. In some embodiments, the time series prediction model is a transformer-based time series prediction model. It will be appreciated by those skilled in the art that the time prediction model has a variety of useful applications in a variety of fields (e.g., in sales and marketing, demand forecasting, power consumption forecasting, etc.).

[0017] In summary, the "modern regime" training framework includes configuring the hyperparameters of the training process in a manner that significantly increases the complexity of the time series forecasting model 30 beyond conventional standards and enables the time series forecasting model 30 to achieve a "double descent" during the training process. In essence, a "double descent" occurs when the performance of the time series forecasting model 30 is allowed to deteriorate significantly after initially improving and reaching a local maximum performance and then improves again to reach a performance that exceeds the initial local maximum performance.

[0018] Most importantly, the complexity of the time series prediction model 30 must be increased by a large factor. The value of this factor can be obtained experimentally and depends on the specific time series prediction model used, the intended application of the model, and the statistical properties of the time series data used to train the model. Model complexity is mainly a function of the number of learnable parameters in the model and the number of training epochs used during training. However, even when the number of learnable parameters and the maximum number of training epochs are set high enough, there is no guarantee that the time series prediction model 30 will reach the complexity required for the "double descent". In particular, other hyperparameters and features of the training strategy may stop the training process prematurely, or otherwise completely prevent the "double descent" from occurring. To this end, other hyperparameters must be adjusted as needed to allow the time series prediction model 30 to reach the required complexity. In at least some embodiments, the patience of the model 30, that is, how long the training lasts after the performance stops improving, is increased to a level much higher than the level used conventionally. In at least some embodiments, the learning rate decay strategy is changed to a custom decay strategy that exponentially decays the learning rate until it reaches a predefined minimum value and then remains constant, where the predefined minimum value is higher than the value used conventionally. Making these changes allows the time series forecasting model 30 to gain the desired complexity and achieve a "double descent" during the training process and provide improved performance.

[0019] Figure 1 is a graph showing model performance in training and testing as a function of model complexity, illustrating the "double dip" phenomenon. In particular, the training loss curve 10 shows the training loss as a function of model complexity, and the test loss curve 20 shows the test loss as a function of model complexity. It can be seen that as the model complexity increases, the training loss continues to decrease, generally with diminishing returns. However, the test loss does not follow the same pattern as the training loss. Instead, the test loss initially decreases along with the training loss as the model initially "learns" the general patterns in the data. However, as the model begins to overfit the training data, these improvements level off. When overfitting occurs, the training loss continues to decrease, but the test loss begins to increase. This is because, at this stage, the model begins to learn random errors / patterns in the training data that do not indicate how the model should generally behave.

[0020] In the traditional regime, model complexity would be constrained to avoid this divergence between training loss and test loss resulting from overfitting the model to the training dataset. However, when model complexity is increased even further, a “double descent” phenomenon can be observed. In particular, this process of divergence between training loss and test loss continues as model complexity increases until the model complexity reaches a certain inflection point at which it reverses and the model can be considered to exit the “traditional” regime and transition into the “modern” regime. After this inflection point, further increases in model complexity cause the test loss to decrease again. At least in some cases, further increases in model complexity can cause the test loss to eventually fall below the best performance achieved in the “traditional” regime. When this happens, the model has exploited the “double descent” phenomenon.

[0021] It has been experimentally shown that the "modern regime" training framework described in this paper produces improved results on a large portion of standard benchmarks for long time series forecasting compared to state-of-the-art models trained using the "traditional" regime training framework. This "modern regime" training framework enables models to overcome the suboptimal results they previously provided and sets new benchmarks for the field of long time series forecasting.

[0022] Figure 4 A table is shown comparing the results of applying the "modern regime" training framework with the results of applying the original training framework. The results listed in the table are determined using various existing models (including, for example, FEDformer, Autoformer, and Informer) and using several existing datasets (ILI, Weather, Traffic, Exchange, Electricity, ETTm2, ETTm1, ETTh2, and ETTh1). In the left portion of the table, the results of standard benchmarking when a subset of the models are trained using the "modern regime" training framework described in this article are listed. In the right portion of the table, the results of the same benchmarking metrics when the models are trained using the original or conventional "traditional" regime training framework of the model are listed. It can be seen that in each case, applying the "modern regime" training framework can provide a considerable performance improvement compared to the "traditional" regime training framework and produce significant improvements in the field of time series forecasting.

[0023] Exemplary Hardware Embodiment Figure 2An exemplary embodiment of a computing device 100 that can be used to train a time series prediction model 30 to perform a time series prediction task is shown. Similarly, the computing device 100 can be used to operate a previously trained time series prediction model 30 to perform a time series prediction task. The computing device 100 includes a processor 110, a memory 120, a display screen 130, a user interface 140, and at least one network communication module 150. It should be understood that the illustrated embodiment of the computing device 100 is only an exemplary embodiment and represents only any of the various modes or configurations of a server, a desktop computer, a laptop computer, a mobile phone, a tablet computer, or any other computing device operating in the manner described herein. In some embodiments, the computing device 100 communicates with a database 102, which can be hosted by another device or stored in the memory 120 of the computing device 100 itself.

[0024] The processor 110 is configured to execute instructions to operate the computing device 100, thereby implementing the features, functions, characteristics, etc. described herein. To this end, the processor 110 is operably connected to the memory 120, the display screen 130, and the network communication module 150. The processor 110 generally includes one or more processors that can operate in parallel or otherwise cooperate with each other. Those of ordinary skill in the art will recognize that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Therefore, the processor 110 may include a system having a central processing unit, a graphics processing unit, a plurality of processing units, a dedicated circuit for implementing a function, programmable logic, or other processing system.

[0025] The memory 120 is configured to store data and program instructions that, when executed by the processor 110, enable the computing device 100 to perform the various operations described herein. As will be appreciated by one of ordinary skill in the art, the memory 120 may be any type of device capable of storing information accessible to the processor 110, such as a memory card, ROM, RAM, hard drive, disk, flash memory, or any of a variety of other computer-readable media used as data storage devices.

[0026] Display screen 130 may include any of various known types of displays, such as an LCD or OLED screen, configured to display a graphical user interface. User interface 140 may include various interfaces for operating computing device 100, such as buttons, switches, a keyboard or other keypad, a speaker, and a microphone. Alternatively or additionally, display screen 130 may include a touch screen configured to receive touch input from a user.

[0027] The network communication module 150 may include one or more transceivers, modems, processors, memories, oscillators, antennas, or other hardware conventionally included in communication modules to enable communication with various other devices. In particular, the network communication module 150 typically includes an Ethernet adapter or a wireless network configured to enable communication with a wired or wireless network. Modules, and / or routers (not shown) configured to enable communication with various other devices. In addition, the network communication module 150 may include module (not shown) and one or more cellular modems configured to communicate with a wireless telephone network.

[0028] In at least some embodiments, the memory 120 stores program instructions of the time series prediction model 30, which are configured to perform time series prediction tasks once training is performed. In at least some embodiments, the database 102 stores a plurality of time series data for training and testing the time series prediction model.

[0029] Methods for training time series forecasting models The following describes various operations and processes for operating the computing device 100 to develop and train the time series prediction model 30 to perform the time series prediction task. In these descriptions, the statement that the method, processor and / or system is performing a task or function refers to the controller or processor (e.g., the processor 110 of the computing device 100) executing programming instructions stored in a non-transitory computer-readable storage medium (e.g., the memory 120 of the computing device 100) operably connected to the controller or processor to manipulate data or operate one or more components in the computing device 100 or the database 102 to perform the task or function. In addition, the steps of the method can be performed in any feasible time order, regardless of the order shown in the figure or the order in which the steps are described.

[0030] Figure 3 A flow chart of a method 200 for training a time series forecast using a "modern regime" training framework is shown. The method 200 advantageously enables the performance of the time series forecast model 30 to exceed the local maximum performance achieved under the "traditional regime" training framework. In particular, the method 200 continues to train the time series forecast model 30 regardless of the tendency of the model to begin to overfit the training data after initially reaching a local maximum performance. The method 200 enables the time series forecast model 30 to have sufficient complexity to pass through this overfitting stage and ultimately achieve performance that exceeds the initial local maximum performance provided by the "traditional regime" training framework.

[0031] The method 200 begins by storing a training data set and a test data set (block 210). In particular, the processor 110 receives and / or the database 102 stores a training data set comprising a first plurality of time series training data samples and a test data set comprising a second plurality of time series training data samples. Each sample in the training and test data sets comprises (i) input time series data and (ii) an output time series data that is subsequent in time to the input time series data. The time series data may comprise a variety of different types of timestamped data, where a sequential time is associated with one or more values ​​(e.g., of a measured or observed quantity). Furthermore, it should be appreciated that the method 200 is primarily agnostic to the particular domain of the time series data.

[0032] The method 200 continues to train the time series prediction model using the training data set (box 220). In particular, the processor 110 uses the training data set to train the time series prediction model 30 to predict future time series data based on the input sequence of time series data. The time series prediction model 30 can include any model with multiple learnable parameters, and it is configured or can be trained to receive past values ​​of a time series as input and determine the expected future values ​​of the corresponding time series as output. In at least some embodiments, the time series prediction model 30 is an artificial neural network. For example, such a neural network can have a transformer-based architecture.

[0033] The processor 110 trains the time series prediction model 30 over the process of multiple training periods. Each training period includes multiple iterations or batches. For each iteration or batch, the processor 110 determines one or more outputs of the time series prediction model 30 by providing an input time series of one or more training samples from the training data set as an input to the time series prediction model 30. The processor 110 determines the training loss based on a comparison of the one or more outputs with the corresponding output time series of the one or more training samples. The processor 110 improves and / or updates multiple learnable parameters of the time series prediction model 30 based on the determined training loss (e.g., using stochastic gradient descent, etc.). For each of the multiple iterations or batches, the iterative training process is repeated until the time series prediction model 30 has seen all or substantially all of the training data sets, thereby ending the training period. The iterative training process is then repeated for multiple training periods.

[0034] In at least some embodiments, the processor 110 trains the time series prediction model 30 using a decaying learning rate strategy. In particular, whenever the parameters of the time series prediction model 30 are improved and / or updated, the improvement and / or update of the parameters is scaled or otherwise adjusted with a learning rate (e.g., a number between 0 and 1). Therefore, the processor 110 improves and / or updates the parameters of the time series prediction model 30 based on both the training loss and the current learning rate. The learning rate decays as the training proceeds, for example, as a function of the current training period. In at least one embodiment, the learning rate decays from the initial learning rate as a function of the current training period until a predefined minimum learning rate is reached, and then remains constant at the predefined minimum learning rate after reaching the predefined minimum learning rate.

[0035] The method 200 continues to periodically evaluate the performance of the time series prediction model for the test data set during training (box 230). In particular, periodically during training, the processor 110 evaluates the performance of the time series prediction model 30 for the test data set. To this end, the processor 110 determines one or more outputs of the time series prediction model 30 by providing an input time series of one or more test samples from the test data set as an input to the time series prediction model 30. The processor 110 determines the test loss based on a comparison of the one or more outputs with the corresponding output time series of the one or more test samples. In general, a relatively small test loss value indicates a relatively high or better performance, and similarly, a relatively large test loss value indicates a relatively low or poor performance.

[0036] The training and evaluation in method 200 lasts for a sufficient number of training epochs so that after the initial improvement and reaching of the local maximum performance, the performance of the time series prediction model is allowed to deteriorate and then improve again to achieve a performance that exceeds the local maximum performance (box 240). In particular, the processor 110 continues to train the time series prediction model 30 and periodically evaluates the performance of the time series prediction model 30 over a sufficient number of training epochs so that after the initial improvement and reaching of the local maximum performance, the performance of the time series prediction model 30 is allowed to deteriorate and then improve again to achieve a performance that exceeds the local maximum performance. In other words, the processor 110 continues to train the time series prediction model 30 until the performance achieves a "double decline". In at least some embodiments, the performance of the time series prediction model 30 must be allowed to deteriorate by at least a predetermined amount and then improve again to achieve a performance that exceeds the local maximum performance, i.e., only a very small deterioration and subsequent improvement is not enough to be called a "double decline".

[0037] Reference again Figure 1As an example, when the processor 110 begins to train the time series prediction model 30, the training loss and the test loss initially decrease together as the number of training epochs increases, and reach a local minimum test loss 22 (indicating a local maximum performance). However, as the processor 110 continues to train the time series prediction model 30, the test loss increases significantly as the number of training epochs increases (indicating a significant deterioration in performance). This difference between the test loss and the training loss indicates that the time series prediction model 30 is overfitting the training data set, and therefore, this stage of training may be referred to herein as the "initial overfitting stage." As the processor 110 continues to train the time series prediction model 30, after a sufficient number of training epochs, the test loss decreases again (indicating an improvement in performance), and eventually drops below the initial local minimum test loss 22 (or exceeds the initial local maximum performance), thereby achieving a "double drop."

[0038] It should be understood that the various hyperparameters of the training process must be appropriately configured so that this "double descent" process can occur during training. Most importantly, the complexity of the time series prediction model 30 must be increased to a sufficient complexity. As described above, model complexity is primarily a function of the number of learnable parameters in the model and the number of training epochs used during training. Therefore, the processor 110 sets (i) the number of learnable parameters of the time series prediction model 30 and (ii) the maximum number of training epochs for training to be high enough so that the performance of the time series prediction model 30 can deteriorate and then improve again after reaching a local maximum performance, i.e., achieving "double descent". In at least some embodiments, the processor 110 sets the number of learnable parameters and the maximum number of training epochs based on user input received via the user interface 140.

[0039] However, even when the number of learnable parameters and the maximum number of training epochs are set high enough, there is no guarantee that the time series forecasting model 30 will achieve the complexity required for the "double descent." In particular, other hyperparameters and features of the training process may stop the training process prematurely, or otherwise prevent the "double descent" from occurring at all.

[0040] As described above, in some embodiments, a decaying learning rate strategy is employed, wherein each time the parameters of the time series prediction model 30 are improved and / or updated, the improvement and / or update of the parameters is scaled or otherwise adjusted with the learning rate. In some embodiments, the learning rate decays from the initial learning rate as a function of the current training epoch until a predefined minimum learning rate is reached, and remains constant at the predefined minimum learning rate after the predefined minimum learning rate is reached.

[0041] If the predefined minimum learning rate is set too low, it may prevent the performance of the time series prediction model 30 from getting worse after reaching the initial local maximum performance, thereby completely preventing the "double drop" from occurring. Therefore, the processor 110 sets the predefined minimum learning rate high enough to allow the performance of the time series prediction model to deteriorate and then improve again after reaching the local maximum performance. In at least some embodiments, the processor 110 sets the predefined minimum learning rate based on user input received via the user interface 140.

[0042] In some training embodiments, a "patience" strategy is employed, wherein the processor 110 terminates training in response to the performance of the time series prediction model 30 not improving within a predetermined number of epochs. If the predetermined number of epochs or "patience" is set too low, it may prevent the performance of the time series prediction model 30 from getting worse after reaching an initial local maximum performance, thereby preventing the "double dip" from occurring altogether.

[0043] Thus, in at least some embodiments, processor 110 sets the predetermined number of epochs or "patience" high enough to allow the performance of the time series prediction model to deteriorate and then improve again after reaching a local maximum performance. In at least some embodiments, processor 110 sets a predefined minimum learning rate based on user input received via user interface 140.

[0044] Alternatively or additionally, in some embodiments, the processor 110 prevents the training process from terminating based on a "patience" strategy until the time series prediction model 30 is allowed to deteriorate and then improve again after reaching a local maximum performance. In other words, the processor 110 terminates training in response to the performance of the time series prediction model 30 not improving within a predetermined number of periods only after the performance of the time series prediction model 30 has been allowed to deteriorate and then improve again after reaching a local maximum performance, i.e., a "double descent" is achieved. If the "double descent" has not occurred, the processor 110 prevents the training process from terminating even if the performance of the time series prediction model 30 has not improved within a predetermined number of periods.

[0045] Embodiments within the scope of the present disclosure may also include non-transitory computer-readable storage media or machine-readable media, which are used to carry or store computer-executable instructions (also referred to as program instructions) or data structures thereon. Such non-transitory computer-readable storage media or machine-readable media can be any available medium that can be accessed by a general-purpose or special-purpose computer. As an example and not limitation, such non-transitory computer-readable storage media or machine-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store computer-executable instructions or data structure forms of required program code components. The above combination should also be included in the scope of non-transitory computer-readable storage media or machine-readable media.

[0046] Computer executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. Computer executable instructions also include program modules executed by computers in a stand-alone or network environment. Typically, program modules include routines, programs, objects, components, and data structures, etc., which perform specific tasks or implement specific abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of program code components for performing the steps of the method disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in these steps.

[0047] While the disclosure has been illustrated and described in the drawings and foregoing description, it should be considered illustrative and not restrictive. It should be understood that only preferred embodiments have been presented and that all changes, modifications and further applications that come within the spirit of the disclosure are intended to be protected.

Claims

1. A method for training a time series prediction model based on deep learning, the method comprising: storing in a memory (i) a training data set including a first plurality of time series data and (ii) a test data set including a second plurality of time series data; training, with a processor, the time series prediction model using the training data set, the time series prediction model having a plurality of parameters learned during training; and periodically during training, using a processor to evaluate the performance of the time series prediction model on the test data set, The training is performed for a sufficient number of training epochs such that after initially improving and reaching a local maximum performance, the performance of the time series forecasting model is allowed to deteriorate and then improve again to reach a performance exceeding the local maximum performance.

2. The method of claim 1 , wherein for each iteration of each training epoch, the training comprises: Determining at least one output of the time series forecasting model by providing at least one corresponding training sample from the training data set as an input to the time series forecasting model; determining a training loss based on the at least one output; and A plurality of parameters of the time series prediction model are improved based on the training loss.

3. The method of claim 2, wherein in each iteration of each training epoch, the improving further comprises: improving a plurality of parameters of the time series prediction model based on the training loss and the learning rate, The learning rate decays from an initial learning rate as a function of a current training epoch until a predefined minimum learning rate is reached, and remains constant at the predefined minimum learning rate after reaching the predefined minimum learning rate.

4. The method according to claim 3, further comprising: The predefined minimum learning rate is set high enough to allow the performance of the time series forecasting model to deteriorate and then improve again after reaching a local maximum performance.

5. The method according to claim 3, wherein: The learning rate decays exponentially from the initial learning rate as a function of the current training epoch until a predefined minimum learning rate is reached.

6. The method according to claim 2, further comprising: In response to performance of the time series forecasting model not improving within a predetermined number of training epochs, the training is terminated.

7. The method according to claim 6, further comprising: The predetermined number of training epochs is set high enough to allow the performance of the time series forecasting model to deteriorate and then improve again after reaching a local maximum performance.

8. The method according to claim 6, further comprising: The training is terminated in response to the performance of the time series prediction model not improving over a predetermined number of training epochs only after the performance of the time series prediction model has been allowed to deteriorate and then improve again after reaching the local maximum performance.

9. The method according to claim 1, further comprising: The number of parameters in the plurality of parameters is set high enough to enable the performance of the time series forecasting model to deteriorate and then improve again after reaching a local maximum performance.

10. The method according to claim 1, further comprising: The maximum number of training epochs for training is set high enough to allow the performance of the time series forecasting model to deteriorate and then improve again after reaching a local maximum performance.

11. The method according to claim 1, wherein: The training is performed for a sufficient number of training epochs so that after initially improving and reaching a local maximum performance, the performance of the time series prediction model is allowed to deteriorate by at least a predetermined amount and then improves again to reach a performance that exceeds the local maximum performance.

12. The method according to claim 1, wherein: The time series prediction model is configured to receive past values ​​of a corresponding time series as input and determine an expected future value of the corresponding time series as output.

13. The method according to claim 1, wherein: The time series prediction model is a neural network model.

14. The method according to claim 13, wherein: The time series forecasting model has a transformer-based architecture.

15. The method according to claim 1, wherein evaluating the performance of the time series forecasting model further comprises: determining at least one additional output of the time series forecasting model by providing at least one corresponding training sample from the test data set as an input to the time series forecasting model; and A test loss is determined based on the at least one further output.

16. A non-transitory computer-readable medium storing program instructions for training a deep learning-based time series prediction model, the program instructions, when executed by a processor, causing the processor to: receive (i) a training data set comprising a first plurality of time series data and (ii) a test data set comprising a second plurality of time series data; Using the training data set to train the time series prediction model, the time series prediction model having a plurality of parameters learned during training; and periodically during training, evaluating the performance of the time series forecasting model on the test dataset, The training is performed for a sufficient number of training epochs such that after initially improving and reaching a local maximum performance, the performance of the time series forecasting model is allowed to deteriorate and then improve again to reach a performance exceeding the local maximum performance.

17. A system for training a time series prediction model based on deep learning, the system comprising: a memory configured to store (i) a training data set including a first plurality of time series data and (ii) a test data set including a second plurality of time series data; and The processor is configured to: Using the training data set to train the time series prediction model, the time series prediction model having a plurality of parameters learned during training; and periodically during training, evaluating the performance of the time series forecasting model on the test dataset, The training is performed for a sufficient number of training epochs such that after initially improving and reaching a local maximum performance, the performance of the time series forecasting model is allowed to deteriorate and then improve again to reach a performance exceeding the local maximum performance.

Images