Systems and methods for online time series forecasting

Abstract

An embodiment provides a combined fast learning network and slow learning network framework (called "FSNet") to train deep neural predictors on-the-fly for online time series prediction. FSNet is built on a deep neural network backbone (a slow learner) with two complementary components to facilitate fast adaptation to both new and recurrent concepts. To this end, FSNet employs a layer-wise adapter to monitor each layer's contribution to the prediction loss via its partial derivatives. The adapter transforms each layer's weights and features at each step based on its recent gradients, enabling fast adaptation of layer-wise granularity to optimize the current loss. In addition, FSNet employs a second complementary associative memory component to store important recurring patterns observed during training. The adapter interacts with the memory to store, update, and retrieve previous transformations, facilitating fast learning of such patterns.

Description

[Technical Field]

[0001] [Cross reference] This disclosure claims priority to U.S. Non-Provisional Patent Application No. 17 / 871,819, filed July 22, 2022, and U.S. Provisional Patent Application No. 63 / 305,145, filed January 31, 2022, each of which is expressly incorporated by reference in its entirety.

[0002] [Technical field] FIELD Embodiments relate generally to machine learning systems, and more particularly to online time series prediction. [Background technology]

[0003] Deep neural network models are widely used in time series forecasting. For example, trained models can be used to forecast time series data, such as continuous market data, weather data, and / or the like, over a period of time into the future. Existing deep models employ batch learning for time series forecasting tasks. Such models often randomly sample lookback and forecast windows during training and freeze the model during evaluation, destroying the time-varying (non-stationary) nature of the time series.

[0004] Therefore, there is a need for an efficient and adaptive deep learning framework for online temporal prediction. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a simplified diagram illustrating an example structure of an FSNet framework for forecasting time series, according to embodiments described herein. [Figure 2] 2 is a simplified diagram illustrating an example structure of a TCN layer (block) of the FSNet framework depicted in FIG. 1 according to embodiments described herein. [Figure 3]FIG. 3 is a simplified diagram illustrating an example structure of an extended convolutional layer in the TCN layer (block) shown in FIG. 2 according to embodiments described herein. [Figure 4] 1 is a simplified diagram of a computing device implementing the FSNet framework, according to some embodiments described herein. [Figure 5] 4 is a simplified pseudo-code segment for a fast learning network and a slow learning network implemented in the FSNet framework shown in FIGS. 1-3, according to an embodiment described herein. [Figure 6] FIG. 6 is a simplified logic flow diagram illustrating an example process corresponding to the pseudo-code algorithm of FIG. 5, according to embodiments described herein. [Figure 7] 10A-10C are exemplary data charts and plots illustrating the performance of FSNet in exemplary data experiments, according to embodiments described herein. [Figure 8] 10A-10C are exemplary data charts and plots illustrating the performance of FSNet in exemplary data experiments, according to embodiments described herein. [Figure 9] 10A-10C are exemplary data charts and plots illustrating the performance of FSNet in exemplary data experiments, according to embodiments described herein.

[0006] In the figures, elements with the same name have the same or similar function. DETAILED DESCRIPTION OF THE INVENTION

[0007] As used herein, the term "network" may comprise any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or therewith.

[0008] As used herein, the term "module" may comprise a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0009] A time series is a set of values ​​corresponding to a parameter of interest at different points in time. Examples of parameters can include stock prices, temperature readings, etc. Time series forecasting is the process of determining a future data point or set of future data points beyond the set of values ​​in a time series. Time series forecasting for dynamic data using deep learning remains challenging.

[0010] Embodiments provide a framework (called "FSNet") that combines a fast learning network and a slow learning network to train deep neural predictors on the fly for online time series prediction. FSNet is built on a deep neural network backbone (a slow learner) with two complementary components to facilitate fast adaptation to both new and recursive concepts. To this end, FSNet employs a layer-by-layer adapter that monitors each layer's contribution to the prediction loss via its partial derivatives. The adapter transforms each layer's weights and features at each step based on their recent gradients, enabling fast layer-by-layer granularity adaptation to optimize the current loss. In addition, FSNet employs a second, complementary associative memory component to memorize important recurring patterns observed during training. The adapter interacts with the memory to store, update, and retrieve previous transformations, facilitating fast learning of such patterns.

[0011] In this way, the FSNet framework can adapt to fast-changing and long-term recurring patterns in time series. Specifically, in FSNet, the deep neural network plays the role of the neocortex, while the adapter and its memory act as the hippocampal component. Overview of the FSNet Framework

[0012] FIG. 1 is a simplified diagram illustrating an example structure of an FSNet framework 100 for forecasting time series, according to embodiments described herein.

[0013] The FSNet framework 100 includes multiple convolution blocks 104a-n connected to a regressor 105. The FSNet framework 100 receives a time series of T observations, each having n dimensions, from an input interface such as a memory or a network adapter.

number

[0014] In one embodiment, the FSNet framework 100 includes a parameter

number

[0015] Based on the TCN backbone 104a-n, the FSNet framework 100 further includes two complementary components: a per-layer adapter φ1 (shown as 315 in FIG. 3 ) for each TCN layer 104a-n, and a per-layer associative memory M1 (shown as 318 in FIG. 3 ) for each TCN layer 104a-n. Thus, the total trainable parameters of the framework are given by ω = {θ l ,φ l}, and all associative memories are l} l=1,...,L is.

[0016] 2 is a simplified diagram illustrating an example structure of a TCN layer (block) 104a of the FSNet framework depicted in FIG. 1, according to an embodiment described herein. In each TCN layer (block), e.g., 104a, a block input 202 may be processed by several extended convolutional layers 204, 206, and the convolutional output is added to the original block input 202 to generate a block output 208. Note that although two extended convolutional layers 204 and 206 are shown in FIG. 2 for illustrative purposes only, any other number of extended convolutional layers may be used in a TCN block.

[0017] In one embodiment, each TCN block 104a may rely on its adapter 315 and associative memory 318 to adapt quickly to changes in the time series data 102 or to learn more efficiently with limited data. Each block or layer 104a-104n may adapt independently, rather than restricting adaptation to the depth of the network, i.e., gradient descent over the depth of the network 104a-n. The partial derivatives of each layer 104a-n

number

number

number

[0018] Thus, each convolutional filter stack is accompanied by an adapter and an associative memory. As further shown in connection with Figure 3, at each layer, the adapter receives the gradient EMA and interacts with the memory and convolutional filters accordingly.

[0019] 3 is a simplified diagram illustrating an example structure of the extended convolutional layer 204 (or 206) in the TCN layer (block) 104a shown in FIG. 2, according to an embodiment described herein. The extended convolutional layer 204 may comprise a convolutional filter 310, a per-layer adapter 315, and a per-layer memory 318. The input 202 to the extended convolutional layer 204 may be provided to the convolutional filter 310, which calculates an exponential moving average (EMA) 313 of the gradient of the TCN backbone. Specifically, because the gradient of a single sample may vary widely and introduce noise into the adaptation parameters, an EMA is used to smooth out noise in online training by:

number

[0020] In some embodiments, the fast adapter 315 may use an element-wise transform as the adaptation process due to its efficiency for continuous learning. The resulting adaptation parameter u l 316 is u l =[α l ;β l ], (i) the weight adaptation parameter α l and (ii) the feature adaptation parameter β l In some embodiments, the high-speed adapter 315 may absorb the bias conversion parameter into α1 for simplicity.

[0021] In one embodiment, layer θ l The adaptation for α may include weight adaptation and feature adaptation, as shown in 319. First, the weight adaptation parameter α l operates on the corresponding weights of the backbone network via element-wise multiplication as follows:

number

number

[0022] Similarly, the gradient feature adaptation component β l and the feature adaptation parameter modifies the convolutional layer feature map based on the element-wise multiplication between the feature adaptation component and the first convolutional layer feature map. For example, the feature adaptation β l Also, the output feature map h l and produces output 322 as follows:

number

[0023] In this way, the convolution layer θ l is the weight adaptation component α l and feature adaptation component β l The data may be updated based on the following:

[0024] In some embodiments, the gradients may be directly mapped to element-wise adaptive parameters, which may result in very high-dimensional mappings.

[0025] In some embodiments, Ω(·;Φ l), to divide the gradient into chunks of equal size and then map each chunk to an element of the adaptation parameters. Specifically, the chunking operation may be implemented as (1) flattening the gradient EMA of the corresponding block of the TCN model 120 into a vector, (2) dividing the gradient vector into d chunks, (3) mapping each chunk to a hidden representation, and (4) mapping each hidden representation to the coordinates of the target adaptation parameter u. For example, using a vectorization operation (vec(·)) that flattens a tensor into a vector and a division operation (e, B) that divides the vector e into B segments, each with size dim(e) / B, the EMA gradient 313 of the backbone layer of the TCN backbone is converted to the adaptation coefficients u via the chunking process as follows:

number

number

[0026] In summary,

number

number

[0027] In one embodiment, in a time series, old patterns may reappear in the future, and it is beneficial to recall similar knowledge from the past to further facilitate learning. Storing the original data can alleviate this problem, but privacy concerns may make this impractical in many domains. Therefore, an associative memory 318 may be implemented to store adaptation coefficients for recurring events encountered during learning. While the adapter 315 can handle fast recent changes over short timescales, recurring patterns are stored in memory 318 and retrieved when they reappear in the future. To this end, each adapter 315

number

[0028] Specifically, because interacting with memory 318 at every step is expensive and susceptible to noise, memory interaction can be triggered only when a substantial change in the representation is detected. The interference between the current and past representations can be characterized by the dot product between the gradients. Therefore, the cosine similarity between the recent gradient and the long-term gradient can be calculated and monitored to trigger memory interaction when the interference falls below a threshold (which may indicate that the pattern has changed significantly). To this end, gradient EMA

number

number

number

[0029] In one embodiment, when the current adaptive parameters may not capture the entire event, which may span several samples, memory read and write operations may be performed using the EMA (with coefficient γ′) of the adaptive parameters to fully capture the current pattern. l The EMA of g ^ l When a memory interaction is triggered, the adapter queries and retrieves the most similar past translation via an attention read operation, which is a weighted sum over memory items:

number

number

number

number

number

[0030] In one embodiment, the FSNet framework described in connection with Figures 1-3 is suitable for task-free online continuous learning scenarios because it does not need to explicitly detect when a task switches. Instead, the task boundary definition can be relaxed to allow the model to continuously improve its learning on the current sample. Computing Environment

[0031] 4 is a simplified diagram of a computing device implementing the FSNet framework, according to some embodiments described herein. As shown in FIG. 4, computing device 400 includes a processor 410 coupled to a memory 420. The operation of computing device 400 is controlled by processor 410. While computing device 400 is shown with only one processor 410, it is understood that processor 410 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs), and / or the like within computing device 400. Computing device 400 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0032] Memory 420 may be used to store software executed by computing device 400 and / or one or more data structures used during operation of computing device 400. Memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media may include a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read from by a processor or computer.

[0033] The processor 410 and / or memory 420 may be arranged in any suitable physical location. In some embodiments, the processor 410 and / or memory 420 may be implemented on the same substrate, in the same package (e.g., a system-in-package), on the same chip (e.g., a system-on-chip), etc. In some embodiments, the processor 410 and / or memory 420 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 410 and / or memory 420 may be located within one or more data centers and / or cloud computing facilities.

[0034] In some examples, memory 420 may include a non-transitory, tangible, machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, memory 420 includes instructions for an online time series prediction module 430, which may be used to implement and / or emulate systems and models and / or implement any of the methods described further herein. In some examples, online time series prediction module 430 may receive input 440, such as time series data within a lookback window, via a data interface 415. Data interface 415 may be either a user interface that receives uploaded time series data or a communications interface that may receive or retrieve previously stored samples of lookback and prediction windows from a database. Time series prediction module 430 may generate output 450, such as a prediction for input 440.

[0035] In some embodiments, the time series prediction module 430 may further include a series of TCN blocks 431a-n (similar to 104a-n shown in FIG. 1) and a regressor 432 (similar to 105 shown in FIG. 1). In one implementation, the time series prediction module 430 and its sub-modules 431-432 may be implemented via software, hardware, and / or a combination thereof.

[0036] Some examples of computing devices, such as computing device 400, may include non-transitory, tangible, machine-readable media containing executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform the processes of the methods described throughout this disclosure. Some common forms of machine-readable media that may contain the processes of the methods are, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card, paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read from by a processor or a computer. Workflow example

[0037] 5 is a simplified pseudocode segment for a fast learning network and a slow learning network implemented in the FSNet framework described in FIGS. 1-3, according to an embodiment described herein. For example, for a stack of L layers (e.g., 104a-n in FIG. 1), the weight adaptation components α l and feature adaptation component β l A forward calculation may be performed to calculate the adaptive parameters, including: The memory read and write operations may be performed via a chunking process, and the adaptive parameters may be updated by a weighted sum of the current adaptive parameters and the past adaptive parameters.

[0038] Weight adaptation and feature adaptation can then be performed according to Equation (5). After updating the adaptation parameters through forward calculations across L layers, predicted data can be generated via regressors (e.g., 105 in Figure 1). The predicted data is then compared with ground truth future data from training samples to calculate a prediction loss, which is then used to update the stack of L layers via backpropagation. The regressors can also be updated via stochastic gradient descent (SGD). The adaptation parameters and EMA adaptation parameters are then updated backward.

[0039] 6 is a simplified logic flow diagram illustrating an example process 600 corresponding to the pseudo-code algorithm of FIG. 5, according to embodiments described herein. One or more of the processes of method 600 may be implemented, at least in part, in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes. In some embodiments, method 600 corresponds to the operation of FSNet framework 100 (FIG. 1) for forecasting time-series data at future timestamps in a dynamic system.

[0040] In step 602, a time series data set including multiple data points corresponding to multiple timestamps within a lookback time window (e.g., 102 in FIG. 1) may be received via a data interface (e.g., 415 in FIG. 4).

[0041] In step 604, a convolutional layer (e.g., block 104a in FIGS. 1-2) from a stack of convolutional layers (e.g., blocks 104a-n in FIG. 1) may calculate a first gradient based on an exponential moving average of the gradients corresponding to the respective convolutional layer, for example, according to equation (1).

[0042] In step 606, a first adaptation parameter u1 corresponding to the convolutional layer may be determined by mapping a portion of the first gradient to an element of the first adaptation parameter. For example, the first adaptation parameter may be a first weight adaptation component α l and the first feature adaptation component β l Includes:

[0043] In step 608, for at least one convolutional layer of the temporal convolutional neural network, a layer prediction loss indicating the loss contribution of each convolutional layer to the overall prediction loss according to the plurality of data points may be optionally determined based on the plurality of data points. For example, the layer prediction loss may be determined by the partial derivative

number

[0044] Optionally, at least one convolutional layer may be updated based on the layer prediction loss in step 610. In this manner, each layer may be independently monitored and modified to learn the current loss by learning through the layer prediction loss.

[0045] In step 612, the cosine similarity between the first gradient of the updated convolutional layer and the long-term gradient associated with the at least one first convolutional layer may be calculated, for example, according to equation (6).

[0046] In step 614, if the cosine similarity is greater than a predefined threshold, method 600 proceeds to step 616, where it performs a chunking process for memory reads and writes. Specifically, in step 616, current adaptive parameters are retrieved from an indexed memory (e.g., 318 in FIG. 3 ) corresponding to the convolutional layer. In step 618, the contents stored in the indexed memory (e.g., 318 in FIG. 3 ) are updated based on the current adaptive parameters and the first adaptive parameters. In step 620, the first adaptive parameters are updated by taking a weighted average with the retrieved current adaptive parameters.

[0047] In step 622, the first weight adaptation component α is calculated, for example, according to equation (5): l and the layer parameter θ corresponding to the first layer l and the adapted layer parameters

number

[0048] In step 624, the feature map h of the first convolutional layer is l is the first feature adaptation component β l For example, the first feature map is a convolution of the adapted layer parameters with the previous adapted feature map from the previous layer. In step 626, the first feature adaptation component β l and the first feature map h of the first convolutional layer l Based on and, the adapted feature map

number

[0049] In step 628, a regressor (e.g., 105 in FIG. 1) may generate time series forecast data corresponding to a future time window based on the final feature map output from the stack of convolutional layers corresponding to the time series data in the lookback time window.

[0050] In step 630, a predicted loss may be calculated based on the generated time series forecast data and ground truth data corresponding to a future time window.

[0051] The stack of convolutional layers and regressors may then be updated based on the predicted loss via backpropagation. In step 632, the regressors may be updated via stochastic gradient descent. Then, in step 634, the gradients and adaptation parameters of each layer in the stack may be updated backwards. Performance Examples

[0052] We conducted data experiments to test the following hypotheses: (i) FSNet facilitates faster adaptation to both novel and recursive concepts compared to existing strategies; (ii) FSNet achieves faster and better convergence than other methods; and (iii) modeling partial derivatives is a key factor for fast adaptation.

[0053] Specifically, a wide range of time-series forecasting datasets were used for data experiments: (i) ETT1 (Zhou et al., Informer: Beyond efficient transformer for long sequence time-series forecasting, in Proceedings of AAAI, 2021) records the target value of "oil temperature" and six electricity load characteristics over a two-year period. The ETTh2 and ETTm1 benchmarks are used, recording observations at hourly and 15-minute intervals, respectively. (ii) The ECL (Electricity Consuming Load)2 dataset collects electricity consumption data from 321 customers from 2012 to 2014. (iii) The Traffic3 dataset records road occupancy rates on freeways in the San Francisco Bay Area. (iv) The Weather4 dataset records 11 weather characteristics for approximately 1,600 locations in the United States at hourly intervals from 2010 to 2013.

[0054] In addition, two synthetic data sets were constructed to explicitly test the model's ability to address novel and recursive concept drift. Tasks can be synthesized by sampling 1,000 samples from a first-order autoregressive process, ARφ(1), with coefficient φ, where different tasks correspond to different φ values. The first synthetic data set, S-Abrupt, contains abrupt and recursive concepts in which samples abruptly switch from one AR process to another in the following order: AR0.1(1), AR0.4(1), AR0.6(1), AR0.1(1), AR0.3(1), AR0.6(1). The second data set, S-Gradual, contains a gradual, gradual shift, starting from the last 20% of each task. In this scenario, the last 20% of samples in a task are averaged from two AR processes in the above order.

[0055] In our implementation, the data is split into a 25:75 ratio between the warm-up and online training phases, and we consider the TCN backbone for our experiments, except for the Informer baseline. The optimization details are described in Zhang et al., "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting," in Proceedings of AAAI, 2021, by optimizing the l2(MSE) loss using the AdamW optimizer. Both the epoch and batch size are set to 1 to comply with the online learning setting. A fair comparison is achieved by ensuring that all baselines use the same total memory budget as FSNet, which includes three times the network size: one working model and two EMAs of its gradients. Therefore, for ER, MIR, and DER++, we use episodic memory to memorize previous samples to meet this budget. For the remaining baselines, we can instead increase the backbone size. Finally, during the warm-up phase, the mean and standard deviation are calculated to normalize the online training samples and perform hyperparameter cross-validation. For all benchmarks, the lookback window length is set to 60 and the prediction horizon is set to H = 1. We test the model's ability to predict longer horizons by varying H ∈ {1, 24, 48}.

[0056] For comparison, sequential training strategies from both continuous learning and time series forecasting are employed. The first is the OnlineTCN strategy, which simply trains continuously (described in Zinkevich, "Online convex programming and generalized infinitesimal gradient ascent," in Proceedings of the 20th international conference on machine learning (icml-03), pages 928-936, 461, 2003). The second is the Experience Replay (ER) strategy (described in Lin, "Self-improving reactive agents based on reinforcement learning, planning and teaching," Machine learning, 8(3-4):293-321, 1992), which employs a buffer to store previous data and interleave older samples during the learning of more recent samples. Three recent advanced variants of ER are presented. First, TFCL (Aljundi et al., Task-free continual learning, in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 11254-11263,325, 2019) introduces a task boundary detection mechanism and knowledge integration strategy by regularizing the network output. Second, MIR (Aljundi et al., Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems, 32:11849-11860, 2019) replaces random sampling in ER with selecting the sample that induces the most forgetting.Finally, DER++ (Buzzega et al., Dark experience for general continual learning: a strong, simple baseline, in 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020) enhances standard ER with a knowledge distillation strategy (described in Hinton et al., Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015). ER and its variants are powerful baselines in online settings because they significantly reduce noise from single samples and benefit from training in mini-batches, which provides faster and better convergence (see Bottou et al., Online learning and stochastic approximations, Online learning in neural networks, 17(9):142, 1998). The aforementioned baselines use the TCN backbone, Informer, but also include time series prediction methods based on the Transformer architecture (Vaswani et al., Attention is all you need. Advances in neural information processing systems, 30, 2017).

[0057] First, we introduce the Online Gradient Descent (OGD) method, which simply trains continuously (Zinkevich, Online convex and generalized infinitesimal gradient ascent, in proceedings of the 20 thThe first baseline is the Experiment Repla (described in Chaudhry et al., On tiny episodic memories in continual learning, arXiv preprint arXiv:1902.10486, 2019) strategy, in which a buffer is employed to remember previous data and interleave older samples while training more recent ones. Another baseline is DER++ (Buzzega et al., Dark experience for general continual learning: a strong, simple baseline, in proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020), which further adds knowledge distillation (Hinton et al., Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531, 2015) loss to ER. ER and DER++ are powerful baselines in online settings because they benefit from training in mini-batches, which significantly reduces noise from single samples and provides faster and better convergence.

[0058] Figure 7 reports the cumulative mean squared error (MSE) and mean absolute error (MAE) at the end of training. We observe that ER and DER++ are strong competitors and can achieve significant improvements over the OGD strategy. However, such methods still cannot perform well under multiple task switching (S-Abrupt). Furthermore, the absence of a clear task boundary (S-Gradua) poses an even more challenging challenge, increasing the error of most models. On the other hand, FSNet shows promising results on all datasets, outperforming most competitive baselines across different prediction horizons. Furthermore, this improvement is significant against synthetic benchmarks, demonstrating that LSFNet can rapidly adapt to non-stationary environments and recall prior knowledge, even in the absence of a clear task boundary.

[0059] Figure 8 reports the convergence behavior of the considered methods. The results demonstrate the advantage of ER due to faster convergence during training compared to OGD. However, it is important to note that memorizing the original data may not be applicable to many domains. For S-Abrupt, most baselines demonstrate an inability to quickly recover from concept drift, as indicated by the increasing error curves. We also observe promising results for FSNet on most datasets, with significant improvements over the baselines on the ETT, WTH, and S-Abrupt datasets. The ECL dataset is more challenging due to missing values ​​(Li et al., 2019) and large intra- and inter-dimensional variation, which may require computing better data normalization. While FSNet achieved promising results on ECL, addressing the challenges mentioned above can further improve its performance. Overall, the results highlight the challenges of online time series forecasting and demonstrate promising results for FSNet.

[0060] The model's prediction quality for S-Abrupt is visualized as shown in Figure 8, since it is a univariate time series. The remaining real-world dataset is multivariate and difficult to visualize. In particular, the model's predictions at two time points are plotted: t = 900 and t = 5900, the end of training, in Figure 9. With limited samples per task and multiple concept drifts, standard online optimization collapsed into a naive solution that predicted random noise around zero. However, FSNet was able to successfully capture the time series patterns and provide better predictions.

[0061] This description and the accompanying drawings illustrating inventive aspects, embodiments, implementations, or applications should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the specification and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the present disclosure. Like numbers in two or more figures represent the same or similar elements.

[0062] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and are not meant to be limiting. Those skilled in the art may recognize other elements not specifically described herein that are within the scope and spirit of the present disclosure. Additionally, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless specifically described otherwise, or if one or more features render the embodiment non-functional.

[0063] While illustrative embodiments have been shown and described, a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some instances, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, substitutions, and modifications. Accordingly, the scope of the present invention is to be limited only by the claims that follow, which claims are appropriately construed broadly and consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A method for predicting time series data at future timestamps in a dynamic system, the method being computer-implemented and comprising: receiving, via a data interface, a time series data set including a plurality of data points corresponding to a plurality of timestamps within a look-back time window; Calculating a first gradient at a first convolutional layer of a stack of convolutional layers based on an exponential moving average of gradients corresponding to the first convolutional layer; determining first adaptation parameters corresponding to the first convolutional layer based on a mapping of portions of the first gradient to elements of the first adaptation parameters; computing an adapted feature map based at least in part on the first adaptation parameters and a previous adapted feature map from a preceding convolutional layer; generating, via a regressor, time series forecast data corresponding to a future time window based on the final feature map output from the stack of convolutional layers corresponding to the time series data in the lookback time window; calculating a prediction loss based on the generated time series prediction data corresponding to the future time window and ground truth data; updating the stack of convolutional layers based on the predicted loss via backpropagation; and A method comprising:

2. The method of claim 1 , wherein the first adaptation parameters include a first weight adaptation component and a first feature adaptation component.

3. For at least one convolutional layer of the temporal convolutional neural network, determining a layer prediction loss based on the plurality of data points, the layer prediction loss indicating a loss contribution of each convolutional layer to an overall prediction loss according to the plurality of data points; updating the at least one convolutional layer based on the layer prediction loss; and The method of claim 2 further comprising:

4. Calculating a cosine similarity between the first gradient of the updated convolutional layer and a long-term gradient associated with the at least one convolutional layer; In response to determining that the cosine similarity is greater than a predefined threshold, Retrieving current adaptation parameters from an indexed memory corresponding to the first convolutional layer; and updating the content stored in the indexed memory based on the current adaptation parameters and the first adaptation parameters; updating the first adaptive parameter by taking a weighted average with the retrieved current adaptive parameter; The method of claim 3 further comprising:

5. calculating adapted layer parameters based on generating first adapted weights based on the first weight adaptation component and layer parameters corresponding to the first layer; generating a feature map of the first convolutional layer using the first feature adaptation component; and The method of claim 4 further comprising:

6. 6. The method of claim 5, wherein the adapted feature map is computed based on the first feature adaptation component and a first feature map of the first convolutional layer, the first feature map being a convolution of the adapted layer parameters with a previous adapted feature map from a previous layer.

7. The stack of convolutional layers and the regressor are updating the regressors via stochastic gradient descent; updating the first gradient and the first adaptive parameter in the first convolutional layer; Updated by The method of claim 6.

8. In response to determining that the cosine similarity is greater than a predefined threshold, Triggering a memory read or write operation that captures the current pattern of the gradient The method of claim 4 further comprising:

9. The current pattern is calculating attention based on the current contents of the memory and current adaptation parameters; selecting a set of highly relevant attentions from the calculated attentions; and updating the current adaptation parameters by taking a weighted sum of the current contents of the memory weighted by the relevant attention set; The method of claim 8 , wherein the

10. performing a write operation to update and store the current contents of the memory based on the updated current adaptive parameters; 10. The method of claim 9, further comprising:

11. 1. A system for predicting time series data at future timestamps in a dynamic system, comprising: a data interface for receiving a time series data set including a plurality of data points corresponding to a plurality of timestamps within a lookback time window; a memory storing a plurality of processor-executable instructions; a processor that reads the instructions from the memory and executes the instructions to perform an operation; and the operation comprises: Calculating a first gradient at a first convolutional layer of a stack of convolutional layers based on an exponential moving average of gradients corresponding to the first convolutional layer; determining first adaptation parameters corresponding to the first convolutional layer based on a mapping of portions of the first gradient to elements of the first adaptation parameters; computing an adapted feature map based at least in part on the first adaptation parameters and a previous adapted feature map from a preceding convolutional layer; generating, via a regressor, time series forecast data corresponding to a future time window based on the final feature map output from the stack of convolutional layers corresponding to the time series data in the lookback time window; calculating a prediction loss based on the generated time series prediction data corresponding to the future time window and ground truth data; updating the stack of convolutional layers based on the predicted loss via backpropagation; and Including, the system.

12. The system of claim 11 , wherein the first adaptation parameters include a first weight adaptation component and a first feature adaptation component.

13. The operation is For at least one convolutional layer of the temporal convolutional neural network, determining a layer prediction loss based on the plurality of data points, the layer prediction loss indicating a loss contribution of each convolutional layer to an overall prediction loss according to the plurality of data points; updating the at least one convolutional layer based on the layer prediction loss; and The system of claim 12 further comprising:

14. The operation is Calculating a cosine similarity between the first gradient of the updated convolutional layer and a long-term gradient associated with the at least one convolutional layer; In response to determining that the cosine similarity is greater than a predefined threshold, Retrieving current adaptation parameters from an indexed memory corresponding to the first convolutional layer; and updating the content stored in the indexed memory based on the current adaptation parameters and the first adaptation parameters; updating the first adaptive parameter by taking a weighted average with the retrieved current adaptive parameter; The system of claim 13 further comprising:

15. The operation is calculating adapted layer parameters based on generating first adapted weights based on the first weight adaptation component and layer parameters corresponding to the first layer; generating a feature map of the first convolutional layer using the first feature adaptation component; and The system of claim 14 further comprising:

16. 16. The system of claim 15, wherein the adapted feature map is computed based on the first feature adaptation component and a first feature map of the first convolutional layer, the first feature map being a convolution of the adapted layer parameters and a previous adapted feature map from a previous layer.

17. The stack of convolutional layers and the regressor are updating the regressors via stochastic gradient descent; updating the first gradient and the first adaptive parameter in the first convolutional layer; The system of claim 16, wherein the update is performed by

18. The operation is In response to determining that the cosine similarity is greater than a predefined threshold, Triggering a memory read or write operation that captures the current pattern of the gradient The system of claim 14 further comprising:

19. The current pattern is calculating attention based on the current contents of the memory and current adaptation parameters; selecting a set of highly relevant attentions from the calculated attentions; and updating the current adaptation parameters by taking a weighted sum of the current contents of the memory weighted by the relevant attention set; 20. The system of claim 18, wherein the

20. 1. A non-transitory processor-readable storage medium storing processor-readable instructions for predicting time series data at future timestamps in a dynamic system, the instructions being executed by a processor to perform operations, the operations comprising: receiving, via a data interface, a time series data set including a plurality of data points corresponding to a plurality of timestamps within a look-back time window; Calculating a first gradient at a first convolutional layer of a stack of convolutional layers based on an exponential moving average of gradients corresponding to the first convolutional layer; determining first adaptation parameters corresponding to the first convolutional layer based on a mapping of portions of the first gradient to elements of the first adaptation parameters; computing an adapted feature map based at least in part on the first adaptation parameters and a previous adapted feature map from a preceding convolutional layer; generating, via a regressor, time series forecast data corresponding to a future time window based on the final feature map output from the stack of convolutional layers corresponding to the time series data in the lookback time window; calculating a prediction loss based on the generated time series prediction data corresponding to the future time window and ground truth data; updating the stack of convolutional layers based on the predicted loss via backpropagation; and 1. A non-transitory processor-readable storage medium comprising:

Images