Systems and methods for online time series forecasting

Abstract

An embodiment provides a combined fast learning network and slow learning network framework (called "FSNet") to train deep neural predictors on-the-fly for online time series prediction. FSNet is built on a deep neural network backbone (a slow learner) with two complementary components to facilitate fast adaptation to both new and recurrent concepts. To this end, FSNet employs a layer-wise adapter to monitor each layer's contribution to the prediction loss via its partial derivatives. The adapter transforms each layer's weights and features at each step based on its recent gradients, enabling fast adaptation of layer-wise granularity to optimize the current loss. In addition, FSNet employs a second complementary associative memory component to store important recurring patterns observed during training. The adapter interacts with the memory to store, update, and retrieve previous transformations, facilitating fast learning of such patterns.

Description

[Technical field]

[0001] [Cross reference] This disclosure claims priority to U.S. Non-provisional Patent Application No. 17 / 871,819, filed July 22, 2022, and U.S. Provisional Patent Application No. 63 / 305,145, filed January 31, 2022, each of which is expressly incorporated by reference in its entirety.

[0002] [Technical field] FIELD OF THE DISCLOSURE Embodiments relate generally to machine learning systems, and more particularly to online time series forecasting. [Background technology]

[0003] Deep neural network models are widely used in time series forecasting. For example, the learned models can be used to forecast time series data such as continuous market data, weather data, and / or the like over a period of the future. Existing deep models employ batch learning for time series forecasting tasks. Such models randomly sample lookback and forecast windows during training and often freeze the model during evaluation, destroying the time-varying (non-stationary) nature of the time series.

[0004] Therefore, there is a need for an efficient and adaptive deep learning framework for online temporal prediction. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a simplified diagram illustrating an example structure of the FSNet framework for forecasting time series, according to embodiments described herein. [Diagram 2] 2 is a simplified diagram illustrating an example structure of a TCN layer (block) of the FSNet framework depicted in FIG. 1 according to an embodiment described herein. [Diagram 3]FIG. 3 is a simplified diagram illustrating an example structure of an extended convolutional layer in the TCN layer (block) depicted in FIG. 2 according to embodiments described herein. [Figure 4] 1 is a simplified diagram of a computing device implementing the FSNet framework according to some embodiments described herein. [Diagram 5] 4 is a simplified pseudocode segment for a fast learning network and a slow learning network implemented in the FSNet framework described in FIGS. 1-3 according to an embodiment described herein. [Figure 6] 6 is a simplified logic flow diagram illustrating an example process corresponding to the pseudo-code algorithm of FIG. 5, according to embodiments described herein. [Figure 7] 13A-13C are exemplary data charts and plots illustrating the performance of FSNet in exemplary data experiments, in accordance with embodiments described herein. [Figure 8] 13A-13C are exemplary data charts and plots illustrating the performance of FSNet in exemplary data experiments, in accordance with embodiments described herein. [Figure 9] 13A-13C are exemplary data charts and plots illustrating the performance of FSNet in exemplary data experiments, in accordance with embodiments described herein.

[0006] In the figures, elements with the same name have the same or similar functions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] As used herein, the term "network" may comprise any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or in conjunction therewith.

[0008] As used herein, the term "module" may comprise a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0009] A time series is a set of values ​​corresponding to a parameter of interest at different times. Examples of parameters can be stock prices, temperature measurements, etc. Time series forecasting is the process of determining a future data point or set of future data points beyond the set of values ​​in a time series. Time series forecasting for dynamic data with deep learning remains challenging.

[0010] An embodiment provides a combined fast learning network and slow learning network framework (called "FSNet") to train deep neural predictors on-the-fly for online time series prediction. FSNet is built on a deep neural network backbone (a slow learner) with two complementary components to facilitate fast adaptation to both new and recurrent concepts. To this end, FSNet employs a layer-wise adapter to monitor each layer's contribution to the prediction loss via its partial derivatives. The adapter transforms each layer's weights and features at each step based on its recent gradients, enabling fast adaptation of layer-wise granularity to optimize the current loss. In addition, FSNet employs a second complementary associative memory component to memorize important recurring patterns observed during training. The adapter interacts with the memory to store, update, and retrieve previous transformations to facilitate fast learning of such patterns.

[0011] In this way, the FSNet framework can adapt to fast-changing and long-term recurring patterns in time series.Specifically, in FSNet, the deep neural network plays the role of the neocortex, while the adapter and its memory act as the hippocampal component. Overview of the FSNet Framework

[0012] FIG. 1 is a simplified diagram illustrating an example structure of an FSNet framework 100 for forecasting time series, according to embodiments described herein.

[0013] The FSNet framework 100 includes a number of convolution blocks 104a-n connected to a regressor 105. The FSNet framework 100 receives a time series of T observations, each having n dimensions, from an input interface such as a memory or a network adapter.

number

[0014] In one embodiment, the FSNet framework 100 includes a parameter

number

[0015] Based on the TCN backbone 104a-n, the FSNet framework 100 provides a layer-by-layer adapter φ for each TCN layer 104a-n. 1 (shown in FIG. 3 as 315) and a layer-by-layer associative memory M for each TCN layer 104a-n. 1 (shown as 318 in FIG. 3). Thus, the total trainable parameters of the framework are l ,φ l}, and all associative memories are l} l=1,...,L It is.

[0016] Figure 2 is a simplified diagram illustrating an example structure of a TCN layer (block) 104a of the FSNet framework depicted in Figure 1, according to an embodiment described herein. In each TCN layer (block), e.g., 104a, a block input 202 may be processed by several extended convolution layers 204, 206, and the convolution output is added to the original block input 202 to generate a block output 208. It should be noted that although two extended convolution layers 204 and 206 are shown in Figure 2 for illustrative purposes only, any other number of extended convolution layers may be used in a TCN block.

[0017] In one embodiment, each TCN block 104a may rely on its adapter 315 and associative memory 318 to adapt quickly to changes in the time series data 102 or to learn more efficiently with limited data. Each block or layer 104a-104n may adapt independently, rather than restricting adaptation to the depth of the network, i.e., gradient descent over the depth of the network 104a-n. The partial derivatives of each layer 104a-n

number

number

number

[0018] Thus, each convolutional filter stack is accompanied by an adapter and an associative memory. As further shown in relation to FIG. 3, at each layer, the adapter receives the gradient EMA and interacts with the memory and the convolutional filters accordingly.

[0019] 3 is a simplified diagram illustrating an example structure of the extended convolutional layer 204 (or 206) in the TCN layer (block) 104a in FIG. 2 according to an embodiment described herein. The extended convolutional layer 204 may comprise a convolutional filter 310, a per-layer adapter 315, and a per-layer memory 318. The input 202 to the extended convolutional layer 204 may be fed to the convolutional filter 310, which calculates an exponential moving average (EMA) 313 of the gradient of the TCN backbone. Specifically, since the gradient of a single sample may vary widely and introduce noise into the adaptation parameters, the EMA is used to smooth out the noise of the online training by:

number

[0020] In some embodiments, the fast adapter 315 may use an element-wise transform as the adaptation process due to its efficiency for continuous learning. The resulting adaptation parameter u l 316 is u l =[α l ;β l ], (i) a weight adaptation parameter α l and (ii) the feature adaptation parameter β l In some embodiments, the high-speed adapter 315 may define the bias conversion parameter as α 1 can be absorbed into

[0021] In one embodiment, the layer θ l The adaptation for may include weight adaptation and feature adaptation, as shown in 319. First, the weight adaptation parameter α loperates on the corresponding weights of the backbone network via element-wise multiplication as follows:

number

number

[0022] Similarly, the feature adaptation component of the gradient, β l and the feature adaptation parameter modifies the convolutional layer feature map based on element-wise multiplication between the feature adaptation component and the first convolutional layer feature map. For example, feature adaptation β l Also, the output feature map h l and produces output 322 as follows:

number

[0023] In this way, the convolution layer θ l is the weight adaptation component α l and feature adaptation component β l The data may be updated based on the above.

[0024] In some embodiments, the gradients may be directly mapped to element-wise adaptive parameters, which may result in very high-dimensional mappings.

[0025] In some embodiments, Ω(·;Φ l), to divide the gradient into equal-sized chunks and then map each chunk to an element of the adaptation parameters. Specifically, the chunking operation may be implemented as (1) flattening the gradient EMA of the corresponding block of the TCN model 120 into a vector, (2) splitting the gradient vector into d chunks, (3) mapping each chunk to a hidden representation, and (4) mapping each hidden representation to the coordinates of the target adaptation parameter u. For example, using a vectorization operation (vec(·)) that flattens a tensor into a vector, and a partitioning operation (e,B) that partitions a vector e into B segments, each with size dim(e) / B, the EMA gradient of the backbone layer 313 of the TCN backbone can be transformed through the chunking process into the adaptation coefficients

number

number

[0026] In summary:

number

number

[0027] In one embodiment, in a time series, old patterns may reappear in the future, and it is beneficial to recall similar knowledge from the past to further facilitate learning. Storing the original data can alleviate this problem, but due to privacy concerns, this may not be applicable in many domains. Therefore, an associative memory 318 may be implemented to store adaptation coefficients for recurrent events encountered during learning. Although the adapter 315 can handle fast recent changes over short time scales, recursive patterns are stored in the memory 318 and retrieved when they reappear in the future. To this end, each adapter 315:

number

[0028] Specifically, since interacting with the memory 318 at every step is expensive and susceptible to noise, memory interaction may be triggered only when a substantial change in the representation is detected. The interference between the current and past representations can be characterized by the dot product between the gradients. Thus, the cosine similarity between the recent gradient and the long-term gradient may be calculated and monitored to trigger memory interaction when the interference falls below a threshold, which may indicate that the pattern has changed significantly. To this end, the gradient EMA

number

number

number

[0029] In one embodiment, when the current adaptive parameters may not capture the entire event, which may span several samples, memory read and write operations may be performed using the EMA of the adaptive parameters (with coefficient γ′) to completely capture the current pattern. l The EMA of g ^ l When a memory interaction is triggered, the adapter queries and retrieves the most similar past transformation via an attention read operation that is a weighted sum over memory items:

number

number

number

number

number

[0030] In one embodiment, the FSNet framework described in connection with Figures 1-3 is suitable for task-free online continuous learning scenarios since it does not need to explicitly detect when a task switches. Instead, the task boundary definition can be relaxed to allow the model to continuously improve its learning on the current sample. Computing Environment

[0031] 4 is a simplified diagram of a computing device implementing the FSNet framework according to some embodiments described herein. As shown in FIG. 4, the computing device 400 includes a processor 410 coupled to a memory 420. The operation of the computing device 400 is controlled by the processor 410. Although the computing device 400 is shown with only one processor 410, it is understood that the processor 410 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and / or the like within the computing device 400. The computing device 400 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0032] The memory 420 may be used to store software executed by the computing device 400 and / or one or more data structures used during operation of the computing device 400. The memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media may include a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read from by a processor or computer.

[0033] The processor 410 and / or memory 420 may be arranged in any suitable physical arrangement. In some embodiments, the processor 410 and / or memory 420 may be implemented on the same substrate, in the same package (e.g., a system in package), on the same chip (e.g., a system on a chip), etc. In some embodiments, the processor 410 and / or memory 420 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 410 and / or memory 420 may be located within one or more data centers and / or cloud computing facilities.

[0034] In some examples, memory 420 may include a non-transitory tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, memory 420 includes instructions for an online time series prediction module 430 that may be used to implement and / or emulate the systems and models and / or to implement any of the methods described further herein. In some examples, online time series prediction module 430 may receive input 440, such as time series data in a lookback window, via a data interface 415. Data interface 415 may be either a user interface that receives uploaded time series data or a communication interface that may receive or retrieve previously stored samples of the lookback and prediction windows from a database. Time series prediction module 430 may generate output 450, such as a prediction for input 440.

[0035] In some embodiments, the time series prediction module 430 may further include a set of TCN blocks 431a-n (similar to 104a-n shown in FIG. 1) and a regressor 432 (similar to 105 shown in FIG. 1). In one implementation, the time series prediction module 430 and its sub-modules 431-432 may be implemented via software, hardware, and / or a combination thereof.

[0036] Some examples of computing devices, such as computing device 400, may include a non-transitory tangible machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform the processes of the methods described throughout this disclosure. Some common forms of machine-readable media that may include the processes of the methods are, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card, a paper tape, any other physical medium having a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read from by a processor or a computer. Workflow example

[0037] 5 is a simplified pseudocode segment for fast and slow learning networks implemented in the FSNet framework described in FIGS. 1-3, according to an embodiment described herein. For example, for a stack of L layers (e.g., 104a-n in FIG. 1), the weight adaptation components α l and feature adaptation component β l A forward calculation may be performed to calculate the adaptation parameters including: The memory read and write operations may be performed via a chunking process, and the adaptation parameters may be updated by a weighted sum of the current and past adaptation parameters.

[0038] Weight adaptation and feature adaptation may then be performed according to equation (5). After updating the adaptation parameters by forward calculation over the L layers, predicted data may be generated via the regressor (e.g., 105 in FIG. 1). The predicted data is then compared with the ground truth future data from the training samples to calculate a prediction loss, and the prediction loss is then used to update the stack of L layers via backpropagation. The regressor may also be updated via stochastic gradient descent (SGD). Then, the adaptation parameters and EMA adaptation parameters are updated backwards.

[0039] 6 is a simplified logic flow diagram illustrating an example process 600 corresponding to the pseudo-code algorithm of FIG. 5, according to embodiments described herein. One or more of the processes of method 600 may be implemented, at least in part, in the form of executable code stored on a non-transitory tangible machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes. In some embodiments, method 600 corresponds to the operation of FSNet framework 100 (FIG. 1) for forecasting time series data at future timestamps in a dynamic system.

[0040] In step 602, a time series data set including a number of data points corresponding to a number of timestamps within a look-back time window (e.g., 102 in FIG. 1) may be received via a data interface (e.g., 415 in FIG. 4).

[0041] In step 604, a convolutional layer (e.g., block 104a in FIGS. 1-2) from a stack of convolutional layers (e.g., blocks 104a-n in FIGS. 1 ) may calculate a first gradient based on an exponential moving average of the gradients corresponding to the respective convolutional layer, for example, according to equation (1).

[0042] In step 606, a first adaptation parameter u corresponding to the convolution layer is calculated. 1may be determined by mapping portions of the first gradient to elements of a first adaptation parameter. For example, the first adaptation parameter may be a first weighted adaptation component α l and the first feature adaptation component β l Includes.

[0043] In step 608, for at least one convolutional layer of the temporal convolutional neural network, a layer prediction loss indicating a loss contribution of each convolutional layer to an overall prediction loss according to the plurality of data points may be optionally determined based on the plurality of data points. For example, the layer prediction loss may be determined based on the plurality of data points as a partial derivative

number

[0044] At least one convolutional layer may optionally be updated based on the layer prediction loss in step 610. In this manner, each layer may be independently monitored and modified to learn the current loss by learning through the layer prediction loss.

[0045] In step 612, a cosine similarity between the first gradient of the updated convolutional layer and a long-term gradient associated with the at least one first convolutional layer may be calculated, for example, according to equation (6).

[0046] In step 614, if the cosine similarity is greater than a predefined threshold, the method 600 proceeds to step 616, where it performs a chunking process for memory reads and writes. Specifically, in step 616, the current adaptive parameters are retrieved from the indexed memory (e.g., 318 in FIG. 3) corresponding to the convolutional layer. In step 618, the contents stored in the indexed memory (e.g., 318 in FIG. 3) are updated based on the current adaptive parameters and the first adaptive parameters. In step 620, the first adaptive parameters are updated by taking a weighted average with the retrieved current adaptive parameters.

[0047] In step 622, a first weight adaptation component α is calculated, for example according to equation (5). l and the layer parameter θ corresponding to the first layer l Based on the layer parameters

number

[0048] In step 624, the feature map h of the first convolutional layer is l The first feature adaptation component β l For example, the first feature map is a convolution of the adapted layer parameters with the previous adapted feature map from the previous layer. In step 626, the first feature adaptation component β l and the first feature map h of the first convolutional layer l Based on and, the adapted feature map

number

[0049] In step 628, a regressor (e.g., 105 in FIG. 1 ) may generate time series forecast data corresponding to a future time window based on the final feature map output from the stack of convolutional layers corresponding to the time series data in the lookback time window.

[0050] In step 630, a prediction loss may be calculated based on the generated time series prediction data and ground truth data corresponding to a future time window.

[0051] The stack of convolutional layers and the regressors may then be updated based on the predicted loss via backpropagation. In step 632, the regressors may be updated via stochastic gradient descent. Then, in step 634, the gradients and adaptation parameters of each layer of the stack may be updated backwards. Performance Examples

[0052] We conducted data experiments to test the following hypotheses: (i) FSNet facilitates faster adaptation to both new and recursive concepts compared to existing strategies; (ii) FSNet achieves faster and better convergence than other methods; and (iii) modeling partial derivatives is a key ingredient for fast adaptation.

[0053] Specifically, a wide range of time-series forecasting datasets were used for data experiments: (i) ETT1 (Zhou et al., Informer: Beyond efficient transformer for long sequence time-series forecasting, in Proceedings of AAAI, 2021) records the target value of "oil temperature" and six electricity load features over a two-year period. The ETTh2 and ETTm1 benchmarks are used, with observations recorded at hourly and 15-minute intervals, respectively. (ii) The ECL (Electricity Consuming Load) 2 dataset collects electricity consumption of 321 clients from 2012 to 2014. (iii) The Traffic 3 dataset records road occupancy rates on freeways in the San Francisco Bay Area. (iv) The Weather 4 dataset records 11 weather features for approximately 1,600 locations in the United States at hourly intervals from 2010 to 2013.

[0054] In addition, two synthetic data sets are constructed to explicitly test the model's ability to deal with new and recurrent concept drift. Tasks can be synthesized by sampling 1,000 samples from a first-order autoregressive process ARφ(1) with coefficient φ, where different tasks correspond to different φ values. The first synthetic data, S-Abrupt, contains abrupt and recurrent concepts where the samples suddenly switch from one AR process to another in the following order: AR0.1(1), AR0.4(1), AR0.6(1), AR0.1(1), AR0.3(1), AR0.6(1). The second data, S-Gradual, contains a step-by-step, gradual shift, starting from the last 20% of each task. In this scenario, the last 20% samples of the task are averaged from the two AR processes in the order as above.

[0055] In the implementation, the data is split into a warm-up phase and an online training phase in a ratio of 25:75, and we consider the TCN backbone for the experiments, except for the Informer baseline. 2Optimization details in Zhang et al., Informer: Beyond efficient transformer for long sequence time-series forecasting, in Proceedings of AAAI, 2021, by optimizing the mean squared error (MSE) loss. Both epoch and batch size are set to 1 to follow the online learning setting. A fair comparison is performed by ensuring that all baselines use the same total memory budget as FSNet, which includes three times the network size, i.e., one working model and two EMAs of its gradients. Thus, for ER, MIR, and DER++, we use episodic memory to memorize previous samples to meet this budget. For the remaining baselines, the backbone size can be increased instead. Finally, in the warm-up phase, the mean and standard deviation are calculated to normalize the online training samples and perform hyperparameter cross-validation. For all benchmarks, the lookback window length is set to 60 and the forecast horizon is set to H = 1. We test the model's ability to forecast longer horizons by varying H ∈ {1, 24, 48}.

[0056] For comparison, sequential training from both continuous learning and time series forecasting are adopted. The first is the OnlineTCN strategy (described in Zinkevich, Online convex programming and generalized infinitesimal gradient ascent, in Proceedings of the 20th international conference on machine learning (icml-03), pages 928-936,461, 2003), which simply trains continuously. The second is the Experience Replay (ER) strategy (described in Lin, Self-improving reactive agents based on reinforcement learning, planning and teaching, Machine learning, 8(3-4):293-321, 1992), where a buffer is adopted to remember previous data and interleave older samples during the learning of newer samples. Three recent advanced variants of ER: First, TFCL (Aljundi et al., Task-free continual learning, in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 11254-11263,325, 2019) introduces a task boundary detection mechanism and a knowledge integration strategy by regularizing the output of the network. Second, MIR (Aljundi et al., Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems, 32:11849-11860, 2019) replaces random sampling in ER with selecting the sample that causes the most forgetting.Finally, DER++ (Buzzega et al., Dark experience for general continual learning: a strong, simple baseline, in 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020) enhances the standard ER with a knowledge distillation strategy (described in Hinton et al., Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015). ER and its variants are powerful baselines in online settings because they benefit from training on mini-batches, which significantly reduces noise from single samples and provides faster and better convergence (see Bottou et al., Online learning and stochastic approximations, Online learning in neural networks, 17(9):142, 1998). The aforementioned baselines use the TCN backbone, Informer, but also include time series prediction methods based on the Transformer architecture (Vaswani et al., Attention is all you need. Advances in neural information processing systems, 30, 2017).

[0057] First, we simply train the Online Gradient Descent (OGD) (Zinkevich, Online convex and generalized infinitesimal gradient ascent, in proceedings of the 20 thinternational conference on machine learning, pp. 928-936, 2003). A large variant of OGD, OGD(L), with twice as many TCN filters per layer is also included, resulting in roughly twice the number of parameters5. Another baseline includes the Experiment Repla (Chaudhry et al., On tiny episodic memories in continual learning, arXiv preprint arXiv:1902.10486, 2019) strategy, where a buffer is employed to remember previous data and interleave older samples during the learning of newer samples. Another baseline includes DER++ (Buzzega et al., Dark experience for general continual learning: a strong, simple baseline, in proceedings of 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020), which further adds knowledge distillation (described in Hinton et al., Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531, 2015) loss to ER. ER and DER++ are strong baselines in the online setting because they benefit from training in mini-batches, which significantly reduces noise from single samples and provides faster and better convergence.

[0058] Figure 7 reports the cumulative mean squared error (MSE) and mean absolute error (MAE) at the end of training. We observe that ER and DER++ are strong competitors and can achieve significant improvements over OGD strategies. However, such methods still cannot perform well under multiple task switching (S-Abrupt). Moreover, the absence of clear task boundaries (S-Gradua) creates even more challenging problems and increases the error of most models. On the other hand, FSNet shows promising results on all datasets and outperforms most competitive baselines across different prediction horizons. Moreover, the improvement is significant over synthetic benchmarks, indicating that LSFNet is able to rapidly adapt to non-stationary environments and recall prior knowledge, even in the absence of clear task boundaries.

[0059] Figure 8 reports the convergence behavior for the considered methods. The results show the advantage of ER due to faster convergence during training compared to OGD. However, it is important to note that memorizing the original data may not be applicable in many domains. In S-Abrupt, most baselines demonstrate the inability to quickly recover from concept drift, as indicated by the increasing error curves. We also observe promising results for FSNet on most datasets, with significant improvements over baselines on ETT, WTH, and S-Abrupt datasets. The ECL dataset is more challenging due to missing values ​​(Li et al., 2019) and high variation within and across dimensions, which may require computing better data normalization. Although FSNet achieved promising results against ECL, its performance can be further improved by addressing the challenges mentioned above. Overall, the results shed light on the challenges of online time series forecasting and demonstrate promising results for FSNet.

[0060] The model's prediction quality on S-Abrupt is visualized as shown in Figure 8 since it is a univariate time series. The remaining real-world dataset is multivariate and difficult to visualize. In particular, the model's predictions at two time points are plotted, namely, t=900 in Figure 9 and t=5900, the end of training. With limited samples per task and multiple concept drifts, standard online optimization collapsed to a naive solution that predicts random noise around zero. However, FSNet is able to capture the time series patterns well and provide better predictions.

[0061] This description and the accompanying drawings illustrating inventive aspects, embodiments, implementations, or applications should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the present disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0062] In this description, specific details are described that describe some embodiments consistent with the present disclosure. Numerous specific details are described to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and are not meant to be limiting. One skilled in the art may recognize other elements that are not specifically described herein but are within the scope and spirit of the present disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless specifically described otherwise or if one or more features render the embodiment non-functional.

[0063] Although illustrative embodiments have been shown and described, a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the scope of the following claims, which claims are appropriately construed broadly and consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A method for predicting time series data at future timestamps in a dynamic system, comprising: receiving, via a data interface, a time series data set including a plurality of data points corresponding to a plurality of timestamps within a lookback time window; Calculating a first gradient at a first convolutional layer of a stack of convolutional layers based on an exponential moving average of gradients corresponding to the first convolutional layer; determining first adaptation parameters corresponding to the first convolutional layer based on a mapping of portions of the first gradient to elements of the first adaptation parameters; computing an adapted feature map based at least in part on the first adaptation parameters and a previous adapted feature map from a preceding convolutional layer; generating, via a regressor, time series forecast data corresponding to a future time window based on a final feature map output from the stack of convolutional layers corresponding to the time series data in the lookback time window; calculating a prediction loss based on the generated time series forecast data and ground truth data corresponding to the future time window; updating the stack of convolutional layers based on the prediction loss via backpropagation; and The method includes:

2. The method of claim 1 , wherein the first adaptation parameters include a first weight adaptation component and a first feature adaptation component.

3. For at least one convolutional layer of the temporal convolutional neural network, determining a layer prediction loss based on the plurality of data points, the layer prediction loss indicating a loss contribution of each convolutional layer to an overall prediction loss according to the plurality of data points; updating the at least one convolutional layer based on the layer prediction loss; and The method of claim 2.

4. Calculating a cosine similarity between the first gradient of the updated convolutional layer and a long-term gradient associated with the at least one convolutional layer; in response to determining that the cosine similarity is greater than a predefined threshold; Retrieving current adaptation parameters from an indexed memory corresponding to the first convolutional layer; and updating content stored in the indexed memory based on the current adaptation parameters and the first adaptation parameters; updating the first adaptive parameter by taking a weighted average with the retrieved current adaptive parameter; The method of claim 3 further comprising:

5. calculating adapted layer parameters based on the first weight adaptation component and generating first adapted weights based on layer parameters corresponding to the first layer; generating a feature map for the first convolutional layer using the first feature adaptation component; The method of claim 4 further comprising:

6. 6. The method of claim 5, wherein the adapted feature map is computed based on the first feature adaptation component and a first feature map of the first convolutional layer, the first feature map being a convolution of the adapted layer parameters and a previous adapted feature map from a previous layer.

7. The stack of convolutional layers and the regressors are updating the regressors via stochastic gradient descent; updating the first gradient and the first adaptation parameter in the first convolutional layer; Updated by The method according to claim 6.

8. in response to determining that the cosine similarity is greater than a predefined threshold; Triggering a memory read or write operation that captures the current pattern of the gradient. The method of claim 4 further comprising:

9. The current pattern is: calculating an attention based on a current content of the memory and current adaptation parameters; selecting a set of highly relevant attentions from the calculated attentions; and updating the current adaptation parameters by taking a weighted sum of the current contents of the memory weighted by the highly relevant attention set; The method of claim 8 , wherein the first and second inputs are captured by:

10. performing a write operation to update and store the current contents of the memory based on the updated current adaptation parameters; The method of claim 9 further comprising:

11. 1. A system for predicting time series data at future timestamps in a dynamic system, comprising: a data interface for receiving a time series data set including a plurality of data points corresponding to a plurality of timestamps within a lookback time window; a memory storing a plurality of processor-executable instructions; a processor that reads the instructions from the memory and executes the instructions to perform an operation; and the operation comprises: Calculating a first gradient at a first convolutional layer of a stack of convolutional layers based on an exponential moving average of gradients corresponding to the first convolutional layer; determining first adaptation parameters corresponding to the first convolutional layer based on a mapping of portions of the first gradient to elements of the first adaptation parameters; computing an adapted feature map based at least in part on the first adaptation parameters and a previous adapted feature map from a preceding convolutional layer; generating, via a regressor, time series forecast data corresponding to a future time window based on a final feature map output from the stack of convolutional layers corresponding to the time series data in the lookback time window; calculating a prediction loss based on the generated time series forecast data and ground truth data corresponding to the future time window; updating the stack of convolutional layers based on the prediction loss via backpropagation; and Including, the system.

12. The system of claim 11 , wherein the first adaptation parameters include a first weight adaptation component and a first feature adaptation component.

13. The operation includes: For at least one convolutional layer of the temporal convolutional neural network, determining a layer prediction loss based on the plurality of data points, the layer prediction loss indicating a loss contribution of each convolutional layer to an overall prediction loss according to the plurality of data points; updating the at least one convolutional layer based on the layer prediction loss; and The system of claim 12 further comprising:

14. The operation includes: Calculating a cosine similarity between the first gradient of the updated convolutional layer and a long-term gradient associated with the at least one convolutional layer; in response to determining that the cosine similarity is greater than a predefined threshold; Retrieving current adaptation parameters from an indexed memory corresponding to the first convolutional layer; and updating content stored in the indexed memory based on the current adaptation parameters and the first adaptation parameters; updating the first adaptive parameter by taking a weighted average with the retrieved current adaptive parameter; The system of claim 13 further comprising:

15. The operation includes: calculating adapted layer parameters based on the first weight adaptation component and generating first adapted weights based on layer parameters corresponding to the first layer; generating a feature map for the first convolutional layer using the first feature adaptation component; The system of claim 14 further comprising:

16. 16. The system of claim 15, wherein the adapted feature map is computed based on the first feature adaptation component and a first feature map of the first convolutional layer, the first feature map being a convolution of the adapted layer parameters and a previous adapted feature map from a previous layer.

17. The stack of convolutional layers and the regressors are updating the regressors via stochastic gradient descent; updating the first gradient and the first adaptation parameter in the first convolutional layer; The system of claim 16, wherein the update is performed by

18. The operation includes: in response to determining that the cosine similarity is greater than a predefined threshold; Triggering a memory read or write operation that captures the current pattern of the gradient. The system of claim 14 further comprising:

19. The current pattern is: calculating an attention based on a current content of the memory and current adaptation parameters; selecting a set of highly relevant attentions from the calculated attentions; and updating the current adaptation parameters by taking a weighted sum of the current contents of the memory weighted by the highly relevant attention set; 20. The system of claim 18, wherein the

20. 1. A non-transitory processor-readable storage medium storing processor-readable instructions for forecasting time series data at future timestamps in a dynamic system, the instructions being executed by a processor to perform operations including: receiving, via a data interface, a time series data set including a plurality of data points corresponding to a plurality of timestamps within a lookback time window; Calculating a first gradient at a first convolutional layer of a stack of convolutional layers based on an exponential moving average of gradients corresponding to the first convolutional layer; determining first adaptation parameters corresponding to the first convolutional layer based on a mapping of portions of the first gradient to elements of the first adaptation parameters; computing an adapted feature map based at least in part on the first adaptation parameters and a previous adapted feature map from a preceding convolutional layer; generating, via a regressor, time series forecast data corresponding to a future time window based on a final feature map output from the stack of convolutional layers corresponding to the time series data in the lookback time window; calculating a prediction loss based on the generated time series forecast data and ground truth data corresponding to the future time window; updating the stack of convolutional layers based on the prediction loss via backpropagation; and 16. A non-transitory processor-readable storage medium comprising:

Images