System and method for automatic forecast algorithm selection based on time series characteristics

Abstract

Systems, methods, and other embodiments associated with characteristic-based selection of time series forecasting algorithms are described. In one embodiment, a method includes analyzing each time series in a training set of time series to produce a characteristic vector for the time series. The method trains an autoencoder to minimize errors between (1) a bottleneck layer and the characteristic vector, and (2) an input layer and an output layer. The method generates a new characteristic vector that fills in gaps between adjacent characteristics. The method inputs the new characteristic vector to the bottleneck layer to generate a test set of time series. The method tests forecasting algorithms using the test set to find out forecasting errors. The method trains a ranking function to assign a rank to each forecasting algorithm based on the provided characteristic vector. And, the method automatically selects one of the forecasting algorithms to monitor an additional time series based on the ranks.

Description

Technical Field

[0001] This publication relates to machine learning, and more specifically, to autoencoder networks. Background Technology

[0002] Forecasting algorithms that predict expected data values ​​can be used to monitor time series of data values. Existing methods for selecting forecasting algorithms for a given time series are resource-intensive and may still result in inaccurate selections for that particular time series. Summary of the Invention

[0003] In one embodiment, this document presents one or more non-transitory computer-readable media. The one or more non-transitory computer-readable media includes computer-executable instructions stored thereon, which, when executed by at least a processor of a computer, cause the computer to perform the steps of a computer-implemented method. The computer-executable instructions cause a computer system to create a time series generator by: (1) accessing a first set of time series; (2) identifying N features for each time series in the first set of time series; (3) training an autoencoder with N nodes in a bottleneck layer based on a loss function that minimizes the difference between the bottleneck layer activation and the N features; and (4) setting the bottleneck layer as the input to the time series generator. The computer-executable instructions cause the computer system to determine gaps in the first set of time series in an N-dimensional feature space. The computer-executable instructions cause the computer system to input one or more feature vectors to fill the gaps into the bottleneck layer to produce a gap-filled set of time series. The computer-executable instructions cause the computer system to combine the first set of time series with the gap-filled set of time series to generate a combined set of time series. The computer-executable instructions cause the computer system to input the combined set of time series into multiple candidate prediction algorithms. Each candidate forecasting algorithm generates a forecast value based on a set of combinations of time series. Computer-executable instructions cause a computer system to determine the forecast error of the plurality of candidate forecasting algorithms based at least on the set of combinations of forecast values ​​and time series. The computer-executable instructions also cause the computer system to train a machine learning model based on the forecast error and N features to determine a ranking of the forecasting algorithms used to forecast a given time series. Furthermore, the computer-executable instructions cause the computer system to select one of the candidate forecasting algorithms based at least on the ranking to generate a prediction for the given time series.

[0004] In one embodiment, a computing system is presented herein. The computing system includes at least one processor connected to at least one memory, and a non-transitory computer-readable medium. The computer-readable medium includes instructions stored thereon, which, when executed by at least the processor, cause the computing system to perform steps of processing. The instructions cause the computing system to analyze each time series in a training set of time series to generate a vector of N features for each time series. The instructions cause the computing system to train an autoencoder using a loss function. The loss function minimizes: (1) the error between the bottleneck layer activation in the autoencoder and the vector of features of the time series, and (2) the error between the input layer and the output layer of the autoencoder. The instructions cause the computing system to generate one or more new vectors of N features by minimizing the gaps between neighboring points in an N-dimensional feature space. The instructions cause the computing system to generate a test set of time series based at least in part on inputting the new vectors of N features into the bottleneck layer of the trained autoencoder. The instructions cause the computing system to (1) input the test set of time series into each of a set of different prediction algorithms, and (2) compute the prediction error of each algorithm based on its performance for each time series. The instructions instruct the computational system to train a ranking function to assign a ranking to each prediction algorithm based on a vector of N provided features. Furthermore, the instructions instruct the computational system to automatically select one prediction algorithm to monitor the additional time series based on the N features processed by the ranking function.

[0005] In one embodiment, this document presents a computer-implemented method. The computer-implemented method includes processing multiple time series in a training set of time series to generate a vector of N features for each of the multiple time series. The computer-implemented method includes training an autoencoder based on a loss function. The loss function minimizes: (1) the difference between the activations of nodes in the bottleneck layer of the autoencoder and the vector of N features of the time series entered at the input layer of the autoencoder, and (2) the difference between the values ​​of the time series entered at the input layer and the values ​​at the output layer of the autoencoder. The computer-implemented method includes generating one or more new vectors of N features by minimizing the gaps between neighboring points in an N-dimensional feature space. The computer-implemented method includes generating a test set of time series based at least in part on inputting the new vectors of N features into the bottleneck layer of the trained autoencoder. The computer-implemented method includes (1) inputting the test set of the time series into each of a set of different prediction algorithms, and (2) calculating the prediction error of each algorithm based on its performance for each time series. The computer-implemented method includes training a ranking function to assign a ranking to each prediction algorithm based on the provided vector of N features. Furthermore, the computer-implemented method includes automatically selecting one of the forecasting algorithms to monitor the additional time series based on N characteristics processed by a ranking function.

[0006] In one embodiment, automatically selecting the forecasting algorithm best suited to monitor additional time series exhibiting specific characteristics offers a substantial advantage when monitoring of the additional time series is used to generate alarms in the processing control loop. Automatic selection of a forecasting algorithm that performs well for a given processing improves the predictive accuracy of the monitoring, thereby allowing for both: (1) earlier initiation of control actions and (2) more accurate control actions. For example, by automatically selecting a forecasting algorithm specifically adapted to equipment and site-specific characteristics in the time series data (such as readings from sensors configured to measure vibrations, temperatures, or other physical phenomena associated with the manufacturing equipment), the electrical controls of the equipment can be signaled more accurately and promptly to proactive adjustments to prevent equipment damage or product quality degradation. Alternatively, for example, by automatically selecting a forecasting algorithm specifically adapted to patient-specific characteristics in the time series data (such as readings from sensors configured to measure heart rate or other physical phenomena associated with the patient), the electrical controls of an intravenous pump can be signaled more accurately and promptly in real-time to proactive administration of precise doses in response to changes in the patient's condition. In other processing control loop applications, automatically selecting the forecasting algorithm best suited to monitor time series exhibiting specific characteristics can produce similar improvements. Attached Figure Description

[0007] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various systems, methods, and other embodiments of this disclosure. It will be appreciated that the element boundaries shown in the figures (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of a boundary. In some embodiments, one element may be implemented as multiple elements, or multiple elements may be implemented as one element. In some embodiments, an element shown as an inner component of another element may be implemented as an outer component, and vice versa. Furthermore, elements may not be drawn to scale.

[0008] Figure 1 An example of an algorithm selection system associated with the selection of feature-based time series forecasting algorithms is illustrated.

[0009] Figure 2 The illustration shows an example of an algorithm selection method associated with the selection of feature-based time series forecasting algorithms.

[0010] Figures 3A-3I An example of automatically and incrementally constructing a forecast algorithm selector at each stage of the construction process is shown.

[0011] Figure 4 The illustration shows an embodiment of a computing system configured with the disclosed example systems and / or methods. Detailed Implementation

[0012] This paper describes a system and method for providing an algorithm for automatically selecting a forecasting algorithm for a time series based on characteristics describing the behavior of the time series. In one embodiment, the algorithm selection system trains and modifies an autoencoder, and uses the autoencoder to train a classification model to select the forecasting model best suited for the time series based on its characteristics.

[0013] For example, the algorithm selection system configures an autoencoder to generate time series based on descriptive characteristics. Then, the algorithm selection system uses the autoencoder to generate the data based on a comprehensive set of characteristics. The algorithm selection system determines the performance of various forecasting algorithms relative to these characteristics. The algorithm selection system trains a machine learning model to rank the models by performance based on these characteristics. Finally, the algorithm selection system selects a model to monitor the time series based on the model's ML ranking.

[0014] In one embodiment, the algorithmic selection system outperforms methods such as Feature-Based Predictive Model Selection (FFORMS) because the algorithmic selection system described herein covers a broader range of domains not addressed by FFORMS. Therefore, advantageously, the algorithmic selection system is domain-agnostic.

[0015] It should be understood that no action or function described or claimed herein is performed by human thought. No action or function described or claimed herein can actually be performed by human thought. Any interpretation that any action or function described or claimed herein can be performed by human thought is inconsistent with and contrary to this disclosure.

[0016] -definition-

[0017] As used herein, the term "time series" refers to a data structure in which a series of data points or readings (such as observed or sampled values) are indexed in chronological order. In one embodiment, the data points of a time series can be indexed using indexes of time points described by timestamps and / or observation numbers. For example, a time series is a sequence of observations of a variable (such as heart rate, vibration, temperature, or other physical phenomenon) over time.

[0018] As used herein, the term "vector" refers to a data structure that includes a set of data points describing a particular entity. For example, a "characteristic vector" includes a set of data points describing various attributes of a time series. And, for example, a time series can be represented as a vector.

[0019] As used herein, when referring to a time series, the term "characteristic" refers to an attribute or feature that describes the behavior of the time series.

[0020] —Example Algorithm Selection System—

[0021] Figure 1 An embodiment of an algorithm selection system 100 associated with the selection of feature-based time series forecasting algorithms is illustrated. The algorithm selection system 100 includes a feature analyzer 105, an autoencoder trainer 110, a gap filler 115, a feature-based time series generator 120, a forecasting algorithm tester 125, a ranking function trainer 130, and a forecasting algorithm selector 135.

[0022] In one embodiment, feature analyzer 105 is configured to analyze each time series in the training set 140 of the time series to produce a vector 145 of N features for each time series. In one embodiment, feature analyzer 105 is configured to access the training set 140 of the time series, identify N features for each time series in the training set, and generate a vector 145 of N features for each time series in the training set 140 of the time series as output. In one embodiment, feature analyzer 105 is configured to perform a set or series of analyses on each time series in the training set of the time series. Each analysis evaluates the time series for a specific feature (or attribute or characteristic) and produces a value for that feature for each time series.

[0023] In one embodiment, the training set 140 of the time series is generated by a synthetic time series generator (STSG) 147. STSG 147 is configured to generate simulated time series including outliers, multiple seasonalities, change points, intermittentity, and high-level effects, thereby producing a realistic simulation of the time series behavior. Alternatively, in one embodiment, the training set 140 of the time series can be accessed from a database of real-world time series including outliers, multiple seasonalities, change points, intermittentity, and high-level effects.

[0024] In one embodiment, the autoencoder trainer 110 is configured to train the autoencoder 150 using a loss function that minimizes: (1) the difference between the activations of the bottleneck layer 151 in the autoencoder 150 and the vector 145 of the time series features, and (2) the difference between the input layer 152 and the output layer 153 of the autoencoder 150. For example, the loss function minimizes the difference between the activations of nodes in the bottleneck layer 151 of the autoencoder 150 and the vectors of N features of each individual time series in the training set 140 of the time series entered at the input layer 152. Furthermore, the loss function minimizes the difference between the values ​​of each time series entered at the input layer 152 of the autoencoder 150 and the values ​​at the output layer 153. This training produces a trained autoencoder 155.

[0025] In one embodiment, gap filler 115 is configured to generate one or more new vectors 160 of N features by minimizing the gaps between neighboring points in an N-dimensional feature space. In another embodiment, gap filler 115 is configured to minimize the gaps using t-distributed random neighbor embeddings to generate new vectors 160 of N features. The new vectors of N features fill the gaps in the N-dimensional feature space.

[0026] In one embodiment, the feature-based time series generator 120 is configured to generate a test set 165 of time series based at least in part on inputting a new vector 160 of N features into a bottleneck layer 151 of a trained autoencoder 155. In one embodiment, the bottleneck layer 151 is set as the input to the trained autoencoder 155 such that the trained autoencoder 155 behaves as a time series generator. In this configuration, the trained autoencoder 155 is configured to accept a vector of values ​​of N features (such as a new vector 160 of N features) at the bottleneck layer 151 and generate a time series at the output layer 153 that represents the values ​​of those N features. In one embodiment, the feature-based time series generator 120 is configured to input one or more features from the new vector 160 of N features into the trained autoencoder 155 to generate a gap-filled set of time series. The gap-filled set of time series thus generated by the autoencoder 155 consists of time series of the new vector 160 of N features that represent filling or closing gaps in an N-dimensional feature space. The gap-filled set of time series can also be more generally referred to as the gap-based set of time series.

[0027] In one embodiment, the feature-based time series generator 120 is configured to generate a test set 165 of the time series by combining a training set 140 of the time series with another time series generated by a trained autoencoder 155 based on a new vector 160 of N features. For example, the feature-based time series generator 120 may combine the training set 140 of the time series with a set of time series gap padding to generate the test set 165 of the time series as a combined set of time series. In one embodiment, the feature-based time series generator 120 is configured to generate the test set 165 of the time series using a trained autoencoder 155 based on a new vector 160 of N features, without combining it with the training set 140 of the time series.

[0028] In one embodiment, the forecast algorithm tester 125 is configured to input a test set 165 of a time series into each of a set of candidate forecast algorithms 167, and to compute a forecast error 170 for each algorithm based on its performance for each time series. The candidate forecast algorithms 167 are different from each other. In one embodiment, the forecast algorithm tester 125 is configured to input the test set 165 of a time series into a plurality of candidate forecast algorithms 167. The forecast algorithm tester 125 is configured such that each candidate forecast algorithm generates a forecast value based on the test set 165 of the time series. The forecast algorithm tester 125 is configured to determine the forecast error 170 of the plurality of candidate forecast algorithms 167 based at least on the values ​​predicted by the candidate forecast algorithms 167 and the actual values ​​of the test set 165 of the time series. The candidate forecast algorithms 167 are candidates or options from which one or more algorithms can be selected for monitoring the time series.

[0029] In one embodiment, ranking function trainer 130 is configured to train ranking function 175 to assign a ranking to candidate prediction algorithms 167 based on a vector of features. In one embodiment, ranking function 175 is a machine learning model (such as a classification model) used to classify prediction algorithms as having a given ranking among candidate prediction algorithms 167 in order to improve their accuracy in predicting time series having the features provided in the vector. In another embodiment, ranking function is a machine learning model (such as a regression model) used to estimate the accuracy of prediction algorithms in predicting time series having the features provided in the vector. In one embodiment, ranking function trainer 130 trains a machine learning model to determine the ranking of a prediction algorithm for predicting a given time series based on a set of N features of the given time series. The ranking function trainer trains the machine learning model based on the ranking assigned to the candidate algorithm for the time series and the N features (145 or 160) of the time series.

[0030] In one embodiment, the forecast algorithm selector 135 is configured to automatically select one forecast algorithm from the forecast algorithms to monitor the additional time series based on the ranking assigned to the N features 180 of the additional time series by a ranking function. In other words, the forecast algorithm selector 135 is configured to select one of the candidate forecast algorithms 167 to generate a prediction for a given time series. The selected algorithm 185 is selected based at least on the ranking assigned to the candidate algorithm 167 by a trained ranking function 175 (which is provided with the N features 180 of the additional time series as input).

[0031] This article presents further details regarding the algorithmic selection system 100. In one embodiment, reference will be made to... Figure 2 Method 200 is used to describe the operation of the algorithm selection system 100. In one embodiment, reference will be made to... Figures 3A-3IWorkflow 300 describes the operation of algorithm selection system 100, illustrating the automated, incremental construction of the forecast algorithm selector.

[0032] —Example Algorithm Selection Method—

[0033] Figure 2 An embodiment of an algorithm selection method 200 associated with the selection of feature-based time series forecasting algorithms is illustrated. Algorithm selection method 200 is an example process that can be used to train a machine learning model to determine which forecasting algorithms are best suited for time series based on feature characteristics. In one embodiment, algorithm selection method 200 includes steps for determining the suitability of a particular type of forecasting algorithm (or forecasting model) in predicting values ​​of a time series exhibiting a given feature profile.

[0034] In one embodiment, as a general overview, algorithm selection method 200 analyzes individual time series in an initial set of time series to identify a set of values ​​describing the characteristics of each time series. Algorithm selection method 200 then uses the time series and characteristics to train an autoencoder to have a bottleneck layer with activations of values ​​approximating the characteristics, and an output layer with activations approximating the time series values. Algorithm selection method 200 also examines specific characteristic values ​​of the initial set of time series plotted in a space with dimensions for each characteristic and finds a new collection of characteristic values ​​to fill the gaps in the characteristic space. Algorithm selection method 200 then feeds these new characteristic values ​​into the bottleneck layer of the trained autoencoder to generate supplementary time series with missing characteristics from the initial set of time series. These supplementary time series can be combined with the initial set of time series to form a test set of time series with dense coverage of the characteristic space. Algorithm selection method 200 then feeds the test set of time series into various prediction algorithms to determine the prediction error of each prediction algorithm. Algorithm selection method 200 thus correlates the prediction error of the prediction algorithm when processing the time series with the characteristics of the time series. Algorithm selection method 200 uses the correlation between forecast error and time series characteristics to train a ranking function to assign a ranking to each forecasting algorithm for monitoring the time series when the characteristics of the time series are provided. Using the trained ranking function, algorithm selection method 200 can automatically select a suitable forecasting algorithm from among the forecasting algorithms used to monitor the time series based on the characteristics of the time series.

[0035] In one embodiment, algorithm selection method 200 is initiated at “Start” block 205 in response to an algorithm selection system (such as algorithm selection system 100) determining one or more of the following: (i) the algorithm selection system has been instructed to create an automatic ranking function; (ii) the algorithm selection system has been instructed to select a forecasting algorithm for monitoring a given time series; (iii) an instruction to execute algorithm selection method 200 has been received; (iv) a user or administrator of the resonance detection system has initiated algorithm selection method 200; (v) it is currently scheduled to run algorithm selection method 200; or (vi) in response to the occurrence of some other condition, algorithm selection method 200 should begin. In one embodiment, a computer system configured by computer-executable instructions to perform the functions of algorithm selection system 100 executes resonance detection method 200. After initiation at “Start” block 205, resonance detection method 200 continues to block 210.

[0036] At box 210, algorithm selection method 200 processes multiple time series in a first set of time series or a training set to generate a vector of N features for each of the multiple time series. For example, algorithm selection method 200 can analyze each time series in the training set of time series to generate a vector of N features for each time series. Therefore, a feature vector of values ​​describing the set of features or characteristics that describe the behavior of the data in the time series can be generated for each time series.

[0037] In one embodiment, algorithm selection method 200 accesses an initial set of time series (also called the first set or training set). For example, the initial set of time series can be loaded from memory or a storage device, or provided by STSG. The time series in this set will be used to train the autoencoder and therefore may be referred to herein as the "training time series". Each training time series has a given length L, for example, L = 1500 observations (or data points). The training set includes S training time series sufficient to train the autoencoder, for example, S = 10000 time series. Other numbers S of training time series may also be sufficient for training the autoencoder, for example, between 5000 and 15000 time series. The number of time series used for training depends on how low the threshold used by the loss function to indicate that the autoencoder has been trained is.

[0038] Training time series exhibit a wide variety of behaviors, such as outliers, multiple seasonality, variance, levels, trends and / or different points of change in seasonality, varying intermittency, and different high-level effects. Training time series can be synthesized to possess these diverse behaviors, or the training time series can be real-world data confirmed to exhibit a wide variety of behaviors. In the case of time series data synthesis, in one embodiment, the set of training time series is generated by a synthetic time series generator (such as STSG 147). The synthetic generation of time series is discussed in more detail below, for example, under the heading "Progress in STSG Generation of Initial Sets of Time Series".

[0039] Although training sets are diverse, the time series within them may not uniformly cover the full range of behaviors that will be used to test the applicability of a particular type of forecasting algorithm / model. Accordingly, an autoencoder-based time series generator can be created (as discussed below in box 215). This autoencoder-based generator is configured to produce supplementary time series based on a feature vector of N features or attributes describing the supplementary time series. By augmenting or supplementing the feature vector to fill gaps in the feature coverage (as discussed below in box 220), the supplementary feature vector can be used to generate additional time series using the autoencoder. These additional time series enrich the initial training set, making the coverage of time series behavior more consistent.

[0040] In one embodiment, a feature vector function is performed on each training time series to generate, extract, or otherwise identify the values ​​of N features for each time series in the training set. Using the feature vector function, the values ​​of the N features for each training time series are determined by a specific analysis specific to each feature. The feature vector function stores the values ​​of the N features as a vector of N values, with one value for each feature. This vector or “feature vector” of feature values ​​is a data structure that includes a set of N values ​​corresponding to the N features. The N features describe the characteristics of the individual time series. In one embodiment, a feature is a canonical feature that expresses the behavior of a time series by essentially using the set of values ​​of the features of the time series. In one embodiment, these N features collectively and compactly describe the behavior recorded by the time series. For example, the N features could be a set of 22 “Catch22” features (listed in Table 1 below).

[0041] In one embodiment, the activity of block 210 is performed by feature analyzer 105. Additional details regarding this characterization of time series through features are provided below, for example, in reference to... Figure 3A And it is provided under the title "Specific features for time series generation".

[0042] At box 215, algorithm selection method 200 trains the autoencoder based on a loss function. The loss function minimizes the difference (“error-1”) between the activations of nodes in the bottleneck layer of the autoencoder and the vector of N features of the time series entered at the input layer of the autoencoder. Furthermore, the loss function also minimizes the difference (“error-2”) between the values ​​of the time series entered at the input layer and the values ​​at the output layer of the autoencoder. More simply, the algorithm selection method trains the autoencoder using a loss function that minimizes the following errors: (1) the error between the activations of the bottleneck layer in the autoencoder and the vector of features of the time series, and (2) the error between the input and output layers of the autoencoder. For example, algorithm selection method 200 trains the autoencoder such that the values ​​at the bottleneck layer match the values ​​of the feature vectors, and that the values ​​at the output layer match the values ​​at the input layer. This training causes the latter half of the autoencoder, from the bottleneck layer to the output layer, to transform the feature values ​​into a time series with those feature values.

[0043] An untrained autoencoder has an input layer, an output layer, and an internal bottleneck layer. Both the input and output layers contain nodes for training each observation (or location) in the time series. In other words, both the input and output layers have L nodes (e.g., 1500). The bottleneck layer has N nodes (e.g., N=22). For example, the bottleneck layer includes N nodes corresponding to N features of the input time series. In the autoencoder, the input time series is reduced from L activation values ​​at the input nodes (one value per observation) to N activation values ​​at the bottleneck nodes (one value per feature), and then restored to L activation values ​​at the output nodes.

[0044] An autoencoder-based time series generator is created based on training time series and the features of each time series. The autoencoder is then trained such that the N activation values ​​of nodes at the bottleneck layer match (or closely approximate) the N features of each training time series, and the L activation values ​​of nodes at the output layer match (or closely approximate) the L activation values ​​of nodes at the input layer. This is based on a loss function configured to minimize the combined error at the bottleneck and output layers. The error at the bottleneck layer is the difference between the N actual feature values ​​of the input time series and the N activation values ​​of the N nodes in the bottleneck layer. The error at the output layer is the difference between the L actual values ​​of the input time series and the L activation values ​​of the L nodes in the output layer.

[0045] In one embodiment, the loss function is a combined loss function that includes the weighted sum of the two measurements (error-1 and error-2) that comprise the aforementioned differences. In one embodiment, the two difference terms are given the same weight, for example, each with a weight of 0.5. In one embodiment, the error is the collective difference between groups of compared values. For error-1, this error is the collective difference between the N values ​​of node activation in the bottleneck layer and the N values ​​of the characteristics corresponding to these nodes generated by the feature analysis of the input time series. For error-2, this error is the collective difference between the L values ​​of the time series at the input layer of the autoencoder and the L values ​​of the corresponding observations in the estimated time series at the output layer of the autoencoder. In one embodiment, errors-1 and error-2 can be measured as the mean absolute scaling error (MASE), mean squared error (MSE), or mean absolute error (MAE) of the differences.

[0046] Once the autoencoder has been trained, it can be used to generate additional time series based on inputs of N features at the bottleneck layer. A vector of N values ​​representing the desired features for the time series can be input to the bottleneck layer of the trained autoencoder, which will then generate a time series with the desired features at the output layer. The bottleneck layer can be set or otherwise specified as input to the trained autoencoder. For example, layers preceding the bottleneck layer in the autoencoder can be pruned. Furthermore, nodes in the bottleneck layer are activated to the degree indicated by the node values ​​given by the input feature vector. Based on the input of a vector specifying N values ​​for the features, the latter half of the trained encoder will output a time series of length L representing the specified values ​​of the features.

[0047] An autoencoder takes a time series at the input layer with as many nodes as the time series length, passes the input constraints through a bottleneck layer with as many nodes as the feature length, and produces an output at the output layer with as many nodes as the time series length. The loss function used during training causes the bottleneck layer activations to closely approximate the feature values, and the output layer activations to closely approximate the time series values ​​provided at the input layer.

[0048] Therefore, in one embodiment, algorithm selection method 200 creates a time series generator from an initial set of time series as follows: Algorithm selection method 200 accesses the initial set of time series (e.g., as discussed above at box 210). Algorithm selection method 200 identifies N features for each time series in the first set of time series (e.g., using the feature vector function discussed at box 210). Algorithm selection method 200 trains an autoencoder with an N-node bottleneck layer based on a loss function that minimizes the difference between the bottleneck layer activation and the N features (as discussed at box 215). Furthermore, algorithm selection method 200 sets the bottleneck layer as input to the time series generator (as discussed at box 215). In one embodiment, the activity of box 215 is performed by autoencoder trainer 110. Additional details regarding autoencoder training are provided below, for example, in reference to... Figure 3B – Figure 3D Provided.

[0049] At box 220, algorithm selection method 200 generates one or more new vectors of N features by minimizing the gaps between neighboring points in the N-dimensional feature space. For example, algorithm selection method 200 determines the gaps in the N-dimensional feature space for an initial set of time series. Algorithm selection method 200 thus determines locations where feature coverage is sparse among multiple time series in the initial set of time series.

[0050] The features of the training vectors can be plotted in an N-dimensional space, also known as the feature space. The features of each training time series in the training set can be plotted in the feature space. In cases where the representation of a given behavior in the training set is insufficient, gaps or sparse regions will exist in the feature space. Additional feature vectors are generated to supplement and fill the gaps in the feature space of the initial set of time series. In this way, time series with unrepresented or insufficiently represented features are now included in these feature vectors. A trained autoencoder will be used to generate time series representing these insufficiently represented behaviors based on the values ​​of the features used to fill or close the gaps, as discussed below in box 225.

[0051] In one embodiment, gaps are identified by reducing an N-dimensional graph to a two-dimensional graph (or other low-dimensional graphs, such as 3D graphs) and then highlighting sparse regions in the two-dimensional graph. In another embodiment, a graph in an N-dimensional feature space is reduced to two dimensions using t-distributed random neighbor embedding (t-SNE) or another suitable nonlinear dimensionality reduction technique. Sparse regions are then identified by measuring the local density of the reduced-dimensional graph using a distance metric. For example, the high Euclidean distance of a point to its nearest neighbor indicates that a low-density or sparse region surrounds the point. Alternatively, regions considered as noise in density-based spatial clustering (DBSCAN) for noisy applications can be considered sparse regions. Highlighting sparse regions can also be based on other metrics, including kernel density estimation (KDE), local outlier factor (LOF) detection algorithms, and Gaussian mixture models (GMM).

[0052] As sparse regions (gaps) are identified, algorithm selection method 200 generates a new set of N feature values ​​that will appear in the sparse regions. In one embodiment, new points in the sparse regions are generated by placing points between existing points via interpolation (such as linear interpolation or spline interpolation). Alternatively, in one embodiment, generating new feature data points to fill the sparse regions can also be performed by averaging the N feature values ​​of neighboring data points to produce new points. In one embodiment, new points are placed between two points in the sparse regions in the N-dimensional feature space. In one embodiment, S (e.g., 10,000) new sets of N feature values ​​are generated.

[0053] Therefore, in one embodiment, the algorithm selection method 200 simplifies the feature map from N dimensions to two dimensions using t-SNE, identifies sparse regions in the two-dimensional feature map by finding regions with relatively high Euclidean distances between neighboring points, and generates new feature data points in the sparse regions by averaging the feature values ​​of neighboring points. Thus, using a bottleneck layer as the input layer allows for the generation of an additional set of time series from feature vectors selected to minimize the gap between two nearest points in the N-dimensional feature space (discussed below at box 225). In one embodiment, the activity of box 220 is performed by gap filler 115. The generation of new feature vectors is further discussed below, for example regarding... Figure 3E Discussed.

[0054] At box 225, the algorithm selection method 200 generates a test set of time series at least in part based on inputting a new vector of N features into the bottleneck layer of a trained autoencoder. For example, the algorithm selection method 200 generates a second set or test set of time series at least in part by inputting feature vectors that fill the gaps into the bottleneck layer to generate further time series with those input features. This set of further time series with the input features that fill the gaps can therefore also be referred to as a gap-filled set of time series. In one embodiment, the algorithm selection method 200 inputs one or more feature vectors that fill the gaps into the bottleneck layer to generate a gap-filled set of time series. In one embodiment, the test set of time series includes an initial set and further time series generated by the time series generator based on the feature vectors that fill the gaps in the feature space.

[0055] In one embodiment, algorithm selection method 200 inputs the S new feature vectors (set of feature values) generated above at box 220 into the bottleneck layer of a trained autoencoder. As discussed further in detail herein, the autoencoder has been trained, so activating the feature nodes of the bottleneck layer will cause the autoencoder to generate time series at the output. In response to the input of the S new feature vectors, the trained autoencoder generates S new time series. Thus, in one embodiment, using the bottleneck layer as the input layer allows for the generation of a gap-filled set of time series based on feature vectors selected to minimize the gap between two nearest points in the N-dimensional feature space. The generated gap-filled set of time series can therefore be considered gap-based because the gap-filled time series constituting the gap-filled set of time series has the property of filling sparse regions of the feature space.

[0056] In one embodiment, S new time series are appended or inserted into the training set to generate a test set of time series signals. For example, the test set would then include 2 × S time series signals. Therefore, in one embodiment, the algorithm selection method 200 combines the training set of time series (i.e., the initial set or first set) with the gap-filling set of time series to generate a combined set of time series as the test set. In one embodiment, S new time series signals are used as the test set without including the training set.

[0057] Therefore, in one embodiment, algorithm selection method 200 accesses a trained autoencoder. For S new feature vectors, algorithm selection method 200 sets the activations of N nodes in the bottleneck layer to the values ​​of the N features corresponding to these nodes, executes the trained autoencoder to the output layer, and stores the activations of L nodes in the output layer as a new time series data structure with a length of L observations, thereby generating and storing S additional time series. In one embodiment, each new time series may be stored in association with the values ​​of the N features input at the bottleneck layer that cause the generation of the new time series. In one embodiment, the activity of block 225 is performed by feature-based time series generator 120. The creation of the test set is also referenced below. Figure 3E discuss.

[0058] At block 230, algorithm selection method 200 inputs a test set of time series to each of a set of different forecasting algorithms and calculates the forecast error of each algorithm based on its performance for each time series. As discussed below at block 240, these different forecasting algorithms become candidates to be selected to monitor additional time series. In one embodiment, algorithm selection method 200 inputs a combined set of time series to a plurality of candidate forecasting algorithms, and each candidate forecasting algorithm is configured to generate a forecast value based on the combined set of time series in response to the input. Thus, in one embodiment, algorithm selection method 200 tests the accuracy of the candidate forecasting algorithms. Algorithm selection method 200 inputs a test set of time series to a plurality of candidate forecasting algorithms. Algorithm selection method 200 executes each candidate forecasting algorithm to generate a forecast value based on the combined set of time series. Algorithm selection method 200 then determines the forecast error of the plurality of candidate forecasting algorithms based at least on the forecast value and the combined set of time series.

[0059] A test set of time series signals is thus used to evaluate the applicability of various types of forecasting models in handling time series exhibiting specific characteristics. The time series signals in the test set are provided one by one to a set of F candidate forecasting algorithms. In one embodiment, there are F=5 forecasting algorithms: an autoregressive integral moving average (ARIMA) model, an error-trend-seasonal (ETS) model, a deep learning model, an error feedback estimation (EFE) model, and a prophet model. For each of the F algorithms, the forecast error between the test time series signal and the forecast is determined. In one embodiment, the forecast error can be the mean absolute scaling error (MASE), mean squared error (MSE), or mean absolute error (MAE) over the length of the test time series signal.

[0060] The forecast error of each of the F forecasting algorithms is stored in association with the characteristic values ​​of the test time series. For example, the characteristic values ​​and forecast errors can be stored in a vector of length N+F. At the end of the test, the performance of each type of forecasting model (measured by the forecast error) is associated with the characteristic of the time series being forecasted.

[0061] Therefore, in one embodiment, algorithm selection method 200 generates performance measurements for each candidate forecasting algorithm on each time series signal in the test set. For each forecasting algorithm, algorithm selection method 200 compares the original test time series with the predictions generated by the forecasting algorithm to measure the difference (residual), and then calculates the error value (e.g., MASE) based on these differences. The errors of the various candidate algorithms for the time series are then stored in association with the characteristics of that time series. In one embodiment, the activities of block 230 are performed by forecasting algorithm tester 125. The evaluation of the forecasting algorithms also refers to... Figure 3F and Figure 3G Let's discuss that below.

[0062] At box 235, algorithm selection method 200 trains a ranking function to assign a ranking to each forecasting algorithm based on a vector of N features provided. For example, algorithm selection method 200 trains a machine learning model to determine the ranking of forecasting algorithms used to forecast a given time series based on the forecast error and the N features. In one embodiment, the machine learning model is trained to automatically rank candidate forecasting models for monitoring the accuracy of the given time series when given N features describing the given time series as input.

[0063] Here, the ranking function (machine learning model) is trained based on the prediction error and N features of the training set of the time series. Once trained, the ranking function will determine the ranking of the prediction algorithm used to predict the given time series based on the input of the set of N features of the given time series.

[0064] In one embodiment, the ranking function includes an ML regression model to estimate the forecast errors of F forecasting algorithms given N features, and includes a ranking function to sort the resulting regression estimates by ranking. The regression model is configured to generate an estimated forecast error for each of the F forecasting algorithms, the estimated forecast error being consistent with the performance of the F forecasting algorithms on a test set of time series signals. The ranking function is configured to sort the F forecasting algorithms in ascending order of the estimated forecast errors and label the forecasting algorithms with the ranking order.

[0065] In one embodiment, the ranking function includes a sorting function to rank F forecasting algorithms in ascending order of their actual forecast errors and to label the F forecasting algorithms with the rank order, and includes an ML classification model to estimate the rank of the F forecasting algorithms given N features. The classification model is configured to generate an estimated rank for each of the F forecasting algorithms, which is consistent with the actual rank of the F forecasting algorithms by forecast error.

[0066] In one embodiment, once trained, the ranking function accepts a vector of N features and produces F prediction models ranked from best to worst (based on estimated prediction errors) for a specific time series with given values ​​of N features. In one embodiment, the activity in block 235 is performed by the ranking function trainer 130. (Training reference for the ranking function) Figure 3H This will be discussed further below.

[0067] At box 240, algorithm selection method 200 automatically selects one forecasting algorithm from the candidate forecasting algorithms to monitor the additional time series based on N characteristics processed by a ranking function. For example, algorithm selection method 200 selects an algorithm from the candidate forecasting algorithms to generate a prediction for a given time series based on at least one ranking.

[0068] In one embodiment, the corresponding cross-validation error (ROCV) between the cross-validation of a specific time series and the cross-validation of F forecasting algorithms is selected. In one embodiment, the corresponding cross-validation error is found only for a subset F' of the F forecasting algorithms that has the minimum estimated forecast error. In other words, the subset F' includes the top few of the F forecasting algorithms. The algorithm with the minimum ROCV is selected as the top-ranked forecasting algorithm. This top-ranked forecasting algorithm can then be deployed to continuously monitor the specific time series.

[0069] In one embodiment, algorithm selection method 200 automatically selects a forecasting algorithm from the forecasting algorithms to be applied to monitor the additional time series. Algorithm selection method 200 assigns a ranking to the forecasting algorithms using a trained ranking function. The ranking is assigned based on an N-dimensional vector of characteristics of the additional time series. Once the ranking is assigned, algorithm selection method 200 selects several top-ranked algorithms from the forecasting algorithms. In one embodiment, the top three forecasting algorithms are selected. Algorithm selection method 200 then calculates the corresponding cross-validation error of the top-ranked algorithms for the additional time series. Furthermore, algorithm selection method 200 selects the top-ranked algorithm with the smallest corresponding cross-validation error as one of the forecasting algorithms. In one embodiment, the activity of block 240 is performed by forecasting algorithm selector 135. Reference will be made below. Figure 3I The automatic selection of forecasting algorithms based on the characteristic profiles of the time series to be monitored is further discussed. After completing box 240, the algorithm selection method 200 proceeds to the "end" box 245, where the algorithm selection method 200 ends.

[0070] The selected algorithm is then automatically used to monitor the additional time series. The steps of method 200 operate to automatically select the algorithm that has the best performance for monitoring the given time series based on the specific characteristics of the given time series.

[0071] —Further Implementation of the Example Algorithm Selection Method—

[0072] In one embodiment, after closing block 240, algorithm selection method 200 further includes the steps of predicting the value of the additional time series and issuing an alarm when the actual value deviates from the prediction. For example, algorithm selection method 200 continues to predict the value of the additional time series using the selected forecasting algorithm. And, if the actual value of the additional time series differs from the predicted value, algorithm selection method 200 generates an alarm. (Detection of abnormal differences and electronic alarms are discussed in further detail in the heading “Detection of Abnormal Deviations and Electronic Alarms” below).

[0073] In one embodiment of the algorithm selection method 200, the bottleneck layer (discussed above at box 215) comprises N nodes corresponding to N features in a vector of N features. For example, the bottleneck layer has 22 nodes corresponding to 22 features of the time series. In one embodiment, these 22 nodes correspond one-to-one with the 22 “catch-22” features of the time series described below.

[0074] In one embodiment of the algorithm selection method 200, time series in the initial set of time series (introduced in box 210 above) are synthesized to simulate a range of time series behaviors including outliers, multiple seasonalities, change points, intermittency, and high-level effects.

[0075] In one embodiment of the algorithm selection method 200, the loss function (discussed above in box 215) is configured to minimize the difference between the bottleneck layer activation and the feature vector by evaluating a) the combined error of the difference between the output and the input and b) the difference between the bottleneck layer and the feature vector of the first set of time series.

[0076] In one embodiment of the algorithm selection method 200, using a bottleneck layer as an input layer allows for the generation of a combined set (or test set) of time series based on feature vectors selected to minimize the gap between two nearest points in the N-dimensional feature space (as discussed above in boxes 220–225).

[0077] In one embodiment of the algorithm selection method 200, the candidate forecasting algorithms include one or more of the following: an autoregressive integral moving average (ARIMA) model, an error-trend-seasonal (ETS) model, a deep learning model, an error feedback estimation (EFE) model, and a prophet model (as discussed above in box 230).

[0078] In one embodiment of the algorithm selection method 200, (discussed at box 210) the operation of analyzing each time series to produce a vector of N features generates a vector to include catch-22 features. Thus, in one embodiment, the N features of each time series in a first set of time series include multiple features selected from: the mode of the z-score distribution, the longest period of consecutive values ​​above the mean, the time interval between successive extreme events above the mean, the time interval between successive extreme events below the mean, the first 1 / e crossover of the autocorrelation function, the first minimum of the autocorrelation function, the total power in the lowest frequency portion of the Fourier power spectrum, the centroid of the Fourier power spectrum, the average error of the rolling multi-sample mean forecast, the time reversibility statistic, self-mutual information, the first minimum of the self-mutual information function, the proportion of a given coefficient where successive differences exceed the standard deviation, the longest period of successive increment reduction, the Shannon entropy of two successive letters in an equally probable 3-letter symbolization, the change in correlation length after iterative differencing, and the exponential fit of successive distances in the 2D embedding space. In one embodiment, this analysis generates a vector that includes the complete set of catch-22 features.

[0079] In one embodiment of algorithm selection method 200, the initial (or training) time series (introduced in box 210) each have the same length. In one embodiment, additional time series generated by an autoencoder to produce time series for a test set may also have the same length as the initial time series. In one embodiment, the length of each time series in the training and test sets is equal. This length is sufficient to convey the behavior of the time series to the machine learning model during the training process. Thus, in one embodiment, the training and test time series have equal lengths, between 1000 and 2000 observations. For example, the length of the training and test time series could be 1500 observations.

[0080] In one embodiment (at box 240), one of the prediction algorithms is automatically selected, and the corresponding cross-validation error is used to evaluate the overfitting of the most accurate prediction algorithm. From the most accurate algorithms, the algorithm with the least overfitting is automatically selected (see below). Figure 3I(As discussed). In one embodiment, algorithm selection method 200 assigns a ranking to the prediction algorithm based on the N-dimensional vector used for the appended time series using a trained ranking function. Algorithm selection method 200 selects three top-ranked algorithms from the prediction algorithms. Algorithm selection method 200 calculates the corresponding cross-validation error of the top-ranked algorithms relative to the appended time series. Furthermore, algorithm selection method 200 selects the top-ranked algorithm with the smallest corresponding cross-validation error as one of the prediction algorithms.

[0081] In one embodiment, minimization of the gap between neighboring points in the N-dimensional feature space (discussed at box 220) is performed based on t-distributed random neighbor embeddings.

[0082] —Discussion and Additional Examples—

[0083] In one embodiment, the algorithm selection system and method described herein address the complex problem of selecting forecasting algorithms for time series forecasting by employing a flexible and inclusive strategy. Given the algorithm selection system's proficiency in managing time series data from a wide variety of industries, fields, or functions, this improvement facilitates cross-industry applications.

[0084] Two prominent algorithm selection strategies are recognized in both industry and academia: brute-force methods and earlier meta-learning methods (such as FFORMS). Brute-force methods involve applying all available algorithms to the dataset in question to identify the most efficient one. However, this approach is clearly resource-intensive. Earlier meta-learning methods (such as FFORMS) are slightly more resource-efficient strategies, using well-known public datasets (such as M3, M4, M5, and / or Kaggle) to construct classifiers or ranking functions. However, existing meta-learning methods only perform satisfactorily with datasets from the domain encountered during the training phase.

[0085] The algorithm selection system and method described in this paper overcome these challenges by providing a novel meta-learning approach. This method abandons the use of public datasets and fully leverages synthetically generated time-series datasets, canonical features of time series data, and autoencoder machine learning to ensure training coverage across all domains of the machine learning tools used to select forecasting algorithms. This expands the applicability and versatility of automatic forecasting algorithm selection.

[0086] Therefore, in one embodiment, the algorithm selection system implements a domain-agnostic approach for developing time series forecasting algorithms. In one embodiment, the algorithm selection system provides a comprehensive approach to time series forecasting that combines synthetic data generation, autoencoder-based feature representation, and adaptive algorithm selection within a domain-agnostic framework. This combination of technologies results in a highly versatile and efficient forecasting solution.

[0087] In one embodiment, the algorithm selection system employs synthetic time series generation to ensure broad domain coverage of the time series activities. For example, the algorithm selection uses a Synthetic Time Series Generator (STSG) to synthesize an initial set of time series. The use of STSG allows for the creation of a wide variety of time series data incorporating a broad range of behaviors, making the resulting model more adaptable to various scenarios and time series characteristics. This aspect represents an improvement over prior art that is domain-specific and therefore not domain-agnostic.

[0088] In one embodiment, the algorithm selection system implements feature representation based on an autoencoder. In this embodiment, the algorithm selection system employs an autoencoder to capture catch-22 features (or characteristics) of the time series. This use of an autoencoder enables a compact and efficient representation of the input time series data, thereby facilitating the identification of gaps in the 22-dimensional vector space and allowing for improved forecast performance. This aspect is an improvement over prior art that fails to identify where the main body of the time series lacks certain characteristics and cannot automatically generate time series to supplement or enhance the main body of the time series based on the provided characteristics.

[0089] In one embodiment, the algorithm selection system fully utilizes the complete coverage of the 22-dimensional catch-22 vector space. In another embodiment, the algorithm selection system and method ensure that the entire 22-dimensional catch-22 vector space is covered. This allows the algorithm selection system to choose forecasting algorithms to handle time series data from any domain, making the algorithm selection system highly versatile and robust. This aspect is an improvement over previous techniques that could make poor forecasting algorithm selection for time series with characteristics in uncovered regions of the catch-22 vector space.

[0090] In one embodiment, the algorithm selection system performs algorithm ranking and selection to choose from multiple candidate prediction algorithms. By ranking various prediction algorithms based on their performance relative to the catch-22 features of the input time series, the proposed method can adaptively select the most suitable algorithm for a given task, thereby producing more accurate prediction results. This aspect is an improvement over previous techniques that did not consider how good a particular prediction algorithm is compared to alternative algorithms.

[0091] In one embodiment, the algorithm selection system is a domain-agnostic approach for selecting forecasting algorithms. The algorithm selection system operates across different domains, thereby allowing for more universally applicable solutions in time series forecasting. In one embodiment, universality is achieved by fully covering the 22-dimensional catch-22 vector space in the set of time series used to test the accuracy of the forecasting model, thus ensuring the suitability of the selected model for various types of time series data.

[0092] In one embodiment, the algorithm selection system operates on the STSG to model a wide range of time series behavior, including outliers, multiple seasonality, change points (variance, level, trend, seasonality), intermittency, and high-level effects. The algorithm selection system then trains an autoencoder on numerous synthetic time series generated by the STSG. The bottleneck layer of the autoencoder comprises 22 nodes, while the input and output layers have 1500 nodes each. The loss function evaluates the combined error of: (a) the difference between the autoencoder's output and input; and (b) the difference between the bottleneck layer and the catch-22 features of the time series. The algorithm selection system visualizes or projects the catch-22 features of the input time series data onto a 2D canvas to accurately identify regions in the 22-dimensional vector space lacking the input time series.

[0093] The algorithm selection system employs the latter half of an autoencoder, using a bottleneck layer as the input layer to process a sufficient number of synthetically constructed catch-22 vectors to ensure that there are no large gaps between two nearest points in the 22-dimensional vector space. These constructed vectors are then used as input to the output of the bottleneck layer to test the accuracy of various forecasting algorithms. The algorithm selection system applies different forecasting algorithms to the output time series from the previous step and records the corresponding forecasting errors (such as MASE) of the algorithms.

[0094] The algorithm selection system then ranks the algorithms used for each time series and trains a ranking function that assigns a ranking to each algorithm based on a given 22-dimensional feature vector representing the input sequence. The algorithm selection system then implements a versatile forecast algorithm selector. In one embodiment, the forecast algorithm selector is applicable to any domain because it covers the entire 22-dimensional vector space to generate time series that are subsequently input into the forecast algorithms.

[0095] —Progress in STSG generation of initial sets of time series—

[0096] In one embodiment, the algorithm selection system begins with an initial set of synthetic time series, such as using STSG. The initial set of time series synthesized by STSG can be used both to train the autoencoder and, once supplemented with further time series to fill gaps in the feature space, to train the prediction model's ranking function. In one embodiment, the progress of synthetic time series dataset generation evolves through several stages: Stage 0: Baseline model; Stage 1: Enriched model; Stage 2: Meta-enhanced model; Stage 3: Dynamic model; and Stage 4: Near-realistic model.

[0097] Phase 0, the baseline model phase, focuses only on the primary data. This represents the initial state of the synthetic time series dataset generation.

[0098] Phase 1, the model enrichment phase, is characterized by incorporating additional data, amplifying the complexity of the main dataset, and embedding a wider range of contextual elements. In one embodiment, the time series is assumed to be a fusion of basic values ​​(for each observation of the time series) augmented by numerous factors, as shown in Equation EQ. 1: target = base_value factor1 factor2 ... factorN + Noise EQ.1 In one embodiment, in phase 1, three different factors were utilized: 1. Weekday Factor: This introduces the concept of time dependence, taking into account the variation between weekdays and weekends within each series; 2. Random Feature Factor, which assigns random weights to each sequence, adds an element of unpredictability to the synthetic data; and 3. The sinusoidal factor is used to inject a sense of seasonality into some sequences, reflecting the periodic trends often observed in real-world data.

[0099] These factors generate numerical coefficients that are integrated into the above formula EQ.1, assigning different weights to produce or achieve the target value.

[0100] Upon completion of Phase 1 (the model enrichment phase), the synthesized ensemble of time series can be acceptable for use in training the autoencoder. However, to generate more realistic time series, further refinement of the time series can be expected at Phase 2.

[0101] Phase 2, the meta-augmentation model phase, integrates metadata into the pre-existing dataset. Integrating metadata into the dataset facilitates a more comprehensive understanding of the data context and interconnections, thereby enhancing the realism and relevance of the synthetic dataset. In one embodiment, STSG employs a top-down approach to generate synthetic time-series data during the meta-augmentation model phase.

[0102] In the meta-enhanced model phase, processing begins with metadata, creating logical columns such as brand, city, and state. Then, synthetic time series are generated for each metadata column, for example, using a library of functions for generating time series. The resulting individual series represent the variability of time series data with location metadata. These individual series are combined by weighted summation to produce a single time series.

[0103] For the output computation in the meta-enhanced model stage, STSG incorporates three further factors: 1. A collection of seven distinct seasonality patterns that capture seasonal variations and trends in the data; 2. Weekend Boost Factor, which combines the impact of weekends on time series data (if applicable); and 3. Random Series Factor, which provides an additional degree of randomness, thereby enhancing the unpredictability of the element.

[0104] The baseline formula therefore underwent modifications during the meta-enhanced model phase, as shown in EQ. 2 and EQ. 3:

[0105] (It should be noted that in EQ. 3, each sequence—Series) brand Series city and Series state —(This was generated using a modified baseline EQ. 2.) This approach results in a better mirroring of the complexity and dynamics of real-world data in synthetic datasets, thus laying the foundation for more accurate and effective forecasts.

[0106] Upon completion of Phase 2 (the meta-augmentation model phase), the synthesized ensemble of time series can be acceptable for use in training the autoencoder. However, to generate more realistic time series, further refinement of the time series can be expected at Phase 3.

[0107] Phase 3, the dynamic model phase, introduces dynamic elements—specifically, shifts and intermittency—into the synthetic dataset. These additions inject another layer of realism into the synthetic dataset by simulating the unpredictability and nonlinearity found in real-world data by considering abrupt changes and discontinuous data points. Thus, the dynamic model phase delves deeper into the complexities of data synthesis, focusing on capturing the sudden shifts, irregularities, and nonlinear relationships often seen in real-world time series data.

[0108] In one embodiment, the processing at the dynamic model stage begins with an enriched dataset from the previous meta-augmentation model stage. This enriched dataset is further augmented by adding two important elements: change points and intermittency.

[0109] A change point is a point in time series data where data undergoes a sharp shift. Four types of change points are introduced to model various aspects of real-world data. These four types include (a) Level Change change points, which mimic a sudden shift in the baseline level of the data; (b) Trend Change change points, which reflect a sharp change in the trend or slope of the data; (c) Seasonality Change change points, which capture a sudden shift in seasonal patterns; and (d) Variance Change change points, which represent a change in the volatility or dispersion of the data. Including these diverse change points allows for the synthesis of data to exhibit non-linear and unpredictable behavior often seen in real-world scenarios.

[0110] Intermittency refers to the sporadic and often unpredictable occurrence of events. By incorporating intermittency into synthetic time series data, the synthesized time series mirrors real-world scenarios where data is not generated continuously or events do not occur at regular intervals.

[0111] Due to the nonlinear properties introduced by various types of change points, the baseline formulas used in previous stages are insufficient to represent the synthetic data in this stage. Instead, more advanced statistical models or machine learning techniques may be needed to capture the complex relationships in the data. The final synthetic time series now becomes a complex nonlinear combination of individual sequences, where each sequence represents a unique piece of metadata, further modulated by change points and intermittency.

[0112] Upon completion of Phase 3 (the dynamic model phase), the synthesized ensemble of time series data is acceptable for use in training the autoencoder. However, to generate time series data that closely mirror real-world data, further refinement of the time series is expected at Phase 4.

[0113] In Phase 4, the near-realistic model phase, the synthetic dataset is tweaked to incorporate the additional complexities and nuances present in real-world data. In this way, the synthetic time-series dataset can become virtually indistinguishable from real-world datasets.

[0114] In one embodiment, the synthetic time series generation described herein can be performed by a synthetic time series generator 147. Synthetic time series generation is used to provide a wide variety of time series—that is, time series exhibiting a wide range of characteristics. In one embodiment, the diversity of the synthetic time series ensures that both the time series and the characteristic vectors associated with the time series (e.g., catch-22 vectors) have different characteristics and cover a wide range of behaviors.

[0115] —Construction of the forecast algorithm selector—

[0116] In one embodiment, Figures 3A-3I An example of automated, incremental construction of a forecast algorithm selector is shown. Figures 3A-3I The workflow 300 is shown in the sequential stages of development as a forecast algorithm selector. (Reference) Figure 3A In one embodiment, construction is initiated from an initial set (or training set) (e.g., 10,000 time series 304) generated using a synthetic time series generator (STSG) 302.

[0117] In one embodiment, the STSG 302 skillfully models a wide variety of time series behaviors in time series generated by the STSG 302. The simulated behaviors encompass outliers, multiple seasonality, change points, intermittency, and high-level effects. Outliers are data values ​​in a time series that significantly differ from other values ​​in the time series in terms of patterns and trends. Seasonality refers to changes in time series values ​​that occur at regular intervals. Change points are locations in a time series where one or more statistical properties adjust sharply. Change points can include reference variance, level, trend, seasonality, or other statistical properties. Intermittency refers to irregular or sporadic observations of time series data over time, resulting in gaps in observations or irregularly spaced observations within the time series. High-level effects are broad or general patterns, phenomena, or factors that influence the behavior of a time series but are not specifically attributed to other behaviors of the time series.

[0118] In workflow 300, STSG 302 generates a sufficient number of time series as an initial set, such as 10,000 time series 304, which is sufficient for training the autoencoder 306. In one embodiment, each time series in the initial set (10,000 time series 304) has the same length. For example, each time series could have a length of 1,500 observations. In this example, 1,000-2,000 observations are sufficient to capture the time series behavior discussed above. If the time series behavior extends beyond a larger number of observations, then the length of the time series can be extended to cover these behaviors.

[0119] Each synthesized time series is processed by a feature function, which identifies the values ​​of a predetermined set of features or attributes of the time series, such as the 22 "catch-22" features listed below. For each time series, the feature function generates a feature vector, which includes the time series'... N The value of each feature in a set of features (such as 10,000 catch-22 vectors 308). For example, in the case where the features to be determined for a time series are 22 “catch-22” features, the feature function can be called a “catch-22 function” and will produce a 22-dimensional vector for each of the 10,000 time series of length 1500.

[0120] Turn now Figure 3B We will examine the autoencoder 306 in more detail. The autoencoder 306 consists of an input layer 312, an output layer 314, and several hidden layers. Within the hidden layers, there is a bottleneck layer 316. In one embodiment, the input layer 312 and the output layer 314 have the same number of nodes as the length of the input time series. For example, if the time series in the initial set is as long as 1500 observations, both the input layer 312 and the output layer 314 have 1500 nodes, with one node corresponding to each observation of the time series. The time series can thus be entirely input into the autoencoder 306 at the input layer 312, and a complete estimated time series is produced at the output layer 314.

[0121] The bottleneck layer has a quantity of N The node, N This is equivalent to the number of features extracted from the time series by the feature function. For example, if the feature function produces values ​​for 22 catch-22 features, then there are exactly 22 nodes in the bottleneck layer 316. It should be noted that other hidden layers may exist in the autoencoder 306 between the input layer 312 and the bottleneck layer 316, and between the bottleneck layer 316 and the output layer 314, but for simplicity, these are not listed here. Figure 3B Not shown in the image.

[0122] Now for reference Figure 3C Workflow 300 is extended to illustrate aspects of training the autoencoder 306. In one embodiment, a combined loss function 322 is used to train the autoencoder 306. The autoencoder 306 is trained to minimize the combined loss function 322. The combined loss function 322 is configured such that the training of the autoencoder 306 minimizes the difference between a first error function (error -1 324) and a second error function (error -2 326). In one embodiment, error -1 324 is the difference (deterministic) between the activations of the bottleneck layer 316 and the corresponding values ​​in the catch-22 vector 308 computed using the catch-22 function. Alternatively, more generally, error -1 324 is the difference between the node activations in the bottleneck layer 316 for a time series entered at the input layer 312 and the values ​​of the N features generated by the feature function for the time series entered at the input layer 312. In one embodiment, error -2 326 is the difference between the input layer 312 and the output layer 314. In other words, error -2 326 is the difference between the observed value of the time series recorded at input layer 312 into autoencoder 306 and the estimated value of the time series output by autoencoder 306 at output layer 314. In one embodiment, the combined loss 322 integrates error -1 324 and error -2 326 into a weighted sum, applying equal weight to each of error -1 324 and error -2 326. By integrating these two errors into the combined loss 322, the training of autoencoder 306 concurrently minimizes both error -1 324 and error -2 326.

[0123] exist Figure 3D At this point, the autoencoder 306 is reconfigured to operate as a feature-based time series generator. After training of the autoencoder 306 is complete, the “first half” of the layers in the autoencoder 306 can be removed. The bottleneck layer 316 is then configured to be used as a new input layer in the “second half” 332 of the trained autoencoder 306, replacing the input layer 312. In one embodiment, the “first half” of the layers of the autoencoder 306 is the layer preceding the bottleneck layer 316, including the input layer 312. And, in one embodiment, the “second half” 332 is the layer from the bottleneck layer 316 to the output layer 314, including the bottleneck layer 316. The output layer 314, and the configuration of the autoencoder 306 between the bottleneck layer 316 and the output layer 314, remain unchanged in the second half 332 configuration.

[0124] Now for reference Figure 3E Algorithm selection system analysis NA vector space of features is used to determine the range of features that represent fewer features in the initial collection of time series. These sparse ranges are also called... N "Gap" in the vector space of features (such as the vector space of catch-22 features). In one embodiment, the algorithm selection system identifies gaps in the vector space of features using T-SNE and data analysis 342. For example, in T-SNE and data analysis 342, the algorithm selection system identifies gaps by visualizing 10,000 catch-22 vectors 308 on a 2D canvas using t-SNE and then highlighting areas where the input time series is lacking in the 22-dimensional vector space. In one embodiment, T-SNE and data analysis 342 are executed simultaneously (or otherwise in parallel) with a feature-based time series generator that reconfigures the autoencoder 306 as the "back half" 332. Subsequently, the algorithm selection system generates further feature vectors to fill the gaps (which can be used to generate a gap-filling set for the time series, as discussed below). For example, the system generates an additional 10,000 feature (catch-22) vectors 344. The generation of these additional feature vectors ensures an even more uniform distribution by minimizing large gaps between neighboring points in the feature vector space.

[0125] In one embodiment, gap-padded time series can be generated using additional feature vectors from the latter half of a previously trained autoencoder. A gap-padded time series is a time series that has features that fill or close gaps in a feature vector space. In one embodiment, the gap-padded set of time series includes multiple gap-padded time series that fill multiple gaps in the feature vector space. The newly generated 10,000 catch-22 vectors 344 are used as input to the latter half 322 of the previously trained autoencoder 306. Inputting the 10,000 catch-22 vectors 344 into the latter half 322 of the previously trained autoencoder 306 results in 10,000 new time series 346. As discussed above, inputting feature vectors into the bottleneck layer 316 of the trained autoencoder 306 causes the latter half 322 of the trained autoencoder 306 to produce time series at the output layer 314 exhibiting the features included in the feature vectors. Inputting feature vectors that fill the gaps in the feature space causes the latter half 322 of the trained autoencoder 306 to generate new time series 346. These new time series 346 exhibit characteristics different from those already present in the initial set 304 of time series, but remain within the distribution of the initial set 304. In short, inputting 10,000 catch-22 vectors 344 that fill the gaps in the feature vector space into the latter half 322 of the trained autoencoder 306 generates a gap-filled set of time series, resulting in 10,000 new time series 346.

[0126] Now for reference Figure 3F The algorithm selection system provides a dataset containing 10,000 new time series 346 as input to each of a set of candidate forecasting algorithms 348. In one embodiment, the dataset provided to each of the candidate forecasting algorithms 348 is a combined test dataset including an initial set 304 of time series and the new time series 346. In one embodiment, the candidate forecasting algorithms 348 include an autoregressive integral moving average (ARIMA) model 350, an error-trend-seasonal (ETS) model 352, a deep learning model 354, an error feedback estimation (EFE) model 356, and a prophet model 358.

[0127] like Figure 3GAs shown, the algorithm selection system determines a prediction error 359 for candidate prediction algorithms 348. Prediction errors are determined for each algorithm and time series. In one embodiment, for each individual time series in the test set, ARIMA error 360, ETS error 362, deep learning error 364, EFE error 366, and prophet error 368 are determined. ARIMA error 360 is the error between the time series input to ARIMA model 350 and the estimated time series output from ARIMA model 350. ETS error 362 is the error between the time series input to ETS model 352 and the estimated time series output from ETS model 352. Deep learning error 364 is the error between the time series input to deep learning model 354 and the estimated time series output from deep learning model 354. EFE error 366 is the error between the time series input to EFE model 356 and the estimated time series output from EFE model 356. The Prophet error 368 is the error between the time series input to the prophet model 358 and the estimated time series output from the prophet model 358.

[0128] In one embodiment, the forecast error is calculated as the mean absolute scaling error (MASE). In other embodiments, the forecast error may be calculated as the mean squared error (MSE) or the mean absolute error (MAE). The forecast error provides a single-valued quantification of how well a candidate forecasting algorithm can predict the values ​​of a given time series with specific characteristics. Therefore, the characteristic values ​​describing a time series can be associated with the forecasting algorithm's predictive performance on the time series. Once the forecast error has been generated for each time series, the algorithm selection system ranks the candidate forecasting algorithms based on their performance for each time series in its test set for the time series. In one embodiment, the ranking can be in ascending order of forecast error. That is, the candidate algorithm with the smallest forecast error for the time series is ranked first in terms of performance for the time series, the candidate algorithm with the second lowest forecast error is ranked second in terms of performance, and so on, with the candidate algorithm with the highest forecast error for the time series ranked so that it is ranked lowest in terms of performance for the time series. These rankings of the candidate forecasting algorithms for the time series can be stored in association with the characteristic vector of the time series.

[0129] Now for reference Figure 3HThe algorithm selection system continues to train a ranking function 370, which assigns a ranking to each algorithm based on an S-dimensional (e.g., 22-dimensional) feature vector representing the input sequence. This ranking function 370 is learned by examining 10,000 pairs of feature vectors (such as catch-22 vectors) and the corresponding algorithm ranking for each vector. In one embodiment, the ranking function is a machine learning (ML) classification algorithm configured to assign rankings to candidate prediction algorithms as classifications based on inputs of S feature values. In one embodiment, the ranking function is a multi-class ML classification algorithm, such as k-nearest neighbors, decision trees, Naive Bayes, random forests, or gradient boosting algorithms.

[0130] Alternatively, in one embodiment, the ranking function 370 is a scoring function trained to estimate the forecast error for each candidate forecasting algorithm based on the provided feature vector. For example, the ranking / scoring function 370 could be a multi-output ML regression model. The ranking / scoring function 370 estimates the forecast error of applying various candidate forecasting algorithms to the time series based on the characteristics of the time series.

[0131] like Figure 3I As shown, utilizing an established ranking / scoring function used as a multi-functional forecasting algorithm selector, the algorithm selection system can determine how well each of the candidate forecasting algorithms 348 performs on a given time series 372 based on a feature vector 374 (e.g., a catch-22 vector) for time series 372. Time series 372 can be of any length and can be obtained from a wide variety of domains. Time series 372 is a time series outside of the initial / training and supplementary / test sets. In one embodiment, time series 372 is a time series for which the most suitable forecasting algorithm is automatically selected for forecast estimation of time series 372. By obtaining the feature vector 374 corresponding to time series 372 and passing this feature vector 374 to the trained ranking / scoring function 370, the algorithm selection system can efficiently determine the ranking 376 of the candidate forecasting algorithms 348. The ranking 376 indicates how accurate the prediction of time series 372 will be by each candidate forecasting algorithm 348.

[0132] The algorithm selection system can then select the top few algorithms 378 (such as the top 3 algorithms) from the list of ranked algorithms. These top few algorithms 378 can then be further evaluated to obtain the optimal forecast performance. The algorithm selection system feeds the time series into each of the top 3 algorithms and calculates their corresponding Rolling Start Cross-Validation (ROCV) errors: Algorithm 1 ROCV error 380, Algorithm 2 ROCV error 382, ​​and Algorithm 3 ROCV error 384. In one embodiment, rolling start cross-validation is performed by incrementally training the candidate algorithms on observations further back in time series 372 and comparing the values ​​of the time series in observations after the values ​​used for training with the actual values ​​of the time series at that observation. In other words, the algorithm is trained on time series 372 until the cutoff observation, recording the difference between the actual observations beyond the cutoff observation and the estimates of the actual observations generated by the trained algorithm, and incrementally advancing the cutoff observation. This process is repeated in a loop of incremental advancement of the cutoff observation, and the ROCV error is calculated based on the pooling of the recorded differences. In one embodiment, the ROCV error is calculated using MASE over all observations.

[0133] The algorithm selection system then uses the ROCV errors 380, 382, ​​and 384 of the top few algorithms 378 to perform algorithm selection 386. ROCV error analysis serves as a safeguard against overfitting to the test dataset. At algorithm selection 386, the algorithm selection system ultimately selects the algorithm that performs best in terms of ROCV error to generate the final forecast. For example, algorithm selection 386 selects the forecasting algorithm with the lowest ROCV error among the top few forecasting algorithms 378 as the top-ranked algorithm. The top-ranked algorithm selected by algorithm selection 386 is thus among the top few algorithms 378 ranked by prediction accuracy, and has been confirmed to have minimal (or no) overfitting to the test dataset due to its low ROCV error.

[0134] The algorithm selection system then deploys the top-ranked algorithm to predict the values ​​of 372 from the 388 input time series. In one embodiment, the predictions from the top-ranked algorithm will be of high quality and produced at a much lower computational (processor and / or memory) cost compared to standard brute-force methods. In one embodiment, the algorithm selection system and method described herein outperform methods such as Feature-Based Prediction Model Selection (FFORMS) because the algorithm selection system and method cover a broader range of areas not addressed by FFORMS.

[0135] —Specific features for time series generation—

[0136] In one embodiment, a specific set of time series characteristics is used for time series generation via a characteristic-based time series generator and for analysis to determine appropriate forecasting algorithms for the time series. Specifically, these are a set of 22 canonical characteristics of time series identified by Carl H. Lubba et al. catch22: Canonical Time- series CHaracteristics 33 Data Mining and Knowledge Discovery 1821, 1833 (August 9, 2019), available at https: / / link.springer.com / article / 10.1007 / s10618-019-00647-x. These 22 properties concisely describe how data in a time series evolves or changes over time (also known as the "dynamics" of the time series). For example, properties can describe the underlying processing, trends, and structure of the observed data points that cause the time series.

[0137] These 22 characteristics are listed in Table 1 below, categorized by the type of information conveyed.

[0138]

[0139] In one embodiment, features other than the 22 features listed in Table 1 can also be used to perform feature-based time series generation using a modified autoencoder.

[0140] —The advantages of the choice—

[0141] In one embodiment, the algorithm selection system improves time series forecasting techniques by providing enhanced forecast accuracy. By employing a versatile and adaptive algorithm selection approach within the algorithm selection system, more accurate time series forecasting solutions can be constructed. The enhanced forecast accuracy enables improved decision-making by downstream systems, leading to improved results.

[0142] In one embodiment, the algorithm selection system improves time series forecasting techniques by extending automatic forecasting algorithm selection to cross-industry applications where automatic algorithm selection was previously impossible. The domain-agnostic nature of the algorithm selection system makes it suitable for various industries that generate time series characteristics beyond those available in publicly available datasets. Therefore, in one embodiment, the algorithm selection system improves time series forecasting techniques by providing more accurate forecasting solutions that are domain-tailored to the time series data used by client systems for monitoring.

[0143] In one embodiment, the algorithm selection system improves time series forecasting techniques by increasing the efficiency of computational resource utilization. For example, the algorithm selection system reduces the time and computational resources used for selecting forecast models and for tuning forecast models. Because the forecast model is trained using a test set that has already been supplemented to fill gaps in the feature space, it will fit the time series data provided by the client system for monitoring with almost no tuning required.

[0144] In one embodiment, the algorithm selection system improves time series forecasting techniques by enhancing the scalability of forecast model selection. In particular, the algorithm selection system's ability to handle a wide range of time series behaviors makes it highly scalable, allowing the selected model to be easily adapted to new markets, industries, or customer needs.

[0145] —Detection and electronic alarms for abnormal deviations—

[0146] In one embodiment, an electronic alarm is generated by orchestrating and transmitting computer-readable messages. The computer-readable messages may include content describing the deviation from the predicted value that triggered the alarm, such as the time or observation at which the deviation was detected, an indication of the time series value that caused the anomaly, and the signal source to which the alarm applies (associated with the additional time series being monitored).

[0147] In one embodiment, an electronic alarm can be generated and sent in response to the detection of a deviation from a predicted time series value. For example, a deviation from the predicted value can be detected (and found to be sufficient grounds for an alarm) if the residuals between the actual and predicted values ​​satisfy a sequential probability ratio test (SPRT) analysis. For instance, SPRT calculates the cumulative sum of log-likelihood ratios for each successive residual between the actual and estimated values ​​of the signal, and compares this cumulative sum to a threshold indicating an abnormal deviation. If this threshold is crossed, an abnormal deviation is detected, and an electronic alarm indicating this deviation can be generated in response. This electronic alarm can be programmed and then transmitted for subsequent display on a display or other actions.

[0148] In one embodiment, an electronic alarm is a message configured to be transmitted over a network (such as a wired network, cellular phone network, Wi-Fi network, or other communication infrastructure). The electronic alarm can be configured to be read by a computing device. The electronic alarm can be configured to be used in response to a request (such as a REST request) to trigger an automated function in response to the detection of an anomaly. In one embodiment, the electronic alarm can be presented in a user interface such as a graphical user interface (GUI) by retrieving its content from a REST API that has already received it. The GUI can present messages, notifications, or other indications that an anomaly has occurred.

[0149] —Cloud or Enterprise Implementation Examples—

[0150] In one embodiment, the system (such as algorithm selection system 100) is a computing / data processing system comprising a collection of computing applications or distributed computing applications accessible and usable by other client computing devices communicating with the system via a network. In one embodiment, algorithm selection system 100 is a component configured to collect, provide, and perform operations on time-series data as a time-series data service. The application and computing system may be configured to operate with or be implemented as a cloud-based network computing system, Infrastructure as a Service (IAAS), Platform as a Service (PAAS), or Software as a Service (SaaS) architecture, or other types of networked computing solutions. In one embodiment, the system provides at least one or more of the functions disclosed herein, along with a graphical user interface for accessing and operating these functions. In one embodiment, algorithm selection system 100 is a centralized server-side application that provides at least the functions disclosed herein and is accessible to a number of users via computing devices / terminals communicating with computers that communicate with algorithm selection system 100 (which acts as one or more servers) via a computer network. In one embodiment, the algorithm selection system 100 may be implemented by a server or other computing device configured with hardware and software to implement the functions and features described herein.

[0151] In one embodiment, components of the algorithm selection system 100 may be implemented as a collection of one or more software modules executed by one or more computing devices specifically configured for such execution. In one embodiment, components of the algorithm selection system 100 are implemented on one or more hardware computing devices or hosts interconnected via a data network. For example, components of the algorithm selection system 100 may be executed by network-connected computing devices of one or more computing hardware shapes, such as a central processing unit (CPU) or general-purpose shape, intensive input / output (I / O) shape, graphics processing unit (GPU) shape, and high-performance computing (HPC) shape.

[0152] In one embodiment, components of the algorithm selection system 100 communicate with each other via electronic messages or signals. These electronic messages or signals can be configured to call functions or procedures that access the characteristics or data of a component, such as, for example, application programming interface (API) calls. In one embodiment, these electronic messages or signals are sent between hosts in a format compatible with Transmission Control Protocol / Internet Protocol (TCP / IP) or other computer networking protocols. Components of the algorithm selection system 100 can (i) generate or compose electronic messages or signals to issue commands or requests to another component, (ii) transmit messages or signals to other components of the algorithm selection system 100, (iii) parse the content of received electronic messages or signals to identify commands or requests that a component can execute, and (iv) automatically execute or enforce commands or requests in response to the identification of the commands or requests. Electronic messages or signals may include queries against a database. Queries can be written and executed using a query language compatible with the database and executed in a runtime environment compatible with the query language.

[0153] In one embodiment, a remote computing system can access information or applications provided by the algorithm selection system 100 (e.g., via a web interface server). In one embodiment, the remote computing system can send requests to and receive responses from the algorithm selection system 100. In one example, access to the information or applications can be achieved using a web browser on a personal computer or mobile device. In one example, communication exchanged with the algorithm selection system 100 can take the form of Remote Representation State Transfer (REST) ​​requests, such as using JavaScript Object Notation (JSON) as the data exchange format, or sending Simple Object Access Protocol (SOAP) requests to or from an XML server. REST or SOAP requests can include API calls to components of the algorithm selection system 100.

[0154] —Software Module Examples—

[0155] Generally, software instructions are designed to be executed by one or more appropriately programmed processors accessing memory. Software instructions can include, for example, computer-executable code and source code that can be compiled into computer-executable code. These software instructions can also include instructions written in interpreted programming languages ​​such as scripting languages.

[0156] In complex systems, such instructions can be arranged into program modules, each performing a specific task, process, function, or operation. The entire set of modules can be operationally controlled or coordinated by an operating system (OS) or other form of organizational platform.

[0157] In one embodiment, one or more of the components described herein are configured as modules stored in a non-transitory computer-readable medium. The modules are configured with stored software instructions that, when executed by at least processor access to memory or a storage device, cause a computing device to perform one or more corresponding functions as described herein.

[0158] In one embodiment, the algorithm selection system and method described herein can be implemented using a computer program product comprising a computer program / instruction that, when executed by a processor, causes the processor to perform any of the methods described herein.

[0159] —Computing Device Examples—

[0160] Figure 4 An example computing system 400 is illustrated, configured and / or programmed as a dedicated computing device having one or more of the example systems and methods described herein and / or their equivalents(s). The example computing device may be a computer 405 including at least one hardware processor 410, memory 415, and input / output ports 420 operably connected via a bus 425. In one example, computer 405 may include feature-based algorithm selection logic 430 configured to facilitate feature-based selection of time series forecasting algorithms, as referenced herein. Figure 1 , Figure 2 , Figures 3A-3I The logic, system, and other embodiments shown and described herein are similar.

[0161] In various examples, logic 430 may be implemented in hardware, one or more non-transitory computer-readable media 437 storing instructions, firmware, and / or a combination thereof. Although logic 430 is shown as a hardware component attached to bus 425, it should be appreciated that in other embodiments, logic 430 may be implemented in processor 410, stored in memory 415, or stored in disk 435.

[0162] In one embodiment, logic 430 or computer is a component (i.e., structure: hardware, non-transitory computer-readable medium, firmware) for performing the described actions. In some embodiments, the computing device may be a server operating in a cloud computing system, a server configured in a Software as a Service (SaaS) architecture, a smartphone, a laptop computer, a tablet computing device, etc.

[0163] For example, the component can be implemented as an application-specific integrated circuit (ASIC) programmed to facilitate the selection of time series forecasting algorithms based on characteristics. The component can also be implemented as stored computer-executable instructions presented as data 440 to computer 405, temporarily stored in memory 415, and then executed by processor 410.

[0164] Logic 430 may also provide components (e.g., hardware, nontransitory computer-readable medium storing executable instructions, firmware) for performing one or more publicly disclosed functions and / or combinations of functions.

[0165] In a typical example configuration describing computer 405, processor 410 can be a variety of processors, including dual-microprocessor and other multiprocessor architectures. Memory 415 can include volatile memory and / or non-volatile memory. Non-volatile memory can include, for example, read-only memory (ROM), programmable ROM (PROM), etc. Volatile memory can include, for example, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), etc.

[0166] Storage disk 435 can be operatively connected to computer 405 via, for example, an input / output (I / O) interface (e.g., card, device) 445 and input / output port 420 controlled by at least input / output (I / O) controller 447. Disk 435 can be, for example, a disk drive, solid-state drive, floppy disk drive, tape drive, Zip drive, flash memory card, memory stick, etc. Furthermore, disk 435 can be an optical disc (CD-ROM) drive, a recordable CD (CD-R) drive, a rewritable CD (CD-RW) drive, a digital video disc ROM (DVD ROM) drive, etc. Therefore, the storage device / disk can include one or more non-transitory computer-readable media. Memory 415 can store, for example, processing 450 and / or data 440. Disk 435 and / or memory 415 can store an operating system that controls and allocates resources of computer 405.

[0167] Computer 405 can interact with, control, and / or be controlled by input / output (I / O) devices via input / output (I / O) controller 447, I / O interface 445, and input / output port 420. Input / output devices may include, for example, one or more network devices 455, a display 470, a printer 472 (such as an inkjet printer, laser printer, or 3D printer), an audio output device 474 (such as a speaker or headset), a text input device 480 (such as a keyboard), a cursor control device 482 for pointing and selecting input (such as a mouse, trackball, touchscreen, joystick, pointing stick, electronic stylus, electronic writing tablet), an audio input device 484 (such as a microphone or external audio player), a video input device 486 (such as a camcorder and still camera, or an external video player), an image scanner 488, a video card (not shown), a disk 435, etc. Input / output port 420 may include, for example, a serial port, a parallel port, and a USB port.

[0168] Computer 405 can operate in a network environment and can therefore be connected to network device 455 via I / O interface 445 and / or I / O port 420. Through network device 455, computer 405 can interact with network 460. Through network 460, computer 405 can logically connect to remote computer 465. Networks with which computer 405 can interact include, but are not limited to, local area networks (LANs), wide area networks (WANs), and other networks.

[0169] —Definitions and Other Embodiments—

[0170] In another embodiment, the described methods and / or their equivalents may be implemented using computer-executable instructions. Thus, in one embodiment, a non-transitory computer-readable / storage medium is configured to have stored computer-executable instructions of an algorithm / executable application that, when executed by one or more machines, cause the one or more machines (and / or associated components) to perform the method. Example machines include, but are not limited to, processors, computers, servers operating in cloud computing systems, servers configured with a Software as a Service (SaaS) architecture, smartphones, etc. In one embodiment, the computing device is implemented using one or more executable algorithms configured to perform any of the disclosed methods.

[0171] In one or more embodiments, the disclosed methods or their equivalents are performed by any of: computer hardware configured to perform the method; or, computer instructions embodied in a module stored in a non-transitory computer-readable medium, wherein the instructions are configured to execute an algorithm that is configured to perform the method when executed at least by a processor of a computing device.

[0172] While the methods illustrated in the diagrams are shown and described as a series of boxes representing the algorithm for the purpose of simplification, it should be understood that these methods are not restricted by the order of the boxes. Some boxes may appear in a different order than those shown and described, and / or may appear simultaneously with other boxes. Moreover, example methods can be implemented using fewer boxes than are shown in all the diagrams. Boxes can be combined or divided into multiple actions / components. Furthermore, additional and / or alternative methods may employ additional actions not illustrated in the boxes.

[0173] The following includes definitions of the selected terms used herein. Definitions include various examples and / or forms of components that fall within the scope of the term and can be used to implement it. These examples are not intended to be restrictive. Both singular and plural forms of the terms may be included in the definitions.

[0174] References to "an embodiment," "an embodiment," "an example," "an example," etc., indicate that one or more embodiments or examples as described may include a particular feature, structure, characteristic, property, element, or limitation, but not every embodiment or example must include that particular feature, structure, characteristic, property, element, or limitation. Furthermore, repeated use of the phrase "in one embodiment" does not necessarily refer to the same embodiment, but may refer to the same embodiment.

[0175] As used herein, a “data structure” is an organization of data stored in memory, storage devices, or other computerized systems within a computing system. A data structure can be any of, for example, a data field, a data file, a data array, a data record, a database, a data table, a graph, a tree, a linked list, etc. A data structure can be formed from and contain many other data structures (e.g., a database includes many data records). Other examples of data structures are also possible according to other embodiments.

[0176] As used herein, "computer-readable medium" or "computer storage medium" means a non-transitory medium that stores instructions and / or data configured to perform one or more of the disclosed functions when executed. In some embodiments, data may be used as instructions. Computer-readable media may take the form of, but is not limited to, non-volatile and volatile media. Non-volatile media may include, for example, optical discs, magnetic disks, etc. Volatile media may include, for example, semiconductor memory, dynamic memory, etc. Common forms of computer-readable media may include, but are not limited to, floppy disks, flexible disks, hard disks, magnetic tapes, other magnetic media, application-specific integrated circuits (ASICs), programmable logic devices, compact discs (CDs), other optical media, random access memory (RAM), read-only memory (ROM), memory chips or cards, memory sticks, solid-state storage devices (SSDs), flash drives, and other media in which a computer, processor, or other electronic device can operate. If each type of medium is selected for implementation in one embodiment, it may include stored instructions of an algorithm configured to perform one or more of the disclosed and / or claimed functions.

[0177] As used herein, “logic” means a component implemented using computer or electrical hardware, a non-transitory medium having stored instructions for executing applications or programs, and / or a combination thereof, to perform any function or action disclosed herein, and / or to cause a function or action from another logic, method, and / or system to be performed as disclosed herein. Equivalent logic may include firmware, a microprocessor programmed with an algorithm, discrete logic (e.g., an ASIC), at least one circuit, analog circuit, digital circuit, programmable logic device, memory device containing instructions for an algorithm, etc., any of which may be configured to perform one or more of the disclosed functions. In one embodiment, logic may include one or more gates, combinations of gates, or other circuit components configured to perform one or more of the disclosed functions. When describing multiple logics, the multiple logics may be combined into one logic. Similarly, when describing a single logic, that single logic may be distributed among multiple logics. In one embodiment, one or more of these logics are corresponding structures associated with performing the disclosed and / or claimed functions. The choice of which type of logic to implement may be based on desired system conditions or specifications. For example, hardware implementation of the function would be chosen if higher speed is considered. If lower cost is a consideration, then stored instructions / executable applications will be chosen to implement the functionality.

[0178] An "operable connection," or a connection through which entities are "operably connected," is a connection capable of sending and / or receiving signals, physical communication, and / or logical communication. An operable connection may include physical interfaces, electrical interfaces, and / or data interfaces. An operable connection may include various combinations of interfaces and / or connections sufficient to allow operable control. For example, two entities may be operably connected to transmit signals to each other directly or through one or more intermediate entities (e.g., processors, operating systems, logic, non-transitory computer-readable media). Logical and / or physical communication channels can be used to create an operable connection.

[0179] As used herein, “user” includes, but is not limited to, one or more persons, computers or other devices, or a combination of these.

[0180] While the disclosed embodiments have been illustrated and described in considerable detail, they are not intended to limit the scope of the appended claims or in any way restrict them to such detail. It is certainly impossible to describe every conceivable combination of components or methods in order to describe all aspects of the subject matter. Therefore, this disclosure is not limited to the specific details or illustrative examples shown and described. Consequently, this disclosure is intended to cover changes, modifications, and variations that fall within the scope of the appended claims.

[0181] As to the extent to which the term “comprising” is used in the specific embodiments or claims, it is intended to be inclusive in a manner similar to that interpreted when the term “comprising” is used as a transitional word in the claims.

[0182] As far as the term "or" is used in the specific embodiments or claims (e.g., A or B), it is intended to mean "A or B or both". When the applicant intends to indicate "only A or B but not both", then the phrase "only A or B but not both" will be used. Therefore, the use of the term "or" herein is inclusive rather than exclusive.

Claims

1. One or more non-transitory computer-readable media, said one or more non-transitory computer-readable media comprising computer-executable instructions stored thereon, said computer-executable instructions causing the computer, when executed by at least a processor of a computer, to: A time series generator is created by: (1) accessing the first set of time series; (2) identifying N features of each time series in the first set of time series; and (3) training an autoencoder with N nodes in the bottleneck layer based on a loss function that minimizes the difference between the bottleneck layer activation and the N features. And (4) set the bottleneck layer as the input to the time series generator; Determine the gaps in the first set of time series in the N-dimensional characteristic space; One or more feature vectors for filling the gaps are input into the bottleneck layer to generate a time series set of gap fillings; The first set of time series is combined with the gap-filling set of time series to generate a combined set of time series; The combined set of time series is input into multiple candidate prediction algorithms, where each candidate prediction algorithm generates a prediction value based on the combined set of time series; The prediction error of the multiple candidate prediction algorithms is determined based at least on a combination of predicted values ​​and time series. A machine learning model is trained based on forecast error and N features to determine the ranking of forecasting algorithms used to forecast a given time series. as well as At least one of the candidate forecasting algorithms is selected based on the ranking to generate a forecast for a given time series.

2. One or more non-transitory computer-readable media as claimed in claim 1, wherein the bottleneck layer comprises N nodes corresponding to N characteristics of the time series.

3. One or more non-transitory computer-readable media as described in claim 1 or 2, wherein time series in a first set of time series are synthesized to simulate a range of time series behaviors, including outliers, multiple seasonality, change points, intermittency, and high-level effects.

4. One or more non-transitory computer-readable media as claimed in any one of claims 1 to 3, wherein the loss function is configured to minimize the difference between the bottleneck layer activation and the feature vector by evaluating a combination error of: a) the difference between the output and the input, and b) the difference between the bottleneck layer and the feature features of a first set of time series.

5. One or more non-transitory computer-readable media as described in any one of claims 1 to 4, wherein using a bottleneck layer as an input layer allows for the generation of a combined set of time series based on a feature vector selected to minimize the gap between two nearest points in an N-dimensional feature space.

6. One or more non-transitory computer-readable media as described in any one of claims 1 to 5, wherein the candidate prediction algorithm comprises one or more of the following: an autoregressive integral moving average (ARIMA) model, an error-trend-seasonal (ETS) model, a deep learning model, an error feedback estimation (EFE) model, and a prophet model.

7. One or more non-transitory computer-readable media as described in any one of claims 1 to 6, wherein the N characteristics of each time series in the first set of time series include: The z-score distribution includes the mode, the longest period of consecutive values ​​above the mean, the time interval between successive extreme events above the mean, the time interval between successive extreme events below the mean, the first 1 / e crossover point of the autocorrelation function, the first minimum of the autocorrelation function, the total power in the lowest part of the Fourier power spectrum, the centroid of the Fourier power spectrum, the average error of the rolling multi-sample mean forecast, time reversibility statistics, self-mutual information, the first minimum of the self-mutual information function, the proportion of a given coefficient whose successive difference exceeds the standard deviation, the longest period of successive increment reduction, the Shannon entropy of two successive letters in an equally probable 3-letter symbolization, the change in correlation length after iterative differencing, and the exponential fit of successive distances in a 2D embedding space.

8. A computing system, comprising: At least one processor is connected to at least one memory; A non-transitory computer-readable medium, including instructions stored thereon, which, when executed by at least a processor, cause a computing system to: Analyze each time series in the training set to generate a vector of N features for each time series. The autoencoder is trained using a loss function that minimizes: (1) the error between the bottleneck layer activations in the autoencoder and the vector of characteristics of the time series, and (2) the error between the input layer and the output layer of the autoencoder; One or more new vectors of N features are generated by minimizing the gap between neighboring points in the N-dimensional feature space; The test set for time series is generated at least in part by inputting a new vector of N features into the bottleneck layer of a trained autoencoder. The test set of time series is input into each of the different forecasting algorithms in the set, and the forecast error of each algorithm is calculated based on its performance for each time series. The ranking function is trained to assign a ranking to each prediction algorithm based on a vector of N provided features; as well as This involves automatically selecting one forecasting algorithm from the N features of the additional time series using a ranking function to monitor the additional time series.

9. A computer-implemented method, the method comprising: Process multiple time series in the training set of time series to generate a vector of N features for each of the multiple time series; The autoencoder is trained based on a loss function that minimizes: (1) the difference between the activation of nodes in the bottleneck layer of the autoencoder and the vector of N features of the time series entered at the input layer of the autoencoder, and (2) the difference between the value of the time series entered at the input layer and the value at the output layer of the autoencoder. One or more new vectors of N features are generated by minimizing the gap between neighboring points in the N-dimensional feature space; The test set for time series is generated at least in part by inputting a new vector of N features into the bottleneck layer of a trained autoencoder. The test set of time series is input into each of the different forecasting algorithms in the set, and the forecast error of each algorithm is calculated based on its performance for each time series. The ranking function is trained to assign a ranking to each prediction algorithm based on a vector of N provided features; as well as This involves automatically selecting one forecasting algorithm from the N features of the additional time series using a ranking function to monitor the additional time series.

10. The computer-implemented method of claim 9, wherein the bottleneck layer comprises N nodes corresponding to N characteristics in a vector of N characteristics.

11. The computer-implemented method of claim 9 or 10, wherein automatically selecting one forecasting algorithm from the forecasting algorithms to monitor the additional time series further comprises: The prediction algorithm is ranked using an N-dimensional vector with an additional time series and a trained ranking function. Select three top-ranked prediction algorithms; Calculate the cross-validation error of the top-ranked algorithms relative to the additional time series; and The algorithm with the smallest corresponding cross-validation error is selected as the top-ranked prediction algorithm.

12. The computer-implemented method of any one of claims 9 to 11, wherein minimizing the gap between neighboring points in the N-dimensional feature space is performed based on t-distributed random neighbor embedding.

13. The computer-implemented method according to any one of claims 9 to 12, further comprising: Forecast the values ​​of the additional time series using the selected forecasting algorithm; as well as An alert is generated when the actual value of the attached time series differs from the predicted value.

14. The computer-implemented method of any one of claims 9 to 13, wherein the length of each time series in the training set and the test set is equal.

15. The computer-implemented method of any one of claims 9 to 14, wherein processing a plurality of time series to generate a vector of N features further comprises generating a vector of N features to include a plurality of catch-22 features.

Images