Automated model selection for time-series outlier detection via META-learning

Abstract

Methods and systems for identifying outliers present in query segments of time-series data and for ranking outlier detection models are disclosed. A computer system can train and use a meta-learner machine learning model. This meta-learner may take as inputs query subsegments of time-series data, which may be derived from a query segment containing one or more outliers, along with baseline subsegments, which may be derived from a baseline segment that does not contain outliers. The meta-learner can determine which types of outliers are contained in the query segment, and can additionally generate ranking lists, which can rank different outlier detection models based on their expected ability to detect said outliers. Using these ranking lists, outlier detection models can be automatically selected and used for outlier detection.

Description

AUTOMATED MODEL SELECTION FOR TIME-SERIES OUTLIER DETECTION VIA META-LEARNINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is non-provisional and claims the benefit of U.S. Provisional Patent Application No. 63 / 511,811, entitled “AUTOMATED MODEL SELECTION FOR TIMESERIES OUTLIER DETECTION VIA META-LEARNING, “ filed on July 3, 2023, which is hereby incorporated by reference in its entirety for all purposes.BACKGROUND

[0002] Time-series outliers can have diverse properties, and as such, some research has been performed in order to identify and detect specific ty pes of outliers. In the medical field, for example, certain types of outliers in time-series heartrate data may correlate with different types of heart disease, and therefore detecting such time-series outliers may be useful for diagnosing and treating patients.

[0003] There are a wide variety of outlier detection models and techniques that can be used to detect a single type of outlier in time-series data. However, no single know n method or technique is effective or valid for detecting multiple types of typical time-series outliers.This can be especially problematic because real-world time-series data often contains multiple types of outliers. As such, these methods often do not meet real-world time-series analysis or forecasting requirements.

[0004] Embodiments of the present disclosure address these and other problems, individually and collectively.SUMMARY

[0005] Methods and systems for identifying outliers present in query segments of timeseries data and for ranking outlier detection models are disclosed. A computer system can train and use a meta-leamer machine learning model. This meta-leamer may take as inputs query subsegments of time-series data, which may be derived from a query segment containing one or more outliers, along with baseline subsegments, which may be derived from a baseline segment that does not contain outliers. The meta-leamer can determinewhich types of outliers are contained in the query7segment, and can additionally generate a ranking lists, which can rank different outlier detection models based on their expected ability to detect said outliers. Using these ranking lists, outher detection models can be automatically selected and used for outher detection.

[0006] One embodiment is directed to a method for using a machine learning model to detect outliers in input time-series data. The machine learning model can comprise a multilayer sub-model and a plurality of head sub-models. The method can be performed by a computer system. The computer system can receive the input time-series data. The input time-series data can include a baseline segment and a query segment. The computer system can segment the baseline segment into one or more baseline subsegments. Likewise, the computer system can segment the query segment into one or more query7subsegments. The computer system can apply the multilayer sub-model to the one or more baseline subsegments and the one or more query subsegments, thereby producing an embedding matrix. The multilayer sub-model can include one or more attention layers. The computer system can apply each head sub-model of the plurality of head sub-models to the embedding matrix, thereby producing a plurality of outlier segment classifications. Each head sub-model of the plurality of head sub-models can correspond to a different outlier type of a plurality of outlier types. Each outher segment classification of the plurality of outlier segment classifications can indicate whether the query7segment contains one or more outliers of an outlier type corresponding to that outlier segment classification.

[0007] Another embodiment is directed to a method for training a machine learning model comprising a multilayer sub-model and a plurality of head sub-models to detect outliers in input time-series data. The method can be performed by a computer system. The computer system can receive a training set comprising a plurality7of query7training segments and a plurality of baseline training segments. The computer system can perform an iterative training process until a terminating condition has been met. The iterative training process can comprise a plurality of training rounds. In each training round, the computer system can sample a baseline training segment from the plurality7of baseline training segments and sample a query training segment from the plurality of query training segments. The computer system can segment the baseline training segment into one or more baseline subsegments. Likewise, the computer system can segment the query training segment into one or morequery subsegments. The computer system can apply the multilayer sub-model to the one or more baseline subsegments and the one or more query subsegments, thereby producing an embedding matrix. The multilayer sub-model can be defined by a set of multilayer submodel parameters. The multilayer sub-model can also include one or more attention layers. The computer system can apply each head sub-model of the plurality of head sub-models to the embedding matrix, thereby producing a plurality of outlier segment classifications. Each head sub-model of the plurality of head sub-models can correspond to a different outlier type of a plurality of outlier types. Each outlier segment classification of the plurality of outlier segment classifications can indicate whether the query training segments contains one or more outliers of an outlier type corresponding to that outlier segment classification. Each head sub-model of the plurality of head sub-models can be defined by a different set of head sub-model parameters of a plurality of sets of head sub-model parameters. The computer system can generate one or more loss values based on the plurality of outlier segment classifications. The computer system can update the machine learning model by updating the set of multilayer sub-model parameters and the plurality of sets of head sub-model parameters based on the one or more loss values. The computer system can perform the iterative training process until the terminating condition has been met..

[0008] Another embodiment is directed to a computer system comprising one or more processors and a non-transitory computer readable medium coupled to the one or more processors, the non-transitory computer readable medium comprising code or instructions, executable by the one or more processors for performing either of the above methods (or any other methods described herein).TERMS

[0009] A “server computer” may include a powerful computer or cluster of computers. For example, the server computer can include a large mainframe, a minicomputer cluster, or a group of servers functioning as a unit. In one example, a server computer can include a database server coupled to a web server. The server computer may comprise one or more computational apparatuses and may use any of a variety of computing structures, arrangements, and compilations for servicing the requests for one or more client computers.

[0010] A “memory ” may include any suitable device or devices that may store electronic data. A suitable memory may comprise a non-transitory computer readable medium thatstores instructions that can be executed by a processor to implement a desired method. Examples of memories include one or more memory chips, disk drives, etc. Such memories may operate using any suitable electrical, optical, and / or magnetic mode of operation. A ■‘memory buffer” can include a region of memory used to temporarily store data.

[0011] A “processor” may include any suitable data computation device or devices. A processor may comprise one or more microprocessors working together to accomplish a desired function. The processor may include a CPU that comprises at least one high-speed data processor adequate to execute program components for executing user and / or system generated requests. The CPU may be a microprocessor such as AMD’s Athlon, Duron and / or Opteron; IBM and / or Motorola’s PowerPC; IBM’s and Sony’s Cell processor; Intel’s Celeron, Itanium. Pentium, Xenon, and / or XScale; and / or the like processor(s).

[0012] A “data set” may include any set of one or more “observations” or “data values.” A “data value” can include any data element. A data value can comprise a “data vector,” one or more values (represented in vector form) corresponding to a data element or observation.Time series data may comprise data for which each data value has an associated “time value,” “time stamp,” or “time indicator.” A “time series data value” may include any data value corresponding to time series data. A “time value” can represent or correspond to the time at which the corresponding data value was observed or collected. Time series data is often ordered chronologically based on time values. A “segment” (sometimes referred to as a “sequence”) of time-series data can comprise a set of (usually sequential) time-series data observations. A “subsegment” of time-series data can comprise a subset of (usually sequential) time-series data observations derived from a segment. A "query segment” of time-series data can comprise a segment of time-series data that can be subject to some form of analysis or a "query." For example, a query segment of heartrate data (e.g., a collection of electrocardiography, seismo-cardiography, and gyro-cardiography signals) can be classified as corresponding to a normal healthy heart, or a heart exhibiting aortic stenosis.

[0013] “Sampling” may include any process or method used to collect data values. Sampling can be used to collect data values from an existing data set. The act of sampling may result in a “sample,” one or more data values collected from the data set during sampling. Data sets can be sampled via a variety of means. For example, “random sampling” involves sampling data values from a data set randomly. A “window” or “windowof data” may include any number of contiguous data elements from a data set. A “window” may be defined by a starting data value and an ending data value, such that the window contains all data values between the starting data value and ending data value (and optionally the starting data value and ending data values themselves). “Window sampling” can be used to sample data values contained within a window of data.

[0014] A “machine learning model” can refer to a software module configured to be run on one or more processors to provide a classification or numerical value of a property of one or more samples. An example type of model is supervised learning that can be used with embodiments of the present disclosure. Example supervised learning models may include different approaches and algorithms including analytical learning, artificial neural network, backpropagation, boosting (meta-algorithm), Bayesian statistics, case-based reasoning, decision tree learning, inductive logic programming, Gaussian process regression, genetic programming, group method of data handling, kernel estimators, learning automata, learning classifier systems, minimum message length (decision trees, decision graphs, etc.), multilinear subspace learning, naive Bayes classifier, maximum entropy classifier, conditional random field, nearest neighbor algorithm, probably approximately correct learning (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, subsymbolic machine learning algorithms, minimum complexity machines (MCM), random forests, ensembles of classifiers, ordinal classification, data pre-processing, handling imbalanced datasets, statistical relational learning, or Proaftn, a multicriteria classification algorithm. The model may include linear regression, logistic regression, deep recurrent neural network (e.g., long short term memory, LSTM), hidden Markov model (HMM), linear discriminant analysis (LDA), k-means clustering, densitybased spatial clustering of applications with noise (DBSCAN), random forest algorithm, support vector machine (SVM), or any model described herein. Supervised learning models can be trained in various ways using various cost / loss functions that define the error from the known label (e.g., least squares and absolute difference from known classification) and various optimization techniques, e.g., using backpropagation, steepest descent, conjugate gradient, and Newton and quasi-Newton techniques.

[0015] The process of “training” a machine learning model may include any steps used to prepare a machine learning model to perform some task. Often training involves determiningor optimizing a set of “parameters” (which characterize the machine learning model) which result in acceptable model performance. Training can be performed in a series of “training rounds” during which training data is used to update the parameters of the machine learning model, for example, based on a loss value.

[0016] A “loss value” or “error value” may include any value that indicates the deviation between a result of some process, method, or function and an expected, desired, or correct result. For example, if a machine learning model can detect anomalies in a data set comprising 100 data values, 17 of which are anomalous, if the machine learning model only detects 15 the 17 anomalous data values, the loss value could comprises, e.g., 2 (17 - 15). Loss values can be used to train and evaluate the training of machine learning models, e.g., by optimizing machine learning model parameters by minimizing the loss value, using processes such as stochastic gradient descent or backpropagation.

[0017] A “hyperparameter” can include any value used to configure a machine learning model that is external to the machine learning model. Typically, a hyperparameter is set, and is not estimated or determined from the training data that is used to train the machine learning model.

[0018] A machine learning model may comprise multiple “sub-models” or “layers,” which may refer to parts of a larger machine learning system. For example, a machine learning model could comprise a long short-term memory layer (which itself can comprise multiple layers), in addition to an attention layer and a linear layer. Layers can sometimes be organized in series, such that the input to a machine learning system is processed by a first set of layers, which produces an output that is then processed by a subsequent set of layers, and so forth until the output of the machine learning model is produced by the final layer in the series.

[0019] “Outlier detection” may include any process or method used to detect outliers. For example, outlier detection can be used to detect outlier data values in a data set. An “outlier” may include anything that deviates from what is standard, normal, or expected, particularly when it deviates to an unusual or extreme degree. An outlier data value can correspond to a particular outlier event. For example, for heartrate data, an outlier can comprise a data value corresponding to a cardiac episode, e.g., myocardial infarction. An “outlier score” may include any value generated during an outlier detection process. An outlier score can indicatethe likelihood that a data value comprises an outlier. For example, an outlier score of “0.81” can indicate that there is a 81% chance that a data value is an outlier.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG. 1 shows an exemplary time-series data segment containing multiple types of outliers.

[0021] FIG. 2 shows a diagram of an exemplary outlier detection process.

[0022] FIG. 3 shows an exemplary' time-series segment containing global point outliers.

[0023] FIG. 4 shows an exemplary' time-series segment containing contextual point outliers.

[0024] FIG. 5 shows an exemplary time-series segment containing shapelet outliers.

[0025] FIG. 6 shows an exemplary time-series segment containing seasonal outliers.

[0026] FIG. 7 shows an exemplary time-series segment containing trend outliers.

[0027] FIG. 8 shows an overview of an exemplary method of outlier detection model selection according to some embodiments.

[0028] FIG. 9 shows an overview of the structure of a meta-leamer machine learning model according to some embodiments.

[0029] FIGS. 10A-10D show a flowchart corresponding to a method of generating outlier segment classifications corresponding to a query segment according to some embodiments.

[0030] FIG. 11 shows an overview of a subsegmentation method according to some embodiments.

[0031] FIGS. 12A-12C show a detailed structural diagram of a meta-leamer machine learning model according to some embodiments.

[0032] FIGS. 13A-13C show detailed structural diagram of an attention layer according to some embodiments.

[0033] FIG. 14 show a structural diagram of ahead sub-model according to some embodiments.

[0034] FIGS. 15A-15E show a flowchart corresponding to a method of training a meta- leaming machine learning model according to some embodiments.

[0035] FIG. 16 shows a training, validation, and test data set according to embodiments.

[0036] FIG. 17 shows an exemplary computer system according to some embodiments.DETAILED DESCRIPTION

[0037] A variety of outlier detection methods, models, and techniques can be used to detect a single type of outlier or anomaly (these terms are used largely interchangeably herein) in time-series data, particularly a single outlier (or anomaly) type from among five major types of time-series outliers, i.e., global point outliers, contextual point outliers, shapelet outliers, seasonal outliers, and trend outliers. As examples, methods and techniques based on autoregression can be effective at detecting global point outliers, and methods and techniques based on the matrix profile can be effective for detecting shapelet outliers. However, no single technique is effective or valid for detecting multiple types of outliers in a given “query segment” of time series data. This can be problematic, as real-world time-series data often contains multiple types of outliers or anomalies, and as such, it can be difficult to detect multiple types of outliers in such query' segments using conventional outlier detection methods, models, or techniques.

[0038] Embodiments address this problem by using a “meta-leamer machine learning model” (sometimes more generically referred to as a “machine learning model” or a “meta- leamer”) to identify types of time-series outliers that may be present in a query segment, as well as optionally rank a plurality of outlier detection models based on their expected ability to detect those types of outliers.

[0039] The meta-leamer can recommend one or more suitable outlier detection models based on the identified types of time series outliers and rankings, which can then be used in order to detect the various types of outliers that could be present in the query segment, and therefore achieve high outlier detection performance. In this way, embodiments of the present disclosure can automatically determine one or more suitable outlier detection models for detecting outliers in a query’ segment of time-series data. Embodiments can be particularly useful for identifying five types of commonly studied outliers in time-series data.These outlier types (described in more detail below) include global points outliers, contextual points outliers, shapelet outliers, seasonal outliers, and trend outliers.

[0040] Methods and processes for training the meta-leamer during a "‘training phase,” and using the meta-leamer during a “production phase” are described in more detail below. In summary, during training, the machine learning model can leam to identify and differentiate between features in query segments and “baseline segments” that indicate whether a query segment likely contains time-series outliers of a variety of outlier types, and further indicate which outlier detection models are likely to perform well at detecting those outliers. Such baseline segments can comprise segments of time-series data that do not contain outliers. As a result of training, the machine learning model can leam to produce a plurality' of “outlier segment classifications” and a plurality' of “ranking lists” based on an input query segment of time-series data. Each outlier segment classification can correspond to an outlier type (e.g., of the five listed above), and can indicate whether the input query segment likely contains one or more time-series outliers of the corresponding outlier type. The outlier segment classifications can be used to determine, e.g., whether or not further outlier detection should be performed on the query segment. As an example, if a shapelet outlier segment classification indicates that there is a low probability of the query segment containing one or more shapelet outliers, then the query segment may not need to be evaluated using a shapelet outlier detection model.

[0041] Each ranking list can likewise correspond to an outlier ty pe and can rank the expected performance of outlier detection models at detecting time-series outliers of the corresponding outlier types in the query segment. These ranking lists can be used to select outlier detection models to detect outliers in the query' segment, e.g., by' selecting the “highest ranked” or best performing outlier detection models corresponding to each outlier segment classification. These best performing outlier detection model can then be used to detect any outliers in the query segment. For example, if shapelet model “B” is expected to perform better than shapelet model “A” (given their relative rankings in the shapelet ranking list), shapelet model B could be used to detect shapelet outliers in the query' segment, rather than shapelet model A.

[0042] As a real-world example, query segments could comprise time-series data corresponding to patient heartrates. Outliers in these heartrates can be indicative of differenttypes of disease. For example, the heartrate of a person suffering from hypo- or hyperthyroidism can exhibit the five classes of outliers described above. By contrast, the heartrate of a person with diabetes may exhibit global point outliers, contextual point outliers, and trend outliers, but may not exhibit seasonal outliers and shapelet outliers. As such, the ty pe and prevalence of outliers present in heartrate time-series data may be indicative of disease.

[0043] A meta-leamer machine learning model according to embodiments could be trained on heartrate data corresponding to healthy individuals and individuals with abnormal heartrates. After training, the heartrate data of a patent can be input into the machine learning model as a query segment, in order to produce a plurality7of outlier segment classifications and a plurality' of ranking lists. These outlier segment classifications could indicate that the query segment likely contained global point outliers, contextual point outliers, and trend outliers. The ranking lists corresponding to these three outlier types could be used to select the best performing outlier detection models for each outlier type, and these outlier detection models could be used to detect specific outliers in the query segment of heartrate data. A doctor could use the outputs of these outlier detection models to aid in diagnosing the patient.

[0044] Embodiments of the present disclosure can be applied to a variety of real-world applications, and not just the heartrate example provided above. For example, different types anomalies in time-series radio-frequency data can be indicative of radar or radar-warning malfunction. As another example, different types of anomalies in time-series credit card transaction data may be indicative of different forms of fraud. As such, embodiments of the present disclosure are useful in a wide variety of time-series anomaly detection activities.I. TIME SERIES DATA, OUTLIERS, AND MACHINE LEARNING

[0045] In order to facilitate a better understanding of embodiments of the present disclosure, some concepts related to time-series outlier detection and machine learning are summarized below.A. Time-Series

[0046] A time-series generally comprises a chronologically -ordered sequence of data values or “observations,” which can be displayed visually as a line or “waveform” on a graph. FIG. 1 shows an exemplary time-series 102 exhibiting both trend outliers 104 andshapelet outliers 106. Typically, such graphs or waveforms are organized chronologically from left to right. For the purpose of simplicity, most depictions of time-series in the figures are one-dimensional, that is, a single data value (represented in FIG. 1 by the vertical position of the time-series) is associated with each point in time. However, in reality, a single observation may comprise multiple time-series data values. As an example, a time-series observation corresponding to patient health data may comprise a heart rate data value, a pulse oximetry data value, a blood pressure data value, etc. Likewise, a time-series observation corresponding to a credit card transaction could comprise a transaction amount, a location associated with that transaction (e.g., a city or country where that transaction took place), an industry indicator (e.g., grocery store), etc.

[0047] As described above, the exemplary time-series 102 contains multiple types of outliers (e.g., trend outliers 104 and shapelet outliers 106), much like many real-world timeseries data sets. As such conventional techniques for detecting outliers (e.g., the application of a single outlier detection model, suitable for detecting a single type of outlier) may be ineffective for detecting both the trend outliers 104 and shapelet outliers 106 in time-series 102.

[0048] The concept of time-series outlier detection is described in broad detail below with reference to FIG. 2. FIG. 2 shows an outlier detection model, method, or process 206, which can take input time-series 202 as an input. This input time-series can comprise a temporally ordered sequence of samples, observations, or data values, which, when plotted, can take the form of a waveform or signal, e.g.. as depicted in FIG. 2. In FIG. 2, input time-series 202 contains outliers 204. These outliers, also referred to as anomalies, can comprise time-series samples, observations, or data values that differ from other samples, observations, or data values in the input time-series 202. In FIG. 2, outliers 204 are examples of seasonal outliers: data values that have a different frequency (in this case a lower frequency) than other data values in input time-series 202.

[0049] The outlier detection model, method, or process 206 can analyze or operate on the input time-series 202 in order to produce an outlier detection signal 208. This signal can comprise a time-series of probability data values. The outlier detection signal 208 can be the same “length'’ as the input time-series 202 (i.e., comprise the same number of observations as the input time-series 202). As such, each probability data value in the outlier detection signal208 can correspond to an observation or data value in the input time-series 202, and each probability data value in the outlier detection signal 208 can indicate the probability that the corresponding observation or data value in the input time-series 202 is an outlier. In FIG. 2, the outlier detection signal 208 approximates a step function (with some noise), indicating a high outlier probability for time-series data values corresponding to outliers 204 and a low outlier probability for other (non-outlier) time-series data values in input time-series 202.

[0050] An outlier detection model, method, or process (such as outlier detection model, method, or process 206) can output data or information indicating outliers in other forms. As such, the outlier detection signal 208 is intended only as a single non-limiting example of a useful output of such a model, method, or process, and is not intended to be limiting. As another example, outlier detection model, method, or process 206 could output an outlier detection table 210, indicating the location of outliers in the input time-series 202. As depicted in outlier detection table 210, the 50th-! 16thobservations from the input time-series 202 comprise detected outliers.

[0051] Alternatively or additionally, some form of data derived from the input time-series, such as a feature vector or embedding vector 212, can be applied as an input to the outlier detection method, model, or process. 206 in order to detect outliers. As described in more detail further below, in some embodiments, a multilayer machine learning model (also referred to as a “meta-1 earner”) can produce feature vectors (or other data derived from a query segment of time-series data) that can be applied to outlier detection models in order to detect outliers.

[0052] Specific outlier detection methods and models themselves are generally not described in much detail herein. It is broadly assumed that a practitioner of methods according to embodiments is already familiar with time-series outlier detection and has the knowledge and ability' to execute outlier detection according to known methods. For example, a practitioner could use techniques such as autoregression (alone or in combination with other signal processing techniques) to detect global point outliers (e.g., global maxima and minima). As another example, a practitioner could use a matrix profile in order to detect shapelet outliers or seasonal outliers.B. Various Outlier Types

[0053] As described above, a time-series can contain various types of outliers. While there are many different outlier types, there are five particular outlier ty pes that are relatively common in segments of time-series data. Many other outlier ty pes can be expressed as variations or combinations of these five outlier types. As such, these outlier types are described briefly below with reference to FIGS. 3-7. However, it should be understood that embodiments of the present disclosure can also be used to detect types of outliers other than (or in addition to) the five outlier types described below.

[0054] FIG. 3 shows an exemplary' time-series segment 302 with global point outliers, e.g., global point outlier 304 (comprising a global minimum) and global point outlier 306 (comprising a global maximum). A global point outlier can comprise a single time-series observation or data value that is larger or smaller (to an unusual or anomalous degree) than all other observations or data values in a given time-series data segment.

[0055] FIG. 4 shows an exemplary' time-series segment 402 with contextual point outliers 404 and 406. A contextual point outlier can comprise a single time-series observation or data value that is larger or smaller (to an unusual or anomalous degree) than other time-series observations or data values in a local region, i.e., a contextual point outlier can comprise a local maxima or minima. As depicted in FIG. 4, neither contextual point outlier 404 nor 406 comprises a global maxima, however, they are both of greater magnitude than the observations preceding and following them.

[0056] FIG. 5 shows an exemplary time-series segment 502 with shapelet outliers 504. A shapelet can comprise a section of the time-series that has an unusually different shape than the rest of the time-series segment. Shapelets can also have a different period than the rest of the time-series segment. Shapelet outliers, such as shapelet outliers 504 can comprise a series of sequential time-series observations corresponding to the shapelet.

[0057] FIG. 6 shows an exemplary time-series segment 602 with seasonal outliers 604. Seasonal outliers can comprise a series of sequential time-series observations corresponding to a section of the time-series which has the same shape, but has a different period (wavelength) from other sections of the time-series segment. In time-series segment 602, theseasonal outliers correspond to a section of the time-series with lower frequency (and therefore a longer wavelength) than other sections of time-series segment 602.

[0058] FIG. 7 shows an exemplary time-series segment 702 with trend outliers 704. Trend outliers can comprise a series of sequential time-series observations corresponding to a “trend” that is otherwise not present in other sections of the time-series segment. In timeseries segment 702, trend outliers 704 correspond to a steep upward trend in the time-series segment 702.

[0059] In embodiments of the present disclosure, a meta-leamer machine learning model can be used to produce a plurality of outlier segment classifications corresponding to a plurality of outlier types. In some embodiments, the plurality of outlier types can include two or more of the outlier ty pes described above, i.e., two or more of a global point outlier type, a contextual point outlier type, a shapelet outlier type, a seasonal outlier type, and a trend outlier type. Each outlier segment classifications can indicate whether an input querysegment of time-series data contains outliers of the corresponding type. As an example, a “shapelet” outlier segment classification can comprise a value such as “85%” indicating that there is a predicted 85% chance that the query segment contains one or more shapelet outliers.C. Machine Learning

[0060] A high level overview of machine learning is provided below in order to facilitate a better understanding of embodiments of the present disclosure. Machine learning models are often defined by sets of parameters, which generally control how the machine learning model produces output data responsive to received input data. As an example, a support vector machine (SVM) is a type of machine learning model that divides data points using a hyperplane. Data on one “side” of the hyperplane is classified as one class (e.g., normal) while data on the other side of the hyperplane is classified as another class (e.g., anomalous). The parameters of the support vector machine can comprise the coefficients used to define the hyperplane. Changing these parameters changes the shape of the hyperplane, and thus changes which data points the SVM classifies as normal or anomalous.

[0061] In broad terms, the process of training a machine learning model can involve determining the set of parameters that achieve the “best” performance, usually based on aloss or error function. A loss function relates the expected or ideal performance of the machine learning model to its actual performance on a (typically labeled) training data set. The loss function t pically decreases in value as the model’s performance improves. As such, training a machine learning model often involves determining the set of parameters that minimize a loss function corresponding to that model. Sometimes a random parameter estimate is generated as an initial parameter “guess,’’ and then a process such as gradient descent is used to iteratively refine the parameter estimate, eventually resulting in a final set of parameters associated with the machine learning model.

[0062] This iterative refinement process can be performed in a series of training “rounds”, “epochs”, or other appropriate divisions. In each round, a machine learning model's performance can be evaluated using the loss function, and the parameters can be updated based on this evaluation, e.g., with the goal of reducing the reducing the result over time. As an example, the gradient of the loss function can be determined in parameter space and can be used to reduce the value of the loss function in successive training rounds. Such a gradient corresponds to a change in model parameters that achieves the greatest immediate reduction in the loss function. By changing the model parameters based on the gradient, the loss function can be reduced during each successive training round. This process can be repeated until a terminating condition has been met. In embodiments of the present disclosure, one ty pe of terminating condition is a defined number of training rounds. This terminating condition can be met if the number of training rounds performed (e.g., by a computer system training the machine learning model) equals or exceeds the defined number of training rounds, at which point the iterative training process has been completed. Another ty pe of terminating condition in embodiments is a convergence condition. This terminating condition can be met if the machine learning model parameters “converge.” In broad terms, convergence is achieved when the value of the loss function, and / or the values of the model parameters change in increasingly small amounts with each successive training round. For example, a convergence condition can be achieved if the value of the loss function decreases by less than 0. 1% in two successive training rounds.

[0063] As described in more detail below, some methods according to embodiments use a multi-layer machine learning model comprising a variety of sub-models. Each sub-model can have its own parameter set, and parameters of the machine learning model cancollectively comprise the parameters of all the sub-models. In some embodiments, each submodel can be trained simultaneously based on a combined loss function, i.e., each set of parameters for each sub-model can each be updated in each training round.D. Meta-leaming

[0064] Meta-leaming is a technique that can be used in embodiments of the present disclosure, and a brief background on meta-leaming is provided to orient the reader. In ■‘typical” machine learning, a machine learning model can be trained to perform a particular task. As an example a machine learning system can be trained to determine whether an input image comprises an image of a dog. By contrast, a meta-leamer can be trained to perform a “meta-task.” As an example, a meta-leamer could be trained to determine which machine learning model of a set of machine learning models is expected to achieve the best performance at determining whether an input image is an image of a dog.

[0065] In embodiments of the present disclosure, a meta-task can involve determining which outlier detection model of a plurality7of outlier detection models is best suited to detect time-series outliers in a query7segment of time-series data. As a result, a pool of multiple outlier detection models can be leveraged to achieve better performance than outlier detection systems that rely on single outlier detection models. A meta-leamer according to embodiments can learn to take a query segment of time-series data as an input and output ranking lists, which can indicate the relative efficacy of different outlier detection models at detecting outliers in the query segment, e.g., better ranked models are expected to achieve better performance at detecting outliers in the time-series data. Whichever time-series outlier detection models are deemed best-suited to detect such outliers can then be used to detect those time-series outliers, e.g., by inputting the query segment into that time-series outlier detection model.E. Convolutional Neural Network

[0066] Some embodiments of the present disclosure make use of a novel deep-learning neural network architecture, described in more detail further below with reference to FIGS. 9 and 12A-12C. This architecture can comprise a multilayer sub-model that can itself comprise a first convolutional neural network, a second convolutional neural netyvork, and a thirdconvolutional neural network. As such, a brief summary' of convolutional neural networks and related concepts is provided below.

[0067] Convolution can refer to a mathematical operation on two functions that produces a third function, describing how the shape of one of the functions is modified by the other function. Discrete convolutions can be applied to discrete data such as matrices. For example, an input matrix can be convolved with a “convolution kernel’' or “filter’, which can comprise another matrix that is ty pically smaller than the input matrix. The convolution kernel is typically “scanned” over elements of the input matrix, and the discrete convolution is performed at each of those elements, resulting in a scalar output for each element scanned during the convolution. Collectively these scalar values can comprise the elements of an output matrix, which can be considered the output of the matrix convolution process. In machine learning contexts, a convolutional “layer” can refer to an element of a machine learning system that convolves an input to the layer (i.e., an input matrix) with a convolution kernel, thereby producing an output matrix.

[0068] Such matrix convolutions generally accomplish two things. First, they usually achieve data reduction, as multiple elements of the input matrix are typically combined to produce a single element of the output matrix, and as such, the output matrix is typically smaller than the input matrix (although techniques such as padding can be used such that the output matrix is the same size as the input matrix or potentially even larger). This can be useful for reducing the size of large inputs (e.g., high resolution images), prior to further processing. Second, the convolution kernel itself transforms the input matrix, and can effectively enhance or attenuate data or features contained in the input matrix based on the values of the elements of the convolution kernel.

[0069] As such, convolution can be used to extract features from the input matrix. If the elements of the convolution kernel are tuned to a particular set of features, then sections of the input matrix that are “highly responsive” to the convolution kernel (i.e., produce a high magnitude scalar as a result of discrete convolution with the convolution kernel) likely contain those features. In machine learning, the elements of a convolution kernel can comprise trainable parameters, enabling a machine learning model to improve at extracting relevant features from an input during a training process. The outputs of a convolution layerare sometimes referred to as ‘‘feature matrices'’, “feature maps”, “feature vectors”, “embedding matrices”, “embedding maps”, “embedding vectors”, or other similar terms.

[0070] A typical convolutional neural network involves a series of alternating convolution layers and subsampling (or “pooling”) layers. These pooling layers can down-sample their inputs, effectively compressing them. As a result, successively smaller sets of features are extracted and dow n-sampled. The final layer of a convolutional neural network can comprise a fully-connected neural network, which can processes the final set of embedding vectors or embedding matrices to produce some output, such as a classification. For example, a convolutional neural network could high resolution images as inputs, successively convolve and pool those images in a series of convolution and pooling layers, and then use a linear neural network to produce classifications indicating whether an animal (e.g., a dog) is present in those images.II. EXAMPLE USE AND ARCHITECTURE OF META-LEARNER

[0071] We now' describe methods and systems according to embodiments. Such methods and systems are typically described from the perspective of a computer system that is instantiating, training, and / or using a meta-leamer machine learning model according to embodiments. A brief overview of such method is provided below with reference to FIG. 8. Methods and systems according to embodiments, along with the structure of the meta-leamer and its components are described in more detail below w ith reference to FIGS. 9-16.A. Overview of use of Meta-leamer to Detect Outliers

[0072] Referring to FIG. 8, a meta-leamer machine learning model 812, comprising a multilayer sub-model and a plurality' of head sub-models (described further below' with reference to FIGS. 9, 12A-12C, and 14) can be trained to produce a plurality of outlier segment classifications 814 and optionally produce a plurality of ranking lists 816 based on an unlabeled query segment 804. A ranking list can be used to determine one or more outlier detection models (of, e.g., a plurality' of outlier detection models 818), that can best detect a given outlier type for a given unlabeled query segment 804 of time-series data. In FIG. 8, the exemplary unlabeled query segment 804 contains seasonal outliers 822. As described above, each outlier segment classification of the plurality of outlier segment classifications 814 can correspond to a different outlier type of a plurality' of outlier types (e.g., the five major typesof time-series outliers described above). Each outlier segment classification of the plurality' of outlier segment classifications 814 can indicate whether the unlabeled query segment 804 contains the corresponding type of outlier or can indicate a probability for each outlier type. For example, a seasonal outlier segment classification could indicate the unlabeled query segment likely contains seasonal outliers 822, while a global point outlier segment classification, contextual points outlier segment classification, shapelet outlier segment classification, and trend outlier segment classification could indicate that the unlabeled query segment is not likely to contain global point outliers, contextual point outliers, shapelet outliers, or trend outliers respectively.

[0073] These outlier segment classifications 814 do not necessarily indicate the temporal location of any outliers (e.g., at a particular index or timestamp) and may instead indicate whether or not unlabeled query segment 804 contains such outliers anywhere within that segment. Later, individual outlier detection models 818 can be used to detect individual outliers within the unlabeled query segment 804. For example, if a subset of the plurality of outlier segment classifications indicate that a query segment contains seasonal outliers 822, then a computer system can use one or more seasonal outlier detection models to detect one or more seasonal outliers in the unlabeled query segment 804. In FIG. 8, such a subset could comprise only a seasonal outlier segment classification, as seasonal outliers 822 may be the only outliers present in exemplary' unlabeled query' segment 804.

[0074] Each ranking list of the plurality' of ranking lists 816 can also correspond to a different outlier type of the plurality of outlier type and can indicate the relative efficacy of a plurality of outlier detection models 818 at detecting outliers of the corresponding outlier type. For example, if the plurality of outlier detection models 818 comprise twenty' outlier detection models, then a seasonal ranking list could comprise twenty' rankings or scores corresponding to the twenty outlier detection models. Even if twenty outlier detection models are all configured to specifically detect the same type of outlier, they might not achieve the same performance when detecting those outliers. Further, some outlier detection models may perform better on some time-series and perform worse on other time-series (e.g., due to their hyperparameters or the nature of such time-series, such as different frequency components). As such, the meta-leamer 812 can generate different sets of ranking lists for different unlabeled query segments, such that the best performing outlier detection modelscan be determined for a given unlabeled query segment (e.g., unlabeled query segment 804) and can be used to detect outliers within that query’ segment.

[0075] Continuing the example, the highest ranked outlier detection model from among the plurality of outlier detection models 818, based on the seasonal ranking list, could be used (e.g., by inputting the unlabeled query segment 804 into the selected outlier detection model) to produce outlier detection results 820, which can specifically indicate outliers in unlabeled query segment 804, e.g., as described above with reference to FIG. 2, or in any other suitable manner. In some cases, multiple outlier detection models of the plurality of outlier detection models 818, such as the top three best performing outlier detection models (as indicated by' the corresponding ranking lists) can be used produce outlier detection results 820, and as such there may be multiple outlier detection results 820.

[0076] In more detail, a computer system can perform a subsegmentation process 806 on the unlabeled query segment 804 and a baseline segment (sometimes referred to as a “warmup segment”) 802. This baseline segment 802 can generally comprise a segment of timeseries data that does not contain outliers. This subsegmentation process 806 (described in more detail below with reference to FIG. 11) can result in one or more baseline subsegments 808 and one or more query subsegments. These baseline subsegments 808 and query subsegments 810 can be input into the meta-leamer 812 in order to produce the outlier segment classifications 814 and ranking lists 816, as described above.

[0077] In broad terms, the meta-leamer 812 can leam to extract relevant outlier detection features by comparing the query subsegments 810 (which may contain outliers) and the baseline subsegments 808 (which don’t contain outliers). The meta-leamer 812 can leam to detect different patterns, features, and correlations between the query subsegments 810 and the baseline subsegments 808, and this extracted information can be used to determine if the unlabeled query' segment 804 contains outliers (such as seasonal outliers 822) or not.B. Example Structure ofMeta-learner

[0078] A more detailed structural summary of an example meta-leamer machine learning model according to embodiments is presented below with reference to FIG. 9. The meta- leamer machine learning model can include one or more layers of various ty pes. In FIG. 9,various layers may be optional, and thus FIG. 9 presets just one exemplary7model architecture.

[0079] A meta-leamer according to embodiments can comprise a novel deep learning architecture, comprising several layers, which can be generally divided into a multilayer submodel 902 and a plurality7of head sub-models 920. Such a meta-leamer can be used to generate a plurality of outlier segment classifications (e.g., outlier segment classifications 930-936) and a plurality of ranking lists (e g., ranking lists 938-944) corresponding to an input query segment that has been segmented into one or more query subsegments 948. Such outlier segment classifications and ranking lists can be used to determine one or more best performing outlier detection models, which could then be used to detect outliers of a variety7of outlier types in the query7segment.

[0080] As an example, the machine learning model could be used to detect outliers in a query segment corresponding to a patient’s health data or monitored heart measurements, e.g., a heartbeat. Such a query segment could comprise a series of, e.g., 1000 time-based observations of a patient’s health. Each observation could comprise 100 different health related data values, such as the patient’s heartrate, blood pressure, blood oxygen level, etc. Outliers in this query segment could be indicative of diseases (e.g., hypothyroidism). A baseline segment (also referred to as a warm-up segment) could correspond to normal health data, e.g., health data that does not contain outliers, such as a normal set of heartrate observations, a normal set of blood pressure observations, etc. For the purpose of example, this baseline segment could also comprise 1000 normal time-based observations of a patient’s health, although there is no requirement that the query segment and baseline segment must be the same length.

[0081] As described above with reference to FIG. 8, the query7segment can be segmented into one or more query subsegments 948. For the purpose of example, these query subsegments 948 could comprise 20 subsegments of the query segment, each containing 50 observations of the patient’s health, each observation comprising 100 data values. Likewise, the baseline segment can be segmented into one or more baseline subsegments 946. For the purpose of example, these baseline subsegments 946 could comprise 20 subsegments of the baseline segment, each containing 50 non-outlier observations of a patient’s health, each nonoutlier observation comprising 100 non-outlier data values. In this disclosure, a variable suchas w can define the “width” of a matrix, such as a subsegment, e.g., the number of observations in that query’ subsegment or baseline subsegment (50 in this example), and a variable such as d can refer to the dimension of each observation, e.g., the number of data values in each observation (100 in this example). As such, each subsegment can be represented as a w by d (50 by 100) matrix.

[0082] These baseline subsegments 946 and query subsegments 948 can be input into the first layer of the multilayer sub-model 902, i.e., the first convolutional neural network 904. First convolutional neural network 904 can comprise an alternating series of convolution layers and pooling layers, followed by a neural network (e.g., a fully-connected neural network), as described above. A convolution kernel can be convolved with each of the baseline subsegments 946 and each of the query subsegments 948. In some embodiments, a single shared convolution kernel can be used for both the baseline subsegments 946 and the query subsegments 948. As an alternative however, one convolution kernel can be applied to the baseline subsegments 946 and another convolution kernel can be applied to the query7subsegments 948. The convolution kernel (along with other parameters of the first convolutional neural network) can comprise trainable parameters that can be trained in order to improve the machine learning model’s ability to extract useful features from the baseline subsegments 946 and query' subsegments 948.

[0083] The outputs of the first convolutional neural network 904 can comprise one or more baseline embedding matrices and one or more query embedding matrices (not pictured). These matrices can effectively comprise features extracted from the baseline subsegments 946 and query subsegments 948 and can thereby comprise new embeddings of the baseline subsegments 946 and query' subsegments 948. While the baseline subsegments 946 and query’ subsegments 948 can represent the time-series data in a time-space, the baseline embedding matrices and query embedding matrices can represent features derived from the time-series data in an embedding-space.

[0084] The size or dimensions of the baseline embedding matrices and the query embedding matrices can be different from the baseline subsegments 946 and query subsegments 948 respectively. For example, if each baseline subsegment and query' subsegment comprises a 50 by 100 matrix, the baseline embedding matrices and query embedding matrices could comprise 50 by 20 matrices, 100 by 200 matrices, etc. As such, inaddition to extracting features, the first convolutional neural network 904 can effectively resize the baseline subsegments 946 and query subsegments 948.

[0085] In some embodiments, the baseline embedding matrices and query’ embedding matrices may have equivalent width as the baseline subsegments 946 and query' subsegments 948 but a lesser dimension or depth. Alternatively, the dimensions of the baseline embedding matrices and query embedding matrices can be controlled by hyperparameters, which can be set at the discretion of practitioners of methods according to embodiments. As such, it is not necessary that the size of the baseline embedding matrices and query embedding matrices maintains any specific relationship with the size of the baseline subsegments 946 and query' subsegments 948.

[0086] In the first pooling operations 906, described in more detail below with reference to FIG. 12A, a computer system instantiating and running the machine learning model can pool the baseline embedding matrices to determine a pooled baseline embedding matrix. The machine learning model can then determine one or more first embedding matrices based on the one or more query' embedding matrices and the pooled baseline embedding matrix. In some embodiments, the computer system can determine one or more embedding difference matrices by calculating one or more matrix differences between the one or more query embedding matrices and the pooled baseline embedding matrix. In some embodiments, each embedding difference matrix of the one or more embedding difference matrices can comprise a difference betw een a different query embedding matrix of the one or more query’ embedding matrices and the pooled baseline embedding matrices. The one or more first embedding matrices can comprise the one or more embedding difference matrices.

[0087] In general terms, the one or more first embedding matrices can encode the difference between features characterizing one or more baseline subsegments 946 (which may not contain outliers), and the one or more query subsegments 948 (which may contain outliers). Intuitively, if the one or more query subsegments 948 do not contain outliers, then the difference between the pooled embedding matrix and the query embedding matrices is expected to be small, whereas if the one or more query' subsegments 948 contain many outliers, the difference between the pooled embedding matrix and the query’ embedding matrix may be larger. As such, the one or more first embedding matrices may correlate with the presence of outliers in the query segment.

[0088] The one or more first embedding matrices can be input into a second convolutional neural network 908. Similar to the first convolutional neural network 904, the second convolutional neural network 908 (and the third convolutional neural network 914) can comprise an alternating series of convolution layers and pooling layers, followed by a neural network, which may be a fully-connected neural network. The output of the second convolutional neural network 908 can comprise one or more second embedding matrices, which may have different dimensions than the one or more first embedding matrices.

[0089] The second embedding matrices can be applied to (e.g., input into) a first attention layer 910. Attention layers, such as the first attention layer 910 and second attention layer 916 are described in more detail below with reference to FIGS. 13A-13C. In broad terms, using a novel self-attention method, the first attention layer 910 can produce one or more first attention matrices. The one or more first attention matrices can be pooled using second pooling operations to produce one or more third embedding matrices, and the one or more third embedding matrices can be applied to a third convolutional neural network 914. The outputs of the third convolutional neural network 914 can comprise one or more fourth embedding matrices, which can be applied to a second attention layer 916 to produce one or more second attention matrices. The one or more second attention matrices can be pooled (e.g., in third pooling operations 918) to produce a pooled second attention matrix, which may also be referred to as an embedding matrix (or referred to as a fifth embedding matrix). This embedding matrix can comprise the output of the multilayer sub-model 902.

[0090] The embedding matrix (fifth embedding matrix) can be applied to each head submodel of the plurality of head sub-models 920. In FIG. 9, a first through N1'1head sub-models 922-928 are shown. In some embodiments, each head sub-model of the plurality of head submodels can correspond to a different outlier type of a plurality of outlier types. For example, there could be five head sub-models corresponding to the five major types of outliers described above, e.g., a global points head sub-model, a contextual points head sub-model, a shapelet head sub-model, a seasonal head sub-model, and a trend head sub-model. In some embodiments, each head sub-model of the plurality of head sub-models can be used to convolve and pool the fifth embedding matrix, thereby producing a feature vector. Each head sub-model can use this feature vector to produce an outlier segment classification of a plurality of outlier segment classifications (e.g., outlier segment classifications 930-936), andcan optionally produce a ranking list of a plurality of ranking lists (e.g., ranking lists 938- 944). The plurality of outlier segment classifications and the plurality of ranking lists can compnse the outputs of the meta-leamer machine learning model.

[0091] As described above, each outlier segment classification can indicate whether a query segment (corresponding to query subsegments 948) as a whole contains one or more outliers of the corresponding outlier type, rather than identifying specific outliers (e.g., specific global point outliers) within the corresponding query segment. In some embodiments, the outlier segment classifications may be binary, e.g., they may compnse true or false values indicating whether the query segment contains an outlier of a given outlier ty pe or does not contain an outlier of the given outlier type. Alternatively, in other embodiments the outlier segment classifications may comprise probability' values, indicating a probability (e.g., 75%) that the query segment contains an outlier of a given outlier type. Likewise, as described above, each ranking list can indicate which outlier detection models are predicted to achieve the best performance at detecting outliers of a corresponding outlier type within the query segment. For example, a ranking list corresponding to shapelet outliers could indicate the relative efficacy of, e.g.. 20 different outlier detection models at detecting shapelet outliers within the query segment. These outlier segment classifications 930-936 and ranking lists 938-944 can be used to select outlier detection models, which can be used to detect outliers of various outlier ty pes in the query' segment, e.g., by selecting the best performing outlier detection model for each outlier type identified by the plurality of outlier segment classifications.C. Production Method

[0092] Having summarized the structure of the meta-leamer machine learning model above with reference to FIG. 9, it may be helpful to describe production methods according to embodiments, along with a more detailed description of the meta-leamer structure, including both the structure of the first and second attention layers and the structure of the head submodels. In general, such production methods can comprise using the trained meta-leamer to generate a plurality of outlier segment classifications and optionally a plurality of ranking lists, as described above. Such description is provided below with reference to the flowchart of FIGS. 10A-10C (depicting methods according to embodiments), the detailed meta-leamer structural diagrams of FIGS. 12A-12C, FIGS. 13A-13C (depicting the structure of the firstattention layer and the second attention layer), FIG. 14 (depicting the structure of the head sub-models), and FIG. 11 (depicting subsegmentation methods). As training methods can generally be understood in the context of production methods, such training methods are described below following the description of the production methods.

[0093] Referring to FIG. 10A, at step 1002 a computer system can receive input time-series data. The input time-series data can include a baseline segment (sometimes referred to as a “warm-up” segment) and a query segment. In some embodiments, the query segment can have a predefined or fixed length, e.g.. a fixed number of time-series observations or data vectors. The query segment can contain one or more outliers of one or more outlier types (e.g., the query segment can contain global point outliers, shapelet outliers, and trend outliers). In some embodiments, the baseline segment can have an arbitrary' length. In some embodiments, the baseline segment can comprise a time-series data segment that does not contain outliers. A computer system or a user of the computer system (e.g., a data scientist) can identify and label a time-series data segment that does not contain outliers and use it as a baseline segment. In some embodiments, the computer system can receive the input timeseries data by retrieving the input time-series data from a database, a data stream, a local memory element (such as a hard drive), cloud storage, an I / O interface, etc.

[0094] As an example, a computer system could receive a query segment corresponding to patient health data for a 15-minute treadmill exercise stress test. Each observation could comprise multiple data values, such as the patient's blood pressure, heart rate, oxygen intake rate, and electrocardiogram signals. The query segment could contain outliers of a variety of outlier types (such as global point outliers corresponding to spiking heartrate, trend oxygen intake outliers corresponding to hyperventilation, etc.), which may be useful for diagnosing the patient. The baseline segment could comprise, e.g., 27 minutes of health data corresponding to a healthy athlete performing the same stress test. An expert in sports medicine could determine that the baseline segment does not contain any outliers or anomalous data values. The computer system could receive this data from e g., a medical database or from the systems or machines used to capture such data during the stress test.

[0095] At step 1004 the computer system can perform any initial processing operations on the input time-series data, which may enable further processing or machine learning based methods according to embodiments. Because there are a variety of potential initialprocessing operations that could be performed, and because such initial processing operations can be performed at the discretion of a practitioner of embodiments, examples of initial processing operations rather than an exhaustive list of different initial processing operations are provided herein. Such initial processing could comprise, for example, encoding categorical data as numerical data, in order to facilitate further processing of such data, e.g., via discrete convolutions performed by the first convolutional neural network. As an example, categorical time-series data such as “excellent”, “good”, “fair”, “poor”, corresponding to a health care provider’s subjective analysis of a patient’s health could be converted to numerical values such as 4, 3, 2, 1. Other initial processing operations could include padding the query segment (such that it meets any fixed length requirements), signal normalization, temporal re-sampling (e.g., down-sampling), etc.

[0096] At step 1006 the computer system can segment the baseline segment into one or more baseline subsegments. Likewise, the computer system can segment the query segment into one or more query subsegments. In some embodiments, the one or more baseline subsegments can comprise an arbitrary (e.g., not predetermined) number of baseline subsegments derived from a “clean” baseline segment of time-series data (i.e., a segment of time-series data that does not contain outliers). In some embodiments, the one more or more query subsegments can comprised a fixed or predetermined number of query subsegments.

[0097] FIG. 11 shows a segmentation process performed on a baseline segment 1102 (which may be of arbitrary length) and on a query segment 1116 (which may be of fixed length). In FIG. 11, the baseline segment 1102 is segmented into an arbitrary number of baseline subsegments 1 104 with fixed length (i.e., baseline subsegments 1106-11 14), and the query segment 1116 is segmented into a fixed number of query subsegments 1118 with fixed length (i.e., query' subsegments 1122-1130).

[0098] In some embodiments, segmenting the baseline segment into one or more baseline subsegments can involve performing a period estimation process on the baseline segment, thereby determining an estimated baseline period. Likewise, segmenting the query segment into one or more query subsegments can comprise performing a period estimation process on the query' segment, thereby determining an estimated query' period. Such period estimation processes can involve using techniques such as the Discrete Fourier Transform to determine frequency components of the baseline segment and query segment, and the period can bedetermined based on those determined frequency components. These period estimation processes can be used to determine the fixed length of the “rolling” or “sliding” windows used to define the length of the baseline subsegments and query subsegments.

[0099] Further, segmenting the baseline segment into one or more baseline subsegments can involve performing a rolling segmentation process on the baseline segment (e.g., based on the estimated baseline period), thereby determining one or more unsealed baseline subsegments. Likewise, segmenting the query segment into one or more query subsegments can involve performing a rolling segmentation process on the query segment based on the estimated query period, thereby determining one or more unsealed query subsegments. Such rolling segmentation processes can generally involve sampling time-series data from the baseline segment and query' segment in “rolling” or “sliding” windows, resulting in e g., an arbitrary number of baseline subsegments and a fixed number of query subsegments. In FIG. 11, baseline subsegments 1106-1114 are depicted as non-overlapping subsegments of the baseline segment 1102, and likewise the query- subsegments 1122-1130 are depicted as non-overlapping subsegments of the query segment 1116. However, the query subsegments and baseline subsegments can be overlapping or non-overlapping, depending on the “stride” or other parameters used to define the rolling segmentation process.

[0100] In some embodiments, segmenting the baseline segment into one or more baseline subsegments can further involve scaling the one or more unsealed baseline subsegments to a fixed length, thereby producing the one or more baseline subsegments. Likewise, segmenting the query segment into one or more query subsegments can further involve scaling the one or more unsealed query' subsegments to a fixed length, thereby producing the one or more query subsegments. This scaling can comprise a form of normalization that can enable further processing of the baseline subsegments and query' subsegments. As depicted in FIG. 12A, these baseline subsegments 1202 and query subsegments 1204 can comprise the input to the first convolutional neural network 1206 of the multilayer sub-model of the meta- leamer machine learning model. Establishing a fixed length for the baseline subsegments and query' subsegments (e.g., via scaling) may facilitate analysis or comparison by the meta- leamer machine learning model, and may enable the use of a shared convolution kernel for both the baseline subsegments and the query subsegments.

[0101] Referring to FIGS. 10A and 12 A, at step 1008 the computer system can apply the first convolutional neural network 1206 to the one or more baseline subsegments 1202 and the one or more query subsegments 1204. As described above with reference to FIG. 9, the first convolutional neural network 1206 can comprise an alternating series of convolution layers and pooling layers, followed by a neural network (e.g., a fully-connected neural network). In some embodiments a shared convolution kernel can be used for both the baseline subsegments 1202 and query subsegments 1204. The computer system can thereby produce one or more baseline embedding matrices 1208 (corresponding to the one or more baseline subsegments 1202), and one or more query embedding matrices 1210 (corresponding to the one or more query subsegments 1204), which can comprise the outputs of the first convolutional neural network 1206. These baseline embedding matrices 1208 and query embedding matrices 1210 can represent the baseline subsegments 1202 and query subsegments 1204 in an embedding space, thereby extracting features from the baseline subsegments 1202 and query' subsegments 1204. Convolutional neural networks, such as the first convolutional neural network 1206 can reduce the embedding size, which can in turn improve the performance of components of the meta-1 earner machine learning model such as the first attention layer 1222, thereby improving the feature extraction process.

[0102] As described above, the dimensions of the baseline embedding matrices 1208 and query' embedding matrices 1210 may be different from the dimensions of the baseline subsegments 1202 and query subsegments 1204 respectively. One or both of the dimensions of the baseline embedding matrices 1208 and query embedding matrices 1210 may be larger or smaller (or the same size) as the dimensions of the baseline subsegments 1202 and query' subsegments 1204. In some embodiments, it may be preferable to maintain a constant width w (e.g., the number of time-series observations in each subsegment), while reducing the dimension d (e.g., the number of data values associated with each observation) of the embedding matrices, as it may reduce the computational intensity of further processing and may therefore improve the speed and efficiency of methods according to embodiments.

[0103] Afterwards, at steps 1010-1012 the computer system can generate one or more first embedding matrices 1216 based on the one or more baseline embedding matrices 1208 and the one or more query embedding matrices 1210. At step 1010, the computer system can pool the one baseline embedding matrices 1208 (e.g., in first pooling operations 1212) therebyproducing a pooled baseline embedding matrix 1214. Pooling techniques such as max pooling or average pooling can be used for this purpose. The pooled baseline embedding matrix 1214 can represent a measure of tendency for the baseline embedding matnces 1208, such as the a “typical” or “average” set of features derived from the baseline subsegments 1202.

[0104] At step 1012 the computer system can determine one or more embedding difference matrices based on the one or more query embedding matrices 1210 and the pooled baseline embedding matrix 1214. The one or more first embedding matrices 1216 can comprise these one or more embedding difference matrices. Each embedding difference matrix of the one or more embedding difference matrices can comprise a difference between a different query' embedding matrix of the one or more query embedding matrices 1210 and the pooled baseline embedding matrix 1214. The computer system can determine these embedding difference matrices using elementwise matrix subtraction. The resulting one or more first embedding matrices 1216 can comprise features that encode or correlate to differences between each query' embedding matrix and the pooled baseline embedding matrix 1214.

[0105] At step 1014 the computer system can apply the second convolutional neural network 1218 to the one or more first embedding matrices 1216, thereby producing the one or more second embedding matrices 1220. The second convolutional neural network 1218 can comprise an alternating series of convolution layers (with respective convolution kernels) and subsampling (pooling) layers, followed by a neural network, which may comprise a fully- connected neural network. As described above with respect to the baseline subsegments 1202, query subsegments 1204, baseline embedding matrices 1208, and query embedding matrices 1210, the inputs and outputs of the second convolutional neural network 1218 may not have the same dimensions, and therefore the one or more second embedding matrices 1220 may not have the same dimensions as the one or more first embedding matrices 1216.1. First Attention Layer

[0106] At step 1016 the computer system can apply the first attention layer 1222 to the one or more second embedding matrices 1220, thereby producing one or more first attention matrices 1224. Steps 1018-1034 of FIGS. 10A and 10B generally comprise steps performed by the computer system (using the first attention layer 1222) in order to produce the one ormore first attention matrices 1224. The structure of the first attention layer 1222 (and byextension the second attention layer 1236 of FIG. 12B) and steps 1018-1034 are described in more detail with reference to FIGS. 13A-13C. In broad terms, the first attention layer 1222 can comprise a novel variation of self-attention ‘'blocks” used in machine learning transformer models.

[0107] At step 1018 the computer system can combine the one or more second embedding matrices 1302 (e.g., second embedding matrices 1304-1310 as depicted in FIG. 13A), thereby- producing a combined second embedding matrix 1312. In FIG. 13 A, nqcan refer to the number of second embedding matrices 1302, w can refer to one dimension of each second embedding matrix of the one or more second embedding matrices 1302 (e.g., the “width,” or number of observations), and d can refer to another dimension of each second embedding matrix of the one or more second embedding matrices 1302 (e.g., the “depth” or the number of data values associated with each observation). Notably, w and d do not have to equal to the width or depth of, e.g., the one or more baseline subsegments, the one or more query subsegments, the one or more first embedding matrices, etc. The one or more second embedding matrices 1302 can be combined using techniques such as concatenation, or byotherwise appending the one or more second embedding matrices 1302 to one another. In some embodiments, the resulting combined second embedding matrix 1312 may have dimensions nq* w by d, however other dimensions are possible (e.g., w by nq* d).

[0108] At step 1020 the computer system can transpose the combined second embedding matrix 1312, thereby producing a combined transposed second embedding matrix 1314. The computer system can use appropriate transposition method to produce the combined transposed second embedding matrix 1314, e.g., using in-place matrix transposition methods, or e.g., by initializing an empty matrix with “swapped” dimensions d by nq* w, then iterating through the combined second embedding matrix 1312 and copying the value of the elements stored therein to the correct location of the empty matrix until the combined transposed second embedding matrix 1314 is produced.

[0109] Some embodiments of the present disclosure make use of a novel self-attention (e.g., correlation) mechanism, which enables feature correlation information to be extracted and encoded, and thereby enhances the meta-leamer machine learning model’s ability to identify the presence of outliers in the query segment (and optionally generate ranking lists).Self-attention is an effective technique for extracting correlational information, which can be useful in identifying anomalous or outlier properties of extracted features (represented by the one or more second embedding matrices 1302). In conventional self-attention processes, matrices are usually multiplied by a value matrix (which can comprise a parameter matrix used to weigh different features).

[0110] By contrast, in embodiments, the combined inputs to the first attention layer can be multiplied by their transpose as a form of autocorrelation. In more detail (and with reference to FIGS. 10B and 13A), at step 1022 the computer system can perform a matrix multiplication on the combined second embedding matrix 1312 and the combined transposed second embedding matrix 1314, thereby producing a first embedding correlation matrix 1316. In FIG. 13 A, the first embedding correlation matrix 1316 can comprise an ng* w by ng* w square matrix, and each element can comprise a correlation value, where the i, 7thelement comprises a correlation value representing the correlation between d observations corresponding to the ithrow of the combined second embedding matrix 1312 and d observations corresponding to the jthrow of the combined second embedding matrix 1312.

[0111] Referring to FIGS. 10B and 13B. the computer system can perform a series of machine learning operations in order to extract useful features from the first embedding correlation matrix 1316. At step 1024 the computer system can perform a pooling operation on the first embedding correlation matrix 1316 (i.e., attention pooling 1320), thereby producing a first pooled embedding correlation matrix 1322. The computer system can accomplish this using any appropriate pooling operation, e.g., average pooling, max pooling, etc.

[0112] At step 1026 the computer system can apply the first attention multilayer perceptron 1324 (e.g., a neural network comprising three or more layers) to the first pooled embedding correlation matrix 1322, thereby producing a first non-normalized correlation matrix 1326.

[0113] At step 1028 the computer system can normalize the first non-normalized correlation matrix 1326 using linear normalization 1328, thereby producing a first normalized correlation matrix 1330. Any appropriate form of linear normalization can be used for this purpose, such as “max-min” normalization, e.g., wherein a normalized value of an element is proportional to the difference between that value and the minimum value in the first nonnormalized correlation matrix 1326 divided by the range, e.g.. such that the minimum valuein the first non-normalized correlation matrix 1326 is normalized to zero and the maximum value in the first non-normalized correlation matrix 1326 is normalized to one, and such that any values between the minimum and maximum value are normalized to between zero and one.

[0114] In the example of FIG. 13B, the resulting first normalized correlation matrix 1330 can comprise an nq* w by d ' matrix. The d ' values in each row of the first normalized correlation matrix 1330 can effectively encode how correlated each row of the combined second embedding matrix 1312 is with each other row of the combined second embedding matrix 1312. In some embodiments, d’ may be less than nq* w, and therefore the operations depicted in FIG. 13B may encode or “compress” the information represented by nq* w row elements of the first embedding correlation matrix 1316 into d’ elements. However, d' may also be greater than nq* w.

[0115] Referring to FIGS. 10B and 13C. at step 1030 the computer system can concatenate the combined second embedding matrix 1312 with the first normalized correlation matrix 1330, thereby generating an enhanced second embedding matrix 1332. As depicted in FIG. 13C, the computer system can, e.g., append the d’ elements in each of the nq* w rows of the first normalized correlation matrix 1330 to the corresponding row of the combined second embedding matrix 1312, thereby producing a nq* w by d + d ’ enhanced second embedding matrix 1332. The enhanced second embedding matrix 1332 can contain both the features represented by the combined second embedding matrix 1312 and the correlations between those features represented by the first normalized correlation matrix 1330, information that may be useful for subsequent machine learning processing.

[0116] At step 1032, the computer system can apply a first attention linear neural network layer 1334 (e.g., a fully-connected neural network layer) to the enhanced second embedding matrix 1332, thereby generating a combined first attention matrix 1336. The first attention linear neural network layer 1334 may benefit from the additional correlation information encoded by the enhanced second embedding matrix 1332, and may therefore produce an output (i.e., the combined first attention matrix 1336) that is more useful for detecting different types of outliers in the input query segment. In some embodiments the combined first attention matrix can have dimensions nq* w by d, i.e., the same dimensions as the combined second embedding matrix 1312. however it may also have different dimensions.

[0117] At step 1034, the computer system can segment the combined first attention matrix 1336, thereby producing one or more first attention matrices 1338 (e.g.. first attention matrices 1340-1346 as depicted in FIG. 13C). This segmentation can enable further processing by other layers of the multilayer sub-model of the meta-leamer machine learning model, e.g., the third convolutional neural network 1232 as depicted in FIG. 12B. In some embodiments, each first attention matrix can have dimensions w by d, e.g., the same dimensions as the one or more second embedding matrices 1302 from FIG. 13 A, however they may also have different dimensions. Referring now to FIGS. 10B and 12B, the one or more first attention matrices 1224 (as indicated in FIG. 12B) can effectively comprise the output of the first attention layer 1222.

[0118] At step 1036, the computer system can generate one or more third embedding matrices 1230 based on the one or more first attention matrices 1224. Step 1036 can comprise a number of sub-steps. For example, the computer system can pool the one or more first attention matrices 1224 (e.g., in second pooling operations 1226), thereby producing a pooled first attention matrix 1228. The computer system can use any appropriate pooling operations (e.g., average pooling or max pooling, as described above) for this purpose. Pooling in this manner can effectively extract information collected or ensembled from the one or more first attention matrices 1224.

[0119] Afterwards, the computer system can determine one or more sum attention matrices. The one or more third embedding matrices 1230 can comprise these one or more sum attention matrices. Each sum attention matrix of the one or more sum attention matrices can comprise a matrix sum of a different first attention matrix (of the one or more first attention matrices 1224) and the pooled first attention matrix 1228. This process can enhancing the information contained in the one or more third embedding matrices 1230.

[0120] At step 1038, the computer system can apply the third convolutional neural network 1232 to the one or more third embedding matrices 1230, thereby producing one or more fourth embedding matrices 1234. The third convolutional neural network 1232 can comprise an alternating series of convolution layers (with respective convolution kernels) and subsampling (pooling) layers, followed by a neural network (which may comprise a fully- connected neural network. The dimensions of the matrix outputs of the third convolutional neural network 1232 (i.e., the one or more fourth embedding matrices 1234) may be differentthan the dimensions of the matrix inputs (i.e., the one or more third embedding matrices 1230).2. Second Attention Layer

[0121] Referring to FIGS. 10C and 12B, at step 1040, the computer system can apply the second attention layer 1236 to the one or more fourth embedding matrices 1234, thereby- producing one or more second attention matrices 1238. The operations performed by the computer system using the second attention layer 1236 may be similar to the operations performed by7the computer system using the first attention layer 1222, e.g., as described above with reference to step 1016 of FIG. 10 A. Likewise, the structure of the second attention layer 1236 may be similar to the structure of the first attention layer 1222. Steps 1042-1058 of FIGS. 10C and 10D generally comprise steps performed by the computer system (using the second attention layer 1236) in order to produce the one or more second attention matrices 1238. These steps are described in more detail with reference to FIGS. 13A-13C. The second attention layer 1236 (like the first attention layer 1222) can comprise a novel variation of self-attention "blocks'’ used in machine learning transformer models.

[0122] At step 1042. the computer system can combine the one or more fourth embedding matrices 1302 (e.g., fourth embedding matrices 1304-1310 as depicted in FIG. 13A), thereby producing a combined fourth embedding matrix 1312. In FIG. 13 A, nqcan refer to the number of fourth embedding matrices 1302, w can refer to one dimension of each fourth embedding matrix of the one or more fourth embedding matrices 1302 (e.g., the "w idth." or number of observations), and d can refer to another dimension of each fourth embedding matrix of the one or more fourth embedding matrices 1302 (e.g., the “depth” or “dimension”, e.g., the number of data values associated with each observation). Notably, w and d do not have to equal to the width or depth of, e.g., the one or more baseline subsegments, the one or more query subsegments, the one or more first embedding matrices, etc. The one or more fourth embedding matrices 1302 can be combined using techniques such as concatenation, or by otherwise appending the one or more fourth embedding matrices 1302 to one another. In some embodiments, the resulting combined fourth embedding matrix 1312 may have dimensions nq* w by d, however other dimensions are possible (e.g., w by nq* d).

[0123] At step 1044, the computer system can transpose the combined fourth embedding matrix 1312, thereby producing a combined transposed fourth embedding matrix 1314. The computer system can use appropriate transposition method to produce the combined transposed fourth embedding matrix 1314, e.g., using in-place matrix transposition or by initializing an empty matrix wi th “swapped” dimensions d by nq* w, then iterating through the combined fourth embedding matrix 1312 and copying the value of the elements stored therein to the correct location of the empty matrix until the combined transposed fourth embedding matrix 1314 is produced.

[0124] At step 1046, the computer system can perform a matrix multiplication on the combined fourth embedding matrix 1312 and the combined transposed fourth embedding matrix 1314, thereby producing a second embedding correlation matrix 1316. In FIG. 13 A, the second embedding correlation matrix 1316 can comprise an n,, * w by nq* w square matrix, and each element can comprise a correlation value, where the i,thelement comprises a correlation value representing the correlation between d observations corresponding to the / row of the combined fourth embedding matrix 1312 and d observations corresponding to the7' row of the combined fourth embedding matrix 1312.

[0125] Referring to FIGS. 10C and 13B. the computer system can perform a series of machine learning operations in order to extract useful features from the second embedding correlation matrix 1316. As these machine learning operations are similar to those described above with reference to FIGS. 10B and 13B, they are not described at the same level of detail and may be better understood with reference to the description above. At step 1048, the computer system can perform a pooling operation on the second embedding correlation matrix 1316 (i.e., attention pooling 1320), thereby producing a second pooled embedding correlation matrix 1322. The computer system can accomplish this using any appropriate pooling operation, e.g., average pooling, max pooling, etc.

[0126] At step 1050, the computer system can apply the second attention multilayer perceptron 1324 (e.g., a neural network comprising three or more layers) to the second pooled embedding correlation matrix 1322, thereby producing a second non-normalized correlation matrix 1326.

[0127] At step 1052, the computer system can normalize the second non-normalized correlation matrix 1326 using linear normalization 1328, thereby producing a secondnormalized correlation matrix 1330. Any appropriate form of linear normalization can be used for this purpose, such as ' inax-min" normalization, as described above.

[0128] In the example of FIG. 13B, the resulting second normalized correlation matrix 1330 can comprise an nq* w by d’ matrix. The d’ values in each row of the second normalized correlation matrix 1330 can effectively encode how correlated each row of the combined fourth embedding matrix 1312 is with each other row of the combined fourth embedding matrix 1312. In some embodiments, d' may be less than nq* w, and therefore the operations depicted in FIG. 13B may encode or “compress” the information represented by nq* w row elements of the second embedding correlation matrix 1316 into d’ elements.However, d’ may also be greater than nq* w.

[0129] Referring to FIGS. 10C and 13C, at step 1054, the computer system can concatenate the combined fourth embedding matrix 1312 with the second normalized correlation matrix 1330. thereby generating an enhanced fourth embedding matrix 1332. As depicted in FIG. 13C, the computer system can, e.g., append the d’ elements in each of the nq* w rows of the second normalized correlation matrix 1330 to the corresponding row of the combined fourth embedding matrix 1312, thereby producing a nq* w by d + d' enhanced fourth embedding matrix 1332. The enhanced fourth embedding matrix 1332 can contain both the features represented by the combined fourth embedding matrix 1312 and the correlations between those features represented by the second normalized correlation matrix 1330, information that may be useful for subsequent machine learning processing.

[0130] At step 1056, the computer system can apply the second attention linear neural network layer 1334 (e.g.. a fully-connected neural network layer) to the enhanced fourth embedding matrix 1332, thereby generating a combined second attention matrix 1336. The second attention linear neural network layer 1334 may benefit from the additional correlation information encoded by the enhanced fourth embedding matrix 1332, and may therefore produce an output (i.e.. the combined second attention matrix 1336) that is more useful for detecting types of outliers in the input query segment, e.g., by producing one or more outlier segment classifications and optionally one or more ranking lists.

[0131] Referring to FIGS. 10D and 13C, at step 1058, the computer system can segment the combined second attention matrix 1336, thereby producing one or more second attention matrices 1338 (e.g., second attention matrices 1340-1346 as depicted in FIG. 13C). Referringnow to FIGS. 10D and 12B, the one or more second attention matrices 1238 (as indicated in FIG. 12B) can effectively comprise the output of the second attention layer 1236. These one or more second attention matrices 1238 can be used to generate the fifth embedding matrix 1242 (also referred to as just '‘the embedding matrix”), which can be input into a plurality of head sub-models in order to produce a plurality of outlier segmentation classifications (and optionally a plurality of ranking lists), as described in further detail below.

[0132] Referring to FIGS. 10D and 12B, at step 1060, the computer system can pool the one or more second attention matrices 1238 (e.g., in third pooling operations 1240), thereby producing a pooled second attention matrix. The fifth embedding matrix 1242 (or just “embedding matrix”) can comprise the pooled second attention matrix. The computer system can accomplish this using any appropriate pooling operation, e.g., average pooling, max pooling, etc. The fifth embedding matrix 1242 can comprise the output of the multilayer submodel of the meta-leamer machine learning model, and can comprise the input to a plurality of head sub-models 1278 depicted in FIG. 12C.3. Head Sub-Models

[0133] Referring to FIGS. 10D and 12C, a plurality of head sub-models 1278 can be used by the computer system to produce a plurality of feature vectors, which can subsequently be used to produce a plurality of outlier segment classifications (e.g., outlier segment classifications 1252-1258) and optionally a plurality of ranking vectors (e.g., ranking vectors 1260-1266). As such, at step 1062, the computer system can apply each head sub-model of the plurality of head sub-models 1278 to the fifth embedding matrix 1242 (also referred to as the “embedding matrix), thereby producing the plurality of outlier segments classifications and (optionally), a plurality of ranking vectors. These ranking vectors can then be matrix multiplied by an outlier detection matrix 1268 in order to produce a plurality of ranking lists (e.g., ranking lists 1270-1276).

[0134] In some embodiments, each head sub-model of the plurality of head sub-models 1278 can correspond to a different outlier type of a plurality of outlier types. For example, if the meta-leamer machine learning model is configured to generate outlier segment classifications corresponding to the five major time-series outlier types described above (i.e.. global point outliers, contextual point outliers, shapelet outliers, seasonal outliers, and trendoutliers), there may be five head sub-models in the plurality of head sub-models 1278, which may be used to produce five feature vectors. These five feature vectors could be used to produce five outlier segment classifications (each corresponding one of the five major outlier types) and optionally five ranking vectors, which can be used to produce five ranking lists. In this way, an outlier segment classification and a ranking list can be produced for each outlier type of the plurality of outlier types.

[0135] As described above, each outlier segment classification of the plurality of outlier segment classifications can indicate whether the input query segment contains one or more outliers of a corresponding outlier type. For example, a shapelet outlier segment classification can indicate whether the query' segment contains one or more shapelet outliers. However, such outlier segment classifications may not specifically identify which observations in the query segment comprise outliers. The outlier segment classifications can indicate whether the query segment contains one or more outliers of a corresponding outlier type in a variety of ways. For example, the plurality' of outlier segment classifications can comprise a plurality of binary' values (i.e., “true’' or “false” values), indicating whether the query segment contains an outlier of the corresponding outlier type (true) or does not contain an outlier of the corresponding outlier type (false). As an alternative, the plurality of outlier segment classifications can comprise a plurality of probabilities indicating the likelihood that the query' segment contains one or more outliers of the corresponding outlier ty pes. For example, a “trend” outlier segment classification of “75%” can indicate that there is a 75% probability that the query segment contains one or more trend outliers. If a query segment includes multiple types of outliers, multiple outlier segment classifications may' comprise “true” values or high probabilities.

[0136] The plurality of outlier segment classifications can comprise the outputs of the plurality of head sub-models 1278 (and the meta-leamer machine learning model as a whole). In some embodiments, the outputs of the plurality of head sub-models 1278 can further comprise the plurality of ranking vectors, which can be used by computer system to generate the plurality of ranking lists, and the plurality of ranking lists can comprise additional outputs of the meta-leamer machine learning model. In some embodiments, the outputs of the plurality of head sub-models 1278 can further comprise the plurality of feature vectors, which can comprise additional outputs of the meta-leamer machine learning model. This pluralityof feature vectors may comprise useful outputs if, for example, one or more outlier detection models use such feature vectors as inputs in order to detect time-series outliers in the input query segment.

[0137] Steps 1064-1070 of FIG. 10D generally comprise steps performed by the computer system (using the plurality of head sub-models 1278) in order to produce the plurality of outlier segment classifications, the plurality of ranking lists, the plurality of feature vectors, etc., as summarized above. The structure of the plurality of head sub-models 1278 and steps 1064-1070 are described in more detail with reference to FIGS. 10D and 14. In general, these steps are described from the perspective of a single head model of the plurality of head sub-models. As such, the description below may describe data or elements in the singular, such as “a feature vector’, “a ranking vector”, “an outlier segment classification,” etc. However, it should be understood that the steps described below can be performed for each head sub-model of the plurality of head sub-models, and therefore can be used to produce a plurality of feature vectors, ranking vectors, outlier segment classifications, etc., even if explicit description to that effect is inadvertently omitted from the following description.

[0138] Referring to FIGS. 10D and 14, at step 1064, the computer system can apply convolution layer 1404 to the fifth embedding matrix 1402 (also referred to as “the embedding matrix”), thereby producing a convolved fifth embedding matrix 1406 (also referred to as a “convolved embedding matrix”). Convolution layer 1404 may comprise a single convolution layer, rather than, e g., an entire convolutional neural network, and may be used to perform a convolution operation on the fifth embedding matrix 1402 using a convolution kernel (which may comprise a trainable parameter of a head sub-model).

[0139] At step 1066, the computer system can apply pooling layer 1408 to the convolved fifth embedding matrix 1406 (alternatively, “the convolved embedding matrix”), thereby producing a feature vector 1410. The computer system can use pooling layer 1408 to produce the feature vector 1410 using any appropriate pooling techniques (e.g.. average pooling or max pooling). By executing step 1066 for each head sub-model of the plurality of head sub-models, the computer system can produce a plurality of feature vectors corresponding to the plurality of head sub-models. In some embodiments, these feature vectors may comprise an input to one or more outlier detection models of a plurality ofoutlier detection models and may be used by the one or more outlier detection models to detect one or more time-series outliers in the query segment.

[0140] At step 1068, the computer system can apply a multilayer perceptron 1412 to the feature vector 1410, thereby producing an outlier segment classification 1414. This outlier segment classification 1414 can be one of a plurality of outlier segment classifications, produced by the computer system by performing step 1068 for each head sub-model of the plurality’ of head sub-models. In some embodiments, the multilayer perceptron 1412 can be followed by a sigmoid activation function in order to produce the outlier segment classification 1414. As described above, each outlier segment classification of the plurality of outlier segment classifications can indicate whether the query segment contains one or more outliers of an outlier type corresponding to the outlier segment classification. Outlier segment classifications (such as outlier segment classification 1414) can comprise binary classifications, or any other appropriate type of classification (e.g., a score, a percentage, a multi-class level, etc.) as described above.

[0141] Optionally at step 1070 the computer system can use each head-sub model of the plurality' of head sub-models to additionally produce a plurality of ranking vectors. The computer system can do so by applying a linear neural network layer 1416 (e.g., a fully- connected neural network) to the feature vector, thereby producing a ranking vector (e.g., ranking vector 1418) of the plurality of ranking vectors. As described below, this plurality of ranking vectors can be later be multiplied by an outlier detection matrix (e.g., outlier detection matrix 1268 in FIG. 12C) in order to produce a plurality of ranking lists. As described in more detail below, each ranking vector can comprise a 1 by k vector, where k is a hyperparameter that comprises a number of elements used to characterize the “response” of a given outlier detection model to an input query’ segment. As described above, the plurality’ of feature vectors, the plurality of outlier segment classifications, and the plurality of ranking vectors can comprise outputs of the plurality of head sub-models.

[0142] Referring to FIGS. 10D and 12C, at step 1072, the computer system can optionally generate a plurality of ranking lists, e.g., using a plurality of ranking vectors and an outlier detection matrix 1268. In some embodiments, the computer system can perform a matrix multiplication between each ranking vector of the plurality of ranking vectors and the outlier detection matrix 1268, thereby producing the plurality of ranking lists (e.g.. ranking lists1270-1276) corresponding to the plurality7of outlier types. Each ranking list of the plurality7of ranking lists can comprise a plurality of rankings, and each ranking in a ranking list can correspond to a different time-senes outlier detection model of a plurality of time-series outlier detection models. Each ranking of each ranking list can indicate a relative efficacy of a corresponding time-series outlier detection model at detecting one or more time-series outliers of a corresponding outlier type in the query segment. As an example, if there are five different outlier types, the plurality of ranking lists could comprise five ranking lists, each corresponding to a different outlier type of the five outlier types. If one of those outlier types is shapelet outliers, and there are a pool of twenty time-series outlier detection models, then the shapelet ranking list could comprise twenty7rankings corresponding to the twenty timeseries outlier detection models. Each ranking in that shapelet ranking list can indicate the relative efficacy of the corresponding time-series outlier detection model (e.g., relative to every other outlier detection model) at detecting one or more shapelet outliers in the query segment.

[0143] While some outlier detection models may perform better, on average, at detecting time-series outliers and detecting particular types of time-series outliers, different outlier detection models may perform better or worse on different time-series. As an example, a given outlier detection model may be effective at detecting trends in high frequency query segments, but not in low frequency query7segments. By generating a plurality7of ranking lists for a given query segment during production, the meta-leamer can better identify the best performing outlier detection model(s) for detecting outliers in the query segment.

[0144] Rankings in the plurality of ranking lists could comprise, e.g., numerical scores, such that higher scores indicate a greater relative efficacy at detecting outliers in the query segment, and lower scores indicate a lesser relative efficacy. As an alternative, a ranking could indicate a placement, e.g., among twenty rankings, a ranking of “1st” could indicate that the corresponding outlier detection model is expected to achieve the best outlier detection performance relative to all other outlier detection models ranked by the corresponding ranking list, while a ranking of “20th'’ could indicate that the corresponding outlier detection model is expected to achieve the worst outlier detection performance relative to all other outlier detection models ranked by the corresponding ranking list.

[0145] In some embodiments, the computer system can generate ranking lists depending on the corresponding outlier segment classifications. For example, the computer system could generate a ranking list for a given outlier type (e.g., trend outliers) if the corresponding outlier segment classification indicates a high likelihood that the query segment contains trend outliers, and could not generate the ranking list otherwise. However, in other embodiments the computer system can generate the ranking lists using the plurality of first ranking vectors and the outlier detection matrix 1268 regardless of the nature of the corresponding outlier segment classifications.

[0146] In some embodiments, the outlier detection matrix 1268 can comprise a matrix of k by n trainable parameters, where n is the number of outlier detection models in the plurality of outlier detection models and k is a hyperparameter. These trainable parameters can be determined during training and fixed during production. Each of the n columns in the outlier detection matrix 1268 can correspond to a different outlier detection model of the plurality of outlier detection models and can comprise k elements that can characterize the “response” of the corresponding outlier detection model to a given input query segment. A higher value of k may result in greater ranking list accuracy, at the cost of additional memory requirements and training time, while a lower value of k may achieve greater speed and memory efficiency, at the cost of accuracy.

[0147] When multiplying a given 1 by k ranking vector by the k by n outlier detection matrix, the result (a ranking list) can comprise a 1 by n vector, where each of the n columns of the ranking list corresponds to a different outlier detection model, and where each of the n columns of the ranking list effectively comprises a dot product of the k elements of the ranking vector and the k elements contained in a corresponding column of the k by n outlier detection matrix. The dot product can effectively correlate two sets of values represented by vectors, as the dot product typically results in a scalar quantity of greater magnitude when the two sets of values are more similar. Hence, greater values (or “scores”) in a 1 by n ranking list can indicate a greater relative efficacy of the corresponding outlier detection model, while lesser scores can indicate a lesser relative efficacy.

[0148] At step 1074, the computer system can optionally determine the best performing outlier detection model(s) for each outlier type, which can subsequently be used to optionally detect one or more time-series outliers in the query segment. In some embodiments, thecomputer system may only determine the best performing outlier detection models for types of outliers that are expected to be in the query segment, e.g., based on the plurality of outlier segment classifications. For example, if a global points outlier segment classification indicates that the query segment is unlikely to contain any global points outliers, then the computer system may not determine the best performing global points outlier detection model, as the computer system may not attempt to detect any global points outliers in the query segment. However, in other embodiments, the computer system may determine the best performing outlier detection model(s) for each outlier type, regardless if such outlier detection models are to be used to detect outliers in the query segment.

[0149] As such, the computer system can optionally determine a subset of outlier pes of the plurality of outlier types. This subset of outlier types may comprise outliers that are likely contained within the query segment. In some embodiments, the computer system can determine, for each outlier type of the plurality of outlier types if a corresponding outlier segment classification indicates that the query segment contains one or more outliers of that outlier type. If the corresponding outlier segment classification indicates that the querysegment contains one or more outliers of that outlier type, the computer system can include that outlier type in the subset of outlier types. For example, if the seasonal outlier segment classification indicates that the query segment contains one or more seasonal outliers, the seasonal outlier type can be included in the subset of outlier types, while if the trend outlier segment classification indicates that the query segment does not contain one or more trend outliers, then the computer system may not include the trend outlier type in the subset of outlier types.

[0150] In other embodiments, each outlier segment classification can comprise a probability indicating a likelihood that the query- segment contains one or more outliers of an outlier type corresponding to that outlier segment classification. In these embodiments, the computer system can optionally compare, for each outlier type of the plurality of outlier types, the probability indicating the likelihood that the query segment contains one or more outliers of the outlier type to a corresponding probability threshold. The computer system can include that outlier type in the subset of outlier types if the probability exceeds the corresponding probability threshold.

[0151] For example, if the shapelet outlier classification indicates a 51% probability7that the query segment contains one or more shapelet outliers, and if a corresponding probability threshold (e.g., a shapelet probability threshold) is 50%. then the computer system can include the shapelet outlier type in the subset of outlier types, indicating that the query segment can be evaluated using the best performing shapelet outlier detection model(s) to detect one or more shapelet outliers in the query segment. Notably, different probability thresholds can be applied to different outlier types, which may7be useful depending on the time-series data under evaluation. For example, perhaps trend outliers in heartrate data may be more indicative of serious disease than other outlier types. As such, a trend outlier probability threshold may be set to a lower probability (e.g., 5%) compared to other outlier type probability7thresholds.

[0152] After optionally determining the subset of outlier types, the computer system can optionally evaluate a corresponding ranking list of the plurality of ranking lists for each outlier type of the subset of outlier ty pes, thereby determining one or more best performing time-series outlier detection models of the plurality of time-series outlier detection models based on a plurality of corresponding rankings in that corresponding ranking lists. For example, if the subset of outlier types comprises global point outliers, trend outliers, and seasonal outliers, the computer system can evaluate a global points ranking list, a trend ranking list, and a seasonal ranking list, in order to determine one or more best performing global point outlier detection models, one or more best performing trend outlier detection models, and one or more best performing seasonal outlier detection models respectively.

[0153] In some embodiments, the one or more best performing time-series outlier detection model corresponding to each outlier type may comprise a single outlier detection model with the best score or ranking in the corresponding ranking list. However, in some cases it may be preferable to determine multiple best performing time-series outlier detection models, such as the top three for each outlier type of the subset of outlier types. The outputs of multiple timeseries outlier detection models can be ensembled in order to perform a time-series outlier classification, which may yield more accurate results than using the output of a single timeseries outlier detection model.

[0154] At step 1076, the computer system can optionally use the one or more best performing time-series outlier detection model to detect one or more time-series outliers ofeach outlier type of the subset of outlier types. In some embodiments, the computer system can, for each outlier type of the subset of outlier types, apply the one or more best performing time-series outlier detection models to the query segment, thereby detecting one or more time-series outliers of that outlier type.

[0155] However, some time-series outlier detection models may use or benefit from additional data generated by the meta-leamer during the steps described above. For example, an outlier detection model could use the feature vectors determined by the plurality of head sub-models to determine if a query segment contains time-series outliers, rather than the query segment itself. As such, in some embodiments, the computer system can optionally apply the one or more best performing time-series outlier detection models to one or more of the following, thereby detecting one or more time-series outliers of the outlier type: (1) the query segment. (2) the one or more first embedding matrices, (3) the one or more second embedding matrices, (4) the one or more first attention matrices, (5) the one or more third embedding matrices, (6) the one or more fourth embedding matrices, (7) the one or more second attention matrices, (8) the embedding matrix (or “fifth embedding matrix”), (9) one or more feature vectors of a plurality of feature vectors produced by the computer system by applying each head sub-model of the plurality of head sub-models to the embedding matrix (or “fifth embedding matrix”), and (9) one or more ranking vectors of the plurality of ranking vectors.

[0156] As described above with reference to FIG. 2, each best performing time-series outlier detection model can classify each time-series observation in the query segment as outliers or non-outliers. For example, each best performing time-series outlier detection model can produce a set of probabilities corresponding to a set of time-series observations in the query7segment, and each probability' can indicate the probability' that the corresponding time-series observation is an outlier of a given outlier type. As one alternative, each best performing time-series outlier detection model could produce an outlier detection table indicating which observations within the query' segments are outliers of the corresponding outlier type. If multiple best performing time-series outlier detection models are used to detect outliers of a given outlier type in the query' segment, the outputs of those best performing time-series outlier detection models could be combined or ensembled in some manner to produce a final classification. For example, three sets of probabilitiescorresponding to the three best performing trend outlier detection models could be averaged to produce a final set of probabilities indicating the likelihood of each query observation comprising a trend outlier.

[0157] However, it should be understood that in some embodiments, the computer system may not determine the best performing outlier detection models (or even produce a plurality of ranking lists) and may not use the best performing outlier detection models to detect one or more time-series outliers in the query segment. In some cases, the computer system may produce the plurality of outlier segment classifications and optionally the plurality of ranking lists, and output those outlier segment classifications and ranking lists to an operator or another computer system, which may use those outlier segment classifications and ranking lists for some purpose, such as selecting the best performing time-series outlier detection models and detecting outliers in the query segment using the selected time-series outlier detection models.D. Training

[0158] Having described methods of using a meta-leamer machine learning model in production, e.g., for the purpose of generating a plurality of outlier segment classifications and a plurality of ranking lists corresponding to an input query segment, it may be useful to describe methods of training the meta-leamer machine learning model, as training methods can generally be understood in context of production. In practice, there are a variety of methods that can be used to train a machine learning model. For ease of explanation, the following description assumes a "‘round based” framework of machine learning model training, e.g., as described above in Section I. However, methods according to embodiments could conceivably be adapted to other training frameworks.

[0159] As a brief review, a machine learning model (such as the meta-leamer machine learning model described herein) can be defined by model parameters, which generally control how the machine learning model processes inputs and produces outputs based on those inputs. If a machine learning model comprises multiple sub-models, each sub-model can have its own set of sub-model parameters, which can collectively comprise the model parameters. During a round of supervised training, a machine learning model can, in effect, perform whatever task it is intended to perform during production, i.e., process receivedinputs and produce corresponding outputs. A machine learning model can perform this task on training data, which may be “labeled” such that the expected or desired output corresponding to that training data is known. For example, for a machine learning model used to classify images as images of dogs or images of cats, the training data may comprise images that are pre-labeled as images of dogs or cats.

[0160] As the labels of the training data are known, the outputs of the machine learning model corresponding to those inputs can be compared to those labels, in order to determine loss or error values. Such loss or error values can relate to the difference between the expected or desired output of the machine learning model (e.g., an accurate classification) and the actual output of the machine learning model. In the context of embodiments, a training query segment could comprise a segment of time-series data for which all the observations in that segment are labeled as outliers or non-outliers, and are further labeled based on outlier type. From these labels, “ground truth” outlier segment classifications can be determined, as the types of outliers present in the training segment are known. Further, the labels could include a training ranking list corresponding to that training query' segment, which indicates how effective each outlier detection model (of a plurality of outlier detection models) were at detecting those outliers in the training segment. A loss or error value can be determined based on the difference betw een the outlier segment classifications and ranking lists produced by the meta-leamer machine learning model during that round of training and the ground truth outlier segment classifications and ranking lists corresponding to the labels. Outlier segment classifications and ranking lists that are more similar to ground truth outlier segment classifications and ground truth ranking lists may result in smaller losses or error values, while more different outlier segment classifications and ranking lists may result in greater losses or error values.

[0161] Metrics of the loss or error value can be used to update the model parameters. For example, the gradient of the loss or error value in “parameter space” relates to the direction of steepest descent in the loss or error value, i.e., the immediate or differential change in model parameters that result in the greatest (expected or otherwise) reduction in the loss or error value. By altering the model parameters in accordance with the determined gradient, the model parameters can be updated in a given training round. A subsequent training round can begin with a new batch of training data, and the machine learning model being trained canproduce a new set of outputs, which can then be compared to the expected or desired outputs based on the training labels corresponding to the training data. From this, loss or error values can again be determined and used to update the model parameters.

[0162] This process can be repeated until a terminating condition has been met. In some embodiments, a terminating condition can comprise a defined number of training rounds (e.g., 10,000) or training “epochs,’' which can comprise complete passes through the entire training data set. Another example of a terminating condition is a convergence condition, in which training is completed once the model parameters and loss or error values converge. Convergence is achieved, broadly, when the model parameters or error values tend to a limit, or in other words, when the change in the model parameters and error values in successive training round becomes successively smaller, until individual rounds of training produce very' little appreciable effect on the machine learning model’s performance. Once the terminating condition (e g., a defined number of training rounds or convergence) has been met, the model parameters can be fixed and used to process inputs (such as query segments of time-series data) during production, e.g., as described above.

[0163] Having completed this brief review of machine learning model training, training methods according to embodiments are described in more detail below with reference to FIGS. 15A-15E (which depict a flowchart of a meta-leamer model training method according to some embodiments), FIGS. 12A-12C (which, as described above, depict the structure of an exemplary meta-leamer machine learning model), FIGS. 13A-13C (which depict the structure of the first attention layer and the second attention layer), FIG. 14 (which depicts the structure of an exemplary' head sub-model) and FIG. 16 (which depicts exemplary training, testing, and validation data). As training involves many of the same operational steps as production (e.g., applying convolutional neural networks or attention layers to embedding matrices, pooling embedding matrices, etc ), much of the description below may be similar to the description of production methods provided above.

[0164] Referring to FIG. 15 A, at step 1502 a computer system can receive a training set comprising a plurality of query training segments and a plurality of baseline training segments. In some embodiments, the computer system can receive the training set from a database, a data stream, a local memory element (such as a hard drive), cloud storage, an I / O interface, etc. In some embodiments, each query training segment of the plurality- of querytraining segments can have a predefined or fixed length and can include one or more outliers of one or more outlier types. In some embodiments, each baseline training segment of the plurality of baseline training segments can comprise a time-series data segment with arbitrary length that may not contain outliers. A computer system or a user of the computer system (e.g., a data scientist) can identify and label time-series data segments that do not contain outliers and use them as baseline training segments. If necessary, the computer system can perform any initial processing operations (e.g., converting categorical data to numerical data, padding, normalization, re-sampling, etc.) on the training set, e.g., as described above with reference to FIG. 10 A.

[0165] FIG. 16 depicts an exemplary meta training set 1650, meta validation set 1652 and meta test set 1654. Such data can be used to train, validate, and test a meta-leamer according to embodiments. These data sets can be further separated into a support set 1656, which can be used to train the meta-leamer machine learning model, and a query set 1658, which can be applied to recommended (e.g., best performing) outlier detection models in order to evaluate the performance of the machine learning model and adjust machine learning model parameters.

[0166] The meta training set 1650 and support set 1656 can comprise a plurality of baseline training segments (1602-1606) and a plurality of query training segments (1618-1622). This plurality of query training segments can also comprise part of the uery set 1658. The meta validation set 1652 and support set 1656 can comprise a plurality of baseline validation segments (1608-1610) and a plurality’ of query validation segments (1624-1626). The plurality of query validation segments can also comprise part of the query set 1658. The meta test set 1654 and support set 1656 can comprise data used to test the meta-leamer machine learning model, and can comprise a plurality7of baseline test segments (1612-1616) and a plurality of query test segments (1628-1632). The plurality of query test segments can also comprise part of the query set 1658.

[0167] Referring back to FIG. 15 A, after receiving the test set, the computer system can begin performing an iterative training process used to train the meta-leamer machine learning model until a terminating condition has been met. As described above, the meta-leamer machine learning model can comprise a multilay er sub-model and a plurality of head submodels. The multilayer sub-model may be defined by a set of multilayer sub-modelparameters, and the plurality of head sub-models may be defined by a plurality of sets of head sub-model parameters. In general terms, the iterative training process can involve iteratively updating the meta-leamer machine learning model by updating the set of multilayer submodel parameters and the plurality of head sub-model parameters until the terminating condition has been met. This iterative training process can comprise a plurality of training rounds.

[0168] At step 1504, the computer system can begin one of these training rounds, which can involve the remainder of the steps depicted in FIGS. 15A-15E, i.e., steps 1504-1584. As described in further detail below and with reference to FIG. 15E, after completing one of these training rounds, the computer system can determine if a terminating condition has been met (indicating completion of training). If the terminating condition has not been met, the computer system can return to step 1504 (e.g.. from step 1582 of FIG. 15E) and perform an additional training round. This process can be repeated until the terminating condition has been met.

[0169] At step 1506 the computer system can sample a baseline training segment from the plurality of baseline training segments. The computer system can use any appropriate sampling process, such as random sampling, or by iterating through the training set in some predefined order and sampling the baseline training segment based on this iteration (e.g., by sampling the first baseline training segment in the first training round, then sampling the second baseline training segment in the second training round, etc.) In some embodiments, the computer system can sample a baseline training segments that has not already been sampled in a previous training round. However, if training is performed over multiple training epochs, then the computer system may sample the same baseline training segment multiple times.

[0170] Likewise, the computer system can sample a query training segment from the plurality of query training segments. The computer system can use any appropriate sampling process, such as random sampling, or by iterating through the training set in some predefined order and sampling the query training segment based on this iteration (e.g., by sampling the first query training segment in the first training round, then sampling the second query7training segment in the second training round, etc.) In some embodiments, the computer system can sample a query training segments that has not already been sampled in a previoustraining round. However, if training is performed over multiple training epochs, then the computer system may sample the same query training segment multiple times.

[0171] At step 1508 the computer system can segment the baseline training segment into one or more baseline subsegments. Likewise, the computer system can segment the query training segment into one or more query subsegments. As depicted in FIG. 12 A, these one or more baseline subsegments 1202 and one or more query subsegments 1204 can comprise the input to a first convolutional neural network 1206. which may comprise the first layer of a multilayer sub-model of the meta-leamer machine learning model. Subsegmentation processes, such as those described above with reference to FIGS. 10A and 11 can be used for this purpose. In some embodiments, segmenting the baseline training segment into one or more baseline subsegments can involve performing a period estimation process on the baseline training segment, thereby determining an estimated baseline period. Likewise, segmenting the query training segment into one or more query subsegments can comprise performing a period estimation process on the query training segment, thereby determining an estimated query period. Such period estimation processes can involve techniques such as the Discrete Fourier Transform to determine frequency components of the baseline training segment and query training segment, and the computer system can determine the period based on those determined frequency components. These period estimation processes can be used to determine the fixed length of rolling or sliding windows used to define the length of the baseline subsegments and query subsegments.

[0172] Further, segmenting the baseline training segment into one or more baseline subsegments can involve performing a rolling segmentation process on the baseline training segment (e.g., based on the estimated baseline period), thereby determining one or more unsealed baseline subsegments. Likewise, segmenting the query7training segment into one or more query subsegments can involve performing a rolling segmentation process on the query training segment based on the estimated query period, thereby determining one or more unsealed query subsegments. Such rolling segmentation processes can generally involve sampling time-series data from the baseline training segment and query7training segment in rolling or sliding windows, resulting in e.g., an arbitrary number of baseline subsegments and a fixed number of query subsegments. Such baseline subsegments and query subsegmentscan be overlapping or non-overlapping, depending on the stride or other parameters used to define the rolling segmentation process.

[0173] In some embodiments, segmenting the baseline training segment into one or more baseline subsegments can further involve scaling the one or more unsealed baseline subsegments to a fixed length, thereby producing the one or more baseline subsegments. Likewise, segmenting the query segment into one or more query subsegments can further involve scaling the one or more unsealed query subsegments to a fixed length, thereby producing the one or more query subsegments.

[0174] After producing the one or more baseline subsegments and the one or more query subsegments, the computer system can apply a multilayer sub-model to the one or more baseline subsegments and the one or more query subsegments, thereby producing an embedding matrix (also referred to as a “fifth embedding matrix”). The multilayer sub-model can be defined by a set of multilayer sub-model parameters, which can be updated during each round of the iterative training process to improve the performance of the multilayer submodel.

[0175] Referring to FIGS. 12A-12C, in some embodiments, the multilayer sub-model can comprise: (1) a first convolutional neural network 1206 defined by a set of first convolutional neural network parameters. (2) a second convolutional neural network 1218 defined by a set of second convolutional neural network parameters, (3) a third convolutional neural network 1232 defined by a third set of convolutional neural network parameters, (4) a first attention layer 1222 defined by a set of first attention layer parameters, and (5) a second attention layer 1236 defined by a set of second attention layer parameters. The set of multilayer sub-model parameters can comprise the set of first convolutional neural network parameters, the set of second convolutional neural network parameters, the set of third convolutional neural network parameters, the set of first attention layer parameters, and the set of second attention layer parameters. During a round of training, these model parameters can be updated collectively based on one or more combined error or loss values determined during training.

[0176] In more detail, and with reference to FIGS. 15A and 12A, at step 1510 the computer system can apply the first convolutional neural network 1206 to the one or more baseline subsegments 1202 and the one or more query' subsegments 1204. As described above with reference to FIG. 9, the first convolutional neural network 1206 can comprise an alternatingseries of convolution layers and pooling layers, followed by a neural network (e.g., a fully- connected neural network). In some embodiments a shared convolution kernel can be used for both the baseline subsegments 1202 and query subsegments 1204. The computer system can thereby produce one or more baseline embedding matrices 1208 (corresponding to the one or more baseline subsegments 1202), and one or more query' embedding matrices 1210 (corresponding to the one or more query subsegments 1204), which can comprise the outputs of the first convolutional neural network 1206.

[0177] Afterwards, at steps 1512- 1514 the computer system can generate one or more first embedding matrices 1216 based on the one or more baseline embedding matrices 1208 and the one or more query' embedding matrices 1210. At step 1512, the computer system can pool the one baseline embedding matrices 1208 (e.g., in first pooling operations 1212) thereby producing a pooled baseline embedding matrix 1214. Pooling techniques such as max pooling or average pooling can be used for this purpose.

[0178] At step 1514 the computer system can determine one or more embedding difference matrices based on the one or more query' embedding matrices 1210 and the pooled baseline embedding matrix 1214. The one or more first embedding matrices 1216 can comprise these one or more embedding difference matrices. Each embedding difference matrix of the one or more embedding difference matrices can comprise a difference betyveen a different query' embedding matrix of the one or more query embedding matrices 1210 and the pooled baseline embedding matrix 1214. The computer system can determine these embedding difference matrices using elementwise matrix subtraction or any other appropriate technique.

[0179] At step 1516 the computer system can apply the second convolutional neural network 1218 to the one or more first embedding matrices 1216, thereby producing the one or more second embedding matrices 1220. The second convolutional neural netyvork 1218 can comprise an alternating series of convolution layers (with respective convolution kernels) and subsampling (pooling) layers, followed by a neural network, which may comprise a fully- connected neural network.1. First Attention Layer

[0180] At step 1518, the computer system can apply the first attention layer 1222 to the one or more second embedding matrices 1220. thereby producing one or more first attentionmatrices 1224. Steps 1520-1536 of FIGS. 15A and 15B generally comprise steps performed by the computer system (using the first attention layer 1222) in order to produce the one or more first attention matrices 1224. The structure of the first attention layer 1222 (and by extension the second attention layer 1236 of FIG. 12B) and steps 1520-1036 are described in more detail with reference to FIGS. 13A-13C. In some embodiments, the first attention layer 1222 can comprise a first attention multilayer perceptron (e.g., 1324 in FIG. 13B) defined by a set of first attention multilayer perceptron parameters. The first attention layer 1222 can further comprise a first attention linear neural network layer (e g., 1334 in FIG. 13C) defined by a set of first attention layer linear neural network parameters. The set of first attention layer parameters (described above) can comprise the set of first attention multilayer perceptron parameters and the set of first attention layer neural network parameters. These parameters can be updated during the iterative training process.

[0181] At step 1520 the computer system can combine the one or more second embedding matrices 1302 (e.g., second embedding matrices 1304-1310 as depicted in FIG. 13A), thereby producing a combined second embedding matrix 1312. The one or more second embedding matrices 1302 can be combined using techniques such as concatenation, or by otherwise appending the one or more second embedding matrices 1302 to one another.

[0182] Referring to FIG. 15B, at step 1522 the computer system can transpose the combined second embedding matrix 1312, thereby producing a combined transposed second embedding matrix 1314. The computer system can use appropriate transposition method to produce the combined transposed second embedding matrix 1314, e.g., using in-place matrix transposition, or by initializing an empty matrix with ‘’swapped” dimensions, then iterating through the combined second embedding matrix 1312 and copying the value of the elements stored therein to the correct location of the empty matrix until the combined transposed second embedding matrix 1314 is produced.

[0183] At step 1524 the computer system can perform a matrix multiplication on the combined second embedding matrix 1312 and the combined transposed second embedding matrix 1314, thereby producing a first embedding correlation matrix 1316. In FIG. 13A, the first embedding correlation matrix 1316 can comprise an nqby nq* w square matrix, and each element can comprise a correlation value, where the i, / thelement comprises a correlation value representing the correlation between d observations corresponding to the Ithrow of the combined second embedding matrix 1312 and d observations corresponding to the jthrow of the combined second embedding matrix 1312.

[0184] At step 1526 the computer system can perform a pooling operation on the first embedding correlation matrix 1316 (i.e., attention pooling 1320), thereby producing a first pooled embedding correlation matrix 1322. The computer system can accomplish this using any appropriate pooling operation, e.g., average pooling, max pooling, etc.

[0185] At step 1528 the computer system can apply a first attention multilayer perceptron 1324 (e.g., a neural network comprising three or more layers) to the first pooled embedding correlation matrix 1322, thereby producing a first non-normalized correlation matrix 1326.As described above, this first attention multilayer perceptron 1324 can be defined by a set of trainable first attention multilayer perceptron parameters.

[0186] At step 1530 the computer system can normalize the first non-normalized correlation matrix 1326 using linear normalization 1328, thereby producing a first normalized correlation matrix 1330. As described above, any appropriate form of linear normalization can be used for this purpose, such as “max-min” normalization.

[0187] Referring to FIGS. 15B and 13C, at step 1532 the computer system can concatenate the combined second embedding matrix 1312 with the first normalized correlation matrix 1330. thereby generating an enhanced second embedding matrix 1332. As depicted in FIG.13C, the computer system can, e.g., append the d’ elements in each of the nq* w rows of the first normalized correlation matrix 1330 to the corresponding row of the combined second embedding matrix 1312, thereby producing a nq* w by d + d’ enhanced second embedding matrix 1332. The enhanced second embedding matrix 1332 can contain both the features represented by the combined second embedding matrix 1312 and the correlations between those features represented by the first normalized correlation matrix 1330.

[0188] At step 1534 the computer system can apply a first attention linear neural network layer 1334 (e.g., a fully-connected neural network layer) to the enhanced second embedding matrix 1332, thereby generating a combined first attention matrix 1336. The first attention linear neural network layer 1334 may benefit from the additional correlation information encoded by the enhanced second embedding matrix 1332, and may therefore produce anoutput (i.e., the combined first attention matrix 1336) that is more useful for detecting different types of outliers in the input query segment.

[0189] At step 1536 the computer system can segment the combined first attention matrix 1336, thereby producing one or more first attention matrices 1338 (e.g., first attention matrices 1340-1346 as depicted in FIG. 13C). This segmentation can enable further processing by other layers of the multilayer sub-model of the meta-leamer machine learning model, e.g.. the third convolutional neural network 1232 as depicted in FIG. 12B. Referring now to FIGS. 15B and 12B, the one or more first attention matrices 1224 can effectively comprise the output of the first attention layer 1222.

[0190] At step 1538 the computer system can generate one or more third embedding matrices 1230 based on the one or more first attention matrices 1224. Step 1538 can comprise a number of sub-steps. The computer system can pool the one or more first attention matrices 1224 (e.g.. in second pooling operations 1226). thereby producing a pooled first attention matrix 1228. The computer system can use any appropriate pooling operations (e.g., average pooling or max pooling) for this purpose. Pooling in this manner can effectively extract information collected or "ensembled'’ from the one or more first attention matrices 1224.

[0191] Afterwards, the computer system can determine one or more sum attention matrices. The one or more third embedding matrices 1230 can comprise these one or more sum attention matrices. Each sum attention matrix of the one or more sum attention matrices can comprise a matrix sum of a different first attention matrix (of the one or more first attention matrices 1224) and the pooled first attention matrix 1228. This process can enhancing the information contained in the one or more third embedding matrices 1230.

[0192] Referring to FIGS. 15C and 12B, at step 1540 the computer system can apply the third convolutional neural network 1232 to the one or more third embedding matrices 1230, thereby producing one or more fourth embedding matrices 1234. The third convolutional neural network 1232 can comprise an alternating series of convolution layers (with respective convolution kernels) and subsampling (pooling) layers, followed by a neural network (which may comprise a fully-connected neural network.2. Second Atention Layer

[0193] At step 1542 the computer system can apply the second attention layer 1236 to the one or more fourth embedding matrices 1234, thereby producing one or more second attention matrices 1238. The operations performed by the computer system using the second attention layer 1236 may be similar to the operations performed by the computer system using the first attention layer 1222. Likewise, the structure of the second attention layer 1236 may be similar to the structure of the first attention layer 1222. Steps 1544-1560 of FIGS. 15C and 15D generally comprise steps performed by the computer system (using the second attention layer 1236) in order to produce the one or more second attention matrices 1238. These steps are described in more detail with reference to FIGS. 13A-13C. The second attention layer 1236 can be defined by a set of second attention layer parameters. As depicted in FIG. 13B, the second attention layer 1236 can comprise a second attention multilayer perceptron 1324 defined by a set of second attention multilayer perceptron parameters. Likewise, as depicted in FIG. 13C, the second attention layer can comprise a second attention linear neural network layer 1334 defined by a set of second attention layer linear neural network parameters. The set of second attention layer parameters can comprise the set of second attention multilayer perceptron parameters and the set of second attention layer linear neural network parameters.

[0194] At step 1544 the computer system can combine the one or more fourth embedding matrices 1302 (e.g., fourth embedding matrices 1304-1310 as depicted in FIG. 13A). thereby producing a combined fourth embedding matrix 1312. The one or more fourth embedding matrices 1302 can be combined using techniques such as concatenation, or by otherwise appending the one or more fourth embedding matrices 1302 to one another.

[0195] At step 1546 the computer system can transpose the combined fourth embedding matrix 1312, thereby producing a combined transposed fourth embedding matrix 1314. The computer system can use appropriate transposition method to produce the combined transposed fourth embedding matrix 1314, e.g., using in-place matrix transposition or by initializing an empty matrix with “swapped’' dimensions, then iterating through the combined fourth embedding matrix 1312 and copying the value of the elements stored therein to the correct location of the empty matrix until the combined transposed fourth embedding matrix 1314 is produced.

[0196] At step 1548 the computer system can perform a matrix multiplication on the combined fourth embedding matrix 1312 and the combined transposed fourth embedding matrix 1314, thereby producing a second embedding correlation matrix 1316. In FIG. 13 A, the second embedding correlation matrix 1316 can comprise an nq* w by nqsquare matrix, and each element can comprise a correlation value, where the i, jthelement comprises a correlation value representing the correlation between d observations corresponding to the Ithrow of the combined fourth embedding matrix 1312 and d observations corresponding to the jthrow of the combined fourth embedding matrix 1312.

[0197] Referring to FIGS. 15C and 13B, the computer system can perform a series of machine learning operations in order to extract useful features from the second embedding correlation matrix 1316. At step 1550. the computer system can perform a pooling operation on the second embedding correlation matrix 1316 (i.e., attention pooling 1320). thereby producing a second pooled embedding correlation matrix 1322. The computer system can accomplish this using any appropriate pooling operation, e.g., average pooling, max pooling, etc.

[0198] At step 1552, the computer system can apply a second attention multilayer perceptron 1324 (e.g., a neural network comprising three or more layers) to the second pooled embedding correlation matrix 1322, thereby producing a second non-normalized correlation matrix 1326.

[0199] At step 1554, the computer system can normalize the second non-normalized correlation matrix 1326 using linear normalization 1328, thereby producing a second normalized correlation matrix 1330. Any appropriate form of linear normalization can be used for this purpose, such as “max-min” normalization, as described above.

[0200] In the example of FIG. 13B, the resulting second normalized correlation matrix 1330 can comprise an nq* w by d’ matrix. The d’ values in each row of the second normalized correlation matrix 1330 can effectively encode how correlated each row of the combined fourth embedding matrix 1312 is with each other row of the combined fourth embedding matrix 1312. In some embodiments, d’ may be less than nq* w, and therefore the operations depicted in FIG. 13B may encode or “compress” the information represented by nq* w row elements of the second embedding correlation matrix 1316 into d’ elements.However, d' may also be greater than nq* w.

[0201] Referring to FIGS. 15C and 13C, at step 1556, the computer system can concatenate the combined fourth embedding matrix 1312 with the second normalized correlation matrix 1330. thereby generating an enhanced fourth embedding matrix 1332. As depicted in FIG. 13C, the computer system can, e.g., append the d’ elements in each of the nq* w rows of the second normalized correlation matrix 1330 to the corresponding row of the combined fourth embedding matrix 1312, thereby producing a nq* w by d + d' enhanced fourth embedding matrix 1332.

[0202] Referring to FIG. 15D and 13C, at step 1558, the computer system can apply a second attention linear neural network layer 1334 (e g., a fully-connected neural network layer) to the enhanced fourth embedding matrix 1332, thereby generating a combined second attention matrix 1336. As described above, the second attention linear neural network layer 1334 can be defined by a set of trainable second attention layer linear neural network parameters.

[0203] At step 1560, the computer system can segment the combined second attention matrix 1336, thereby producing one or more second attention matrices 1338 (e.g., second attention matrices 1340-1346 as depicted in FIG. 13C). Referring now to FIGS. 15D and 12B, the one or more second attention matrices 1238 can effectively comprise the output of the second attention layer 1236. These one or more second attention matrices 1238 can be used to generate the fifth embedding matrix 1242 (also referred to as just the “embedding matrix’"), which can be input into a plurality of head sub-models in order to produce a plurality of outlier segmentation classifications (and optionally a plurality of ranking lists). These outlier segment classifications and ranking lists can be compared to training labels in order to determine one or more loss or error values, which can be used to update the machine learning model parameters.

[0204] At step 1562, the computer system can pool the one or more second attention matrices 1238 (e.g., in third pooling operations 1240), thereby producing a pooled second attention matrix. The fifth embedding matrix 1242 (also referred to as an "‘embedding matrix”) can comprise the pooled second attention matrix. The computer system can accomplish this using any appropriate pooling operation, e.g., average pooling, max pooling, etc. The fifth embedding matrix 1242 can comprise the output of the multilayer sub-model ofthe meta-leamer machine learning model, and can comprise the input to a plurality of head sub-models 1278 depicted in FIG. 12C.3. Head Sub-Models

[0205] Referring to FIGS. 10D and 12C, at step 1564, the computer system can apply each head sub-model of the plurality of head sub-models 1278 (e.g., head sub-models 1244-1250) to the fifth embedding matrix 1242 (or “embedding matrix”), thereby producing a plurality of outlier segment classifications and optionally a plurality of ranking vectors, e.g., as described above with reference to the production phase.

[0206] Each head sub-model of the plurality of head sub-models can correspond to a different outlier type of a plurality of outlier types. Further, each head sub-model of the plurality of head sub-models can be defined by a different set of head sub-model parameters of a plurality of sets of head sub-model parameters. Each head sub-model of the plurality of head sub-models can comprise a different convolution layer (e.g., convolution layer 1404 as depicted in FIG. 14) defined by a different set of convolution layer parameters. Further, each head sub-model of the plurality of head sub-models can comprise a different pooling layer (e.g., pooling layer 1408) defined by a different set of pooling layer parameters.Additionally, each head sub-model of the plurality of head sub-models can comprise a different linear neural network layer (e.g., linear neural network layer 1416) defined by a different set of linear neural network layer parameters, and a different multilayer perceptron (e.g., multilayer perceptron 1412) defined by a different set of multilayer perceptron parameters. In some embodiments, each set of head sub-model parameters can comprise a different set of convolution layer parameters, a different set of pooling layer parameters, a different set of linear neural network layer parameters, and a different set of multilayer perceptron parameters.

[0207] In more detail, the plurality of head sub-models 1278 can be used by the computer system to produce a plurality of feature vectors, which can subsequently be used to produce a plurality of outlier segment classifications (e.g., outlier segment classifications 1252-1258) and optionally a plurality of ranking vectors (e.g., ranking vectors 1260-1266). As such, at step 1062, the computer system can apply each head sub-model of the plurality of head submodels 1278 to the fifth embedding matrix 1242 (or “embedding matrix”), thereby producingthe plurality of outlier segments classifications and (optionally), a plurality7of ranking vectors. These ranking vectors can then be matrix multiplied by an outlier detection matrix 1268 in order to produce a plurality of ranking lists (e.g., ranking lists 1270-1276).

[0208] Steps 1566-1572 generally comprise steps performed by the computer system (using the plurality' of head sub-models 1278) in order to produce the plurality' of outlier segment classifications, the plurality7of ranking lists, the plurality7of feature vectors, etc., as described above. The structure of the plurality of head sub-models 1278 and steps 1566-1572 are described in more detail with reference to FIG. 14.

[0209] At step 1566 the computer system can apply convolution layer 1404 to the fifth embedding matrix 1402 (or “embedding matrix”), thereby7producing a convolved fifth embedding matrix 1406 (or “convolved embedding matrix”). Convolution layer 1404 may comprise a single convolution layer, rather than, e g., an entire convolutional neural network, and may be used to perform a convolution operation on the fifth embedding matrix 1402 using a convolution kernel (which may comprise a trainable parameter).

[0210] At step 1568 the computer system can apply pooling layer 1408 to the convolved fifth embedding matrix 1406 (or “convolved embedding matrix”), thereby producing a feature vector 1410. The computer system can use pooling layer 1408 to produce the feature vector 1410 using any appropriate pooling techniques (e.g., average pooling or max pooling). By executing step 1568 for each head sub-model of the plurality of head sub-models, the computer system can produce a plurality7of feature vectors corresponding to the plurality7of head sub-models.

[0211] At step 1570 the computer system can apply multilayer perceptron 1412 to the feature vector 1410, thereby producing an outlier segment classification 1414. This outlier segment classification 1414 can be one of a plurality7of outlier segment classifications, produced by the computer system by performing step 1570 for each head sub-model of the plurality7of head sub-models. In some embodiments, the multilayer perceptron 1412 can be followed by a sigmoid activation function in order to produce the outlier segment classification 1414. Each outlier segment classification of the plurality of outlier segment classifications can indicate whether the training query segment contains one or more outliers of an outlier type corresponding to the outlier segment classification. The multilayer perceptron 1412 can be defined by a set of trainable multilayer perceptron parameters.

[0212] At step 1572 the computer system can use each head-sub model of the plurality of head sub-models to additionally produce a plurality of ranking vectors. The computer system can do so by applying a linear neural network layer 1416 (e.g., a fully-connected neural network) to the feature vector, thereby producing a ranking vector (e g., ranking vector 1418) of the plurality of ranking vectors. As described above, the layer linear neural network layer 1416 can be defined by a set of trainable layer linear neural network parameters. As described below, this plurality of ranking vectors can be later be multiplied by an outlier detection matrix (e.g., outlier detection matrix 1268 in FIG. 12C) in order to produce a plurality of ranking lists.

[0213] At step 1574 the computer system can optionally perform a matrix multiplication between each ranking vector of the plurality of ranking vectors and an outlier detection matrix 1268, thereby producing the plurality of ranking lists (e.g.. ranking lists 1270-1276) corresponding to the plurality of outlier types. Each ranking list of the plurality' of ranking lists can comprise a plurality of rankings, and each ranking in a ranking list can correspond to a different time-series outlier detection model of a plurality' of time-series outlier detection models. Each ranking of each ranking list can indicate a relative efficacy of a corresponding time-series outlier detection model at detecting one or more time-series outliers of a corresponding outlier type in the query segment. The outlier detection matrix can be defined by a set of outlier detection matrix parameters, which can comprise a set of coefficients of the outlier detection matrix and which may be updated during training.

[0214] Referring to FIG. 15E, at step 1576 the computer system can generate one or more loss values based on the plurality of outlier segment classifications and optionally the plurality of ranking lists. The computer system can generate the one or more loss values by comparing the plurality of outlier segment classifications and the plurality of ranking lists based on any training labels associated with the training query segment. The best outlier detection model for a given training query segment may already be known based on the training labels, or may be derived from, e g., a query set such as query set 1 58 in FIG. 1 . As such, the “true ranking lists” describing how well each outlier detection model detects outliers of each outlier type may be known or can be determined. The computer system can compare these true ranking lists to the plurality of ranking lists produced by the meta-leamer machine learning model. If the plurality of ranking lists are similar to the true ranking lists.then a corresponding loss value may be low, while if the plurality of ranking lists are dissimilar to the true ranking lists, a corresponding loss value may be high. Similarly, because the training query segment can be labeled, the types of outliers present in that training segment can be known. As such, ’’true outlier segment classifications” can be derived from such labels, and those true outlier segment classifications can be compared to the outlier segment classifications produced by the machine learning model in order to derive a corresponding loss value. In some embodiments, a loss value corresponding to the ranking lists and a loss value corresponding to the outlier segment classifications can be combined using a combined loss function (e.g., a weighted average) to produce a single combined loss value.

[0215] At step 1578, the computer system can update the machine learning model by updating the multilayer sub-model parameters, the plurality of sets of head sub-model parameters, and optionally the set of outlier detection matrix parameters based on the one or more loss values, thereby training the machine learning model. The computer system can use any appropriate technique for updating the model parameters, such as using stochastic gradient descent to determine differential changes in the multilayer sub-model parameters, the plurality of sets of head sub-model parameters, and the set of outlier detection matrix parameters that result in the greatest immediate reduction to the one or more loss values produced at step 1576.

[0216] At step 1580, the computer system can determine if the terminating condition has been met. As described above, in some embodiments, the terminating condition can comprise a defined number of training rounds, and the terminating condition can be met if a total number of training rounds performed equals or exceeds the defined number of training rounds. In other embodiments, the terminating condition can comprise a convergence condition. The terminating condition can be met if the set of multilayer sub-model parameters, the plurality of sets of head sub-model parameters, and the set of outlier detection matrix parameters converge, e.g., exhibit little to no change in consecutive training rounds.

[0217] If the terminating condition has not been met, at step 1582, the computer system can repeat the iterative training process and return to step 1504 of FIG. 15 A. The computer system can sample a new baseline training segment and query training segment and performan additional round of training. This process can be repeated until the terminating condition has been met.

[0218] Otherwise, at step 1584, the computer system can complete the iterative training process. At this point, the parameters of the meta-leamer machine learning model can be fixed, and the meta-leamer machine learning model can be used to generate a plurality of outlier segment classifications and optionally a plurality of ranking lists for unlabeled query segments, as described above with reference to the production phase.III. COMPUTER SYSTEM

[0219] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. Examples of such subsystems are shown in FIG. 17 in computer system 1700. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be the components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components. Examples of computer systems include desktop and laptop computers, tablets, mobile phones and other mobile devices.

[0220] The subsystems shown in FIG. 17 are interconnected via a system bus 1712. Additional subsystems such as a printer 1708, keyboard 1718, storage device(s) 1720, monitor 1724 (e.g., a display screen, such as an LED), which is coupled to display adapter 1714, and others are shown. Peripherals and input / output (I / O) devices, which couple to I / O controller 1702, can be connected to the computer system by any number of means known in the art such as input / output (I / O) port 1716 (e.g.. USB, FireWire®). For example, I / O port 1716 or external interface 1722 (e.g. Ethernet, Wi-Fi, etc.) can be used to connect computer system 1700 to a wide area network such as the Internet, a mouse input device, or a scanner. The interconnection via system bus 1712 allows the central processor 1706 to communicate with each subsystem and to control the execution of a plurality of instructions from system memory 1704 or the storage device(s) 1720 (e.g., a fixed disk, such as a hard drive, or optical disk), as well as the exchange of information betw een subsystems. The system memory 1704 and / or the storage device(s) 1720 may embody a computer readable medium. Another subsystem is a data collection device 1710, such as a camera, microphone, accelerometer, andthe like. Any of the data mentioned herein can be output from one component to another component and can be output to the user.

[0221] A computer system can include a plurality of the same components or subsystems, e.g., connected together by external interface 1722, by an internal interface, or via removable storage devices that can be connected and removed from one component to another component. In some embodiments, computer systems, subsystem, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components.

[0222] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components.

[0223] A computer system can include a plurality of the components or subsystems, e.g., connected together by external interface or by an internal interface. In some embodiments, computer systems, subsystems, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components.

[0224] It should be understood that any of the embodiments of the present invention can be implemented in the form of control logic using hardware (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software with a generally programmable processor in a modular or integrated manner. As used herein a processor includes a single-core processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and / or methods to implement embodiments of the present invention using hardware and a combination of hardware and software.

[0225] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as. for example. Java. C, C++, C#, Objective-C, Swift, or scripting language such as Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission, suitable media include random access memory' (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk), flash memory, and the like. The computer readable medium may be any combination of such storage or transmission devices.

[0226] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium according to an embodiment of the present invention may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g. a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer or other suitable display for providing any of the results mentioned herein to a user.

[0227] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Thus, embodiments can be involve computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective steps or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or in a different order. Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be optional. Additionally, and of the steps of any of the methods can be performed with modules, circuits, or other means for performing these steps.

[0228] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the invention.However, other embodiments of the invention may be involve specific embodiments relating to each individual aspect, or specific combinations of these individual aspects. The above description of exemplary embodiments of the invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form described, and many modifications and variations are possible in light of the teaching above. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications to thereby enable others skilled in the art to best utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated.

[0229] The above description is illustrative and is not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of the disclosure. The scope of the invention should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the pending claims along with their full scope or equivalents.

[0230] One or more features from any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the invention.

[0231] A recitation of “a”, “an” or “the” is intended to mean “one or more” unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusive or.” and not an “exclusive or” unless specifically indicated to the contrary.

[0232] All patents, patent applications, publications and description mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted to be prior art.

Claims

WHAT IS CLAIMED IS:

1. A method for using a machine learning model to detect outliers in input time-series data, wherein the machine learning model comprises a multilayer sub-model and a plurality of head sub-models, the method comprising performing, by a computer system: receiving the input time-series data, the input time-series data including a baseline segment and a query segment; segmenting the baseline segment into one or more baseline subsegments; segmenting the query segment into one or more query subsegments; applying the multilayer sub-model to the one or more baseline subsegments and the one or more query subsegments, thereby producing an embedding matrix, wherein the multilayer sub-model includes one or more attention layers; and applying each head sub-model of the plurality of head sub-models to the embedding matrix, thereby producing a plurality of outlier segment classifications, wherein each head sub-model of the plurality of head sub-models corresponds to a different outlier type of a plurality’ of outlier types, and wherein each outlier segment classification of the plurality of outlier segment classifications indicates whether the query segment contains one or more outliers of an outlier type corresponding to the outlier segment classification.

2. The method of claim 1, wherein the plurality of outlier ty pes comprise two or more of the following: a global point outlier type; a contextual point outlier type; a shapelet outlier type; a seasonal outlier type; and a trend outlier type.

3. The method of claim 1, wherein the baseline segment comprises a time-series data segment that does not contain outliers.

4. The method of claim 1, wherein: segmenting the baseline segment into the one or more baseline subsegments comprises:performing a period estimation process on the baseline segment, thereby determining an estimated baseline period, performing a rolling segmentation process on the baseline segment based on the estimated baseline period, thereby determining one or more unsealed baseline subsegments, scaling the one or more unsealed baseline subsegments to a fixed length, thereby producing the one or more baseline subsegments; and segmenting the query segment into the one or more query subsegments comprises: performing a period estimation process on the query7segment, thereby determining an estimated query period, performing a rolling segmentation process on the query7segment based on the estimated query period, thereby determining one or more unsealed query7subsegments, and scaling the one or more unsealed query subsegments to a fixed length, thereby producing the one or more query subsegments.

5. The method of claim 1, wherein applying each head sub-model of the plurality7of head sub-models to the embedding matrix comprises, for each head sub-model of the plurality of head sub-models: convolving and pooling the embedding matrix, thereby producing a feature vector; and applying a multilayer perceptron to the feature vector, thereby producing an outlier segment classification of the plurality of outlier segment classifications.

6. The method of claim 5, wherein applying each head sub-model of the plurality7of head sub-models to the embedding matrix additionally produces a plurality of ranking vectors, wherein convolving and pooling the embedding matrix comprises: applying a convolution layer to the embedding matrix, thereby producing a convolved embedding matrix, applying a pooling layer to the convolved embedding matrix, thereby producing the feature vector,wherein applying each head sub-model of the plurality of head sub-models to the embedding matrix further comprises, for each head sub-model of the plurality of head sub-models: applying a neural network layer to the feature vector, thereby producing a ranking vector of a plurality of ranking vectors; and generating a plurality of ranking lists based on the plurality of ranking vectors.

7. The method of claim 1, wherein the embedding matrix comprises a fifth embedding matrix, wherein the multilayer sub-model comprises a first convolutional neural network, a second convolutional neural network, a third convolutional neural network, a first attention layer, and a second attention layer, wherein the one or more attention layers comprise the first attention layer and the second attention layer, and wherein applying the multilayer sub-model to the one or more baseline subsegments and the one or more query subsegments comprises: applying the first convolutional neural network to the one or more baseline subsegments and the one or more query subsegments, thereby producing one or more baseline embedding matrices and one or more query embedding matrices; generating one or more first embedding matrices based on the one or more baseline embedding matrices and the one or more query embedding matrices; applying the second convolutional neural network to the one or more first embedding matrices, thereby producing one or more second embedding matrices; applying the first attention layer to the one or more second embedding matrices, thereby producing one or more first attention matrices; generating one or more third embedding matrices based on the one or more first attention matrices; applying the third convolutional neural network to the one or more third embedding matrices, thereby producing one or more fourth embedding matrices; applying the second attention layer to the one or more fourth embedding matrices, thereby producing one or more second attention matrices; and pooling the one or more second attention matrices, thereby producing a pooled second attention matrix, wherein the fifth embedding matrix comprises the pooled second attention matrix.

8. The method of claim 7, wherein generating the one or more first embedding matrices based on the one or more baseline embedding matrices and the one or more query embedding matrices comprises: pooling the one or more baseline embedding matrices, thereby producing a pooled baseline embedding matrix; and determining one or more embedding difference matrices, each embedding difference matrix of the one or more embedding difference matrices comprising a difference between a different query' embedding matrix of the one or more query' embedding matrices and the pooled baseline embedding matrix, wherein the one or more first embedding matrices comprise the one or more embedding difference matrices.

9. The method of claim 7, wherein generating the one or more third embedding matrices based on the one or more first attention matrices comprises: pooling the one or more first attention matrices, thereby producing a pooled first attention matrix; and determining one or more sum attention matrices, each sum attention matrix of the one or more sum attention matrices comprising a matrix sum of a different first attention matrix of the one or more first attention matrices and the pooled first attention matrix, wherein the one or more third embedding matrices comprises the one or more sum attention matrices.

10. The method of claim 7, wherein applying the first attention layer to the one or more second embedding matrices, thereby producing the one or more first attention matrices comprises: combining the one or more second embedding matrices, thereby producing a combined second embedding matrix: transposing the combined second embedding matrix, thereby producing a combined transposed second embedding matrix; performing a matrix multiplication on the combined second embedding matrix and the combined transposed second embedding matrix, thereby producing a first embedding correlation matrix;performing a pooling operation on the first embedding correlation matrix, thereby producing a first pooled embedding correlation matrix; applying a multilayer perceptron to the first pooled embedding correlation matrix, thereby producing a first non-normalized correlation matrix; normalizing the first non-normalized correlation matrix using linear normalization, thereby producing a first normalized correlation matrix; concatenating the combined second embedding matrix with the first normalized correlation matrix, thereby generating an enhanced second embedding matrix; applying a linear neural network layer to the enhanced second embedding matrix, thereby generating a combined first attention matrix; and segmenting the combined first attention matrix, thereby producing the one or more first attention matrices.

11. The method of claim 7, wherein applying the second attention layer to the one or more fourth embedding matrices, thereby producing the one or more second attention matrices comprises: combining the one or more fourth embedding matrices, thereby producing a combined fourth embedding matrix; transposing the combined fourth embedding matrix, thereby producing a combined transposed fourth embedding matrix; performing a matrix multiplication on the combined fourth embedding matrix and the combined transposed fourth embedding matrix, thereby producing a second embedding correlation matrix; performing a pooling operation on the second embedding correlation matrix, thereby producing a second pooled embedding correlation matrix; applying a multilayer perceptron to the second pooled embedding correlation matrix, thereby producing a second non-normalized correlation matrix; normalizing the second non-normalized correlation matrix using linear normalization, thereby producing a second normalized correlation matrix; concatenating the combined fourth embedding matrix with the second normalized correlation matrix, thereby generating an enhanced fourth embedding matrix; applying a linear neural network layer to the enhanced fourth embedding matrix, thereby generating a combined second attention matrix; andsegmenting the combined second attention matrix, thereby producing the one or more second attention matrices.

12. The method of claim 1, wherein applying each head sub-model of the plurality' of head sub-models to the embedding matrix further comprises producing a plurality of ranking vectors, and wherein the method further comprises: generating a plurality of ranking lists using the plurality of ranking vectors and an outlier detection matrix.

13. The method of claim 12, wherein each ranking list of the plurality of ranking lists comprises a plurality of rankings, wherein each ranking in a ranking list corresponds to a different time-series outlier detection model of a plurality of time-series outlier detection models, wherein each ranking of each ranking list indicates a relative efficacy of a corresponding time-series outlier detection model at detecting one or more timeseries outliers of a corresponding outlier type in the query segment, and wherein generating the plurality of ranking lists using the plurality of ranking vectors and the outlier detection matrix comprises: performing a matrix multiplication between each ranking vector of the plurality of ranking vectors and the outlier detection matrix, thereby producing the plurality of ranking lists.

14. The method of claim 7, wherein applying each head sub-model of the plurality of head sub-models to the embedding matrix further comprises producing a plurality of ranking vectors, and wherein the method further comprises: generating a plurality of ranking lists using the plurality of ranking vectors and an outlier detection matrix; determining, based on the plurality of outlier segment classifications, a subset of outlier types of the plurality of outlier types; and for each outlier type of the subset of outlier types: evaluating a corresponding ranking list of the plurality of ranking lists, thereby determining one or more best performing time-series outlier detection models of the plurality of time-series outlier detection models based on a plurality of corresponding rankings in that corresponding ranking list, andapplying the one or more best performing time-series outlier detection models to one or more of the following, thereby detecting one or more time-series outliers of the outlier type: the query segment, the one or more first embedding matrices, the one or more second embedding matrices, the one or more first attention matrices. the one or more third embedding matrices, the one or more fourth embedding matrices, the one or more second attention matrices, the embedding matrix, one or more feature vectors of a plurality of feature vectors produced by the computer system by applying each head sub-model of the plurality7of head sub-models to the embedding matrix, and one or more ranking vectors of the plurality of ranking vectors.

15. The method of claim 14, wherein each outlier segment classification comprises a binary7classification, and wherein determining, based on the plurality7of outlier segment classifications, a subset of outlier types of the plurality of outlier ty pes comprises: for each outlier type of the plurality of outlier types, if a corresponding outlier segment classification indicates that the query segment contains one or more outliers of that outlier type, including that outlier type in the subset of outlier ty pes.

16. The method of claim 14, wherein each outlier segment classification comprises a probability7indicating a likelihood that the query segment contains one or more outliers of an outlier type corresponding to that outlier segment classification, and wherein determining, based on the plurality of outlier segment classifications, a subset of outlier ty pes of the plurality of outlier types comprises: for each outlier type of the plurality of outlier types, comparing the probability indicating the likelihood that the query7segment contains one or more outliers of the outlier ty pe to a corresponding probability7threshold, and including that outlier ty pe in the subset of outlier types if the probability exceeds the corresponding probability threshold.

17. A method for training a machine learning model comprising a multilayer sub-model and a plurality of head sub-models to detect outliers in input time-series data, the method comprising performing, by a computer system: receiving a training set comprising a plurality of query training segments and a plurality of baseline training segments; and performing an iterative training process until a terminating condition has been met, the iterative training process comprising a plurality of training rounds, each training round comprising: sampling a baseline training segment from the plurality of baseline training segments, sampling a query training segment from the plurality of query training segments, segmenting the baseline training segment into one or more baseline subsegments, segmenting the query training segment into one or more query subsegments, applying the multilayer sub-model to the one or more baseline subsegments and the one or more query subsegments, thereby producing an embedding matrix, wherein the multilayer sub-model is defined by a set of multilayer sub-model parameters, and wherein the multilayer sub-model includes one or more attention layers, applying each head sub-model of the plurality of head sub-models to the embedding matrix, thereby producing a plurality of outlier segment classifications, wherein each head sub-model of the plurality of head sub-models corresponding to a different outlier type of a plurality of outlier types, wherein each outlier segment classification of the plurality of outlier segment classifications indicates whether the query training segment contains one or more outliers of an outlier type corresponding to that outlier segment classification, and wherein each head sub-model of the plurality of head sub-models is defined by a different set of head sub-model parameters of a plurality of sets of head sub-model parameters, generating one or more loss values based on the plurality of outlier segment classifications,updating the machine learning model by updating the set of multilayer sub-model parameters and the plurality of sets of head sub-model parameters based on the one or more loss values, and performing the iterative training process until the terminating condition has been met.

18. The method of claim 17, wherein: applying the embedding matrix to each head sub-model of the plurality of head sub-models further produces a plurality of ranking vectors; the one or more loss values are generated based on the plurality of ranking vectors in addition to the plurality of outlier segment classifications; the method further comprises performing a matrix multiplication between each ranking vector of the plurality of ranking vectors and an outlier detection matrix, thereby producing a plurality of ranking lists corresponding to the plurality' of outlier types, wherein the outlier detection matrix is defined by a set of outlier detection matrix parameters; updating the machine learning model further comprises updating the set of outlier detection matrix parameters; the multilayer sub-model comprises a first convolutional neural network defined by a set of first convolutional neural network parameters, a second convolutional neural network defined by a set of second convolutional neural network parameters, a third convolutional neural network defined by a set of third convolutional neural network parameters, a first attention layer defined by a set of first attention layer parameters, and a second attention layer defined by a set of second attention layer parameters, wherein the one or more attention layers comprise the first attention layer and the second attention layer, and wherein the set of multilayer sub-model parameters comprise the set of first convolutional neural network parameters, the set of second convolutional neural network parameters, the set of third convolutional neural network parameters, the set of first attention layer parameters, and the set of second attention layer parameters; each head sub-model of the plurality of head sub-models comprises a different convolution layer defined by a different set of convolution layer parameters, a different pooling layer defined by a different set of pooling layer parameters, a different linear neural network layer defined by a different set of linear neural network layer parameters, and a different multilayer perceptron defined by a different set of multilayer perceptron parameters,wherein each set of head sub-model parameters of the plurality of sets of head sub-model parameters comprise the different set of convolution layer parameters, the different set of pooling layer parameters, the different set of linear neural network layer parameters, and the different set of multilayer perceptron parameters; and the set of outlier detection matrix parameters comprise a set of coefficients of the outlier detection matrix.

19. The method of claim 18, wherein: the first attention layer comprises a first attention multilayer perceptron defined by a set of first attention multilayer perceptron parameters and a first attention linear neural network layer defined by a set of first attention layer linear neural network parameters, and wherein the set of first attention layer parameters comprise the set of first attention multilayer perceptron parameters and the set of first attention layer linear neural network parameters; and the second attention layer comprises a second attention multilayer perceptron defined by a set of second attention multilayer perceptron parameters and a second attention linear neural network layer defined by a set of second attention layer linear neural network parameters, and wherein the set of second attention layer parameters comprise the set of second attention multilayer perceptron parameters and the set of second attention layer linear neural network parameters.

20. A computer system comprising: one or more processors: and a non-transitory computer readable medium coupled to the one or more processors, the non-transitory computer readable medium comprising code or instructions, executable by the one or more processors for performing a method of any one of claims 1-19.

Images