Method and system for automatically detecting anomalies in data

Abstract

A method and system for detecting anomalous transition point candidates in performance metadata. The method can be used in computer system performance monitoring. Anomalous candidates are identified that indicate a possible transition to or from an anomalous behavior pattern in a process that generated the performance metadata, e.g., by comparing z-scores on the left and right of various timestamps, and identifying anomalous candidates when the z-scores are significantly different. Anomalous candidates occur individually, not as endpoints of anomalous intervals. For at least one of the anomalous candidates, an explanatory predicate can be generated that indicates a human-readable explanation of the behavior of the process. A set of anomalies can then be filtered, e.g., by removing those that do not have explanatory predicates, or replacing a cluster of anomalies with the most relevant one.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to the prior U.S. Patent Application No. 16 / 511,759, filed July 15, 2019, entitled “Method and System for Automatic Anomaly Detection Data,” which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present invention relates to the field of automatic monitoring of data, such as performance data, and more particularly to a method and system for automatically detecting anomalies in data, such as time series performance metadata. BACKGROUND

[0004] Computer systems, and in particular database management systems, can be monitored in general by collecting and processing data streams received by such systems. The data can be time series data indicative of one or more aspects of the computer system, such as processor and memory usage. For example, the data can be indicative of the proportion of time spent processing user data, system data, and idle time. In addition, the data can be indicative of latency and time spent servicing one or more different interrupts. Such data is an example of performance metadata. The collected data streams can contain anomalies that can be indicative of abnormal conditions. Most of the time, the collected data items have expected values; in some cases, also expected periods and frequencies. Occasionally, however, something unexpected happens, data values can be extremely abnormal, unexpected peaks and valleys occur, or unexpected periods and frequencies occur. Some of these events can be easily explained, but others can require investigation, explanation, and ultimately resolution.

[0005] Other systems, such as computer systems, weather sensors, monitoring devices, Internet of Things (IoT), and stock ticker generators also generate data streams that can contain anomalies. An anomaly can be generally described as an unexpected, out-of-the-ordinary, abnormal, or irregular data value.

[0006] Due to the sheer volume of data and the limited manpower to process it, a significant portion of the automatically collected data is archived, aggregated, or in some cases simply discarded. Domain experts can detect anomalies by observing data values and graphs plotted from them. However, due to the sheer volume of data, domain experts do not have the time to look at all, or even a small fraction of, the relevant data.

[0007] Accordingly, what is needed is a method and system for automatically detecting anomalies, for example, in performance metadata, which alleviates or eliminates one or more limitations of the prior art.

[0008] This background information is presented to reveal information believed by the applicant to be of possible relevance to the present application. No admission is necessarily intended, nor should be construed, that any of the preceding information constitutes prior art against the present application. SUMMARY

[0009] It is an object of embodiments of the present application to provide a method and system for data anomaly detection and anomaly reporting. The data can be performance metadata.

[0010] According to an embodiment of the present application, a method in an electronic device for computer system performance monitoring is provided. The computer system can be a database management system. The method includes obtaining data indicative of usage of or more components of the computer system. The method further includes detecting one or more anomaly transition point candidates in the data. Each of the one or more anomaly transition point candidates is indicative of a possible transition to or from an anomalous behavior pattern in a process by which the computer system generates the metadata. Each of the anomaly transition point candidates is an independent transition point that is not paired with other ones of the anomaly transition point candidates. The method further includes reporting one or more of the anomaly transition point candidates as an anomaly transition point.

[0011] According to an embodiment of the present application, more generally, a method in an electronic device for automatically detecting anomalies in data is provided. The method includes detecting one or more anomaly transition point candidates in the data. Each of the one or more anomaly transition point candidates is indicative of a possible transition to or from an anomalous behavior pattern in a process by which the data is generated. Each of the anomaly transition point candidates is an independent transition point that is not paired with other ones of the anomaly transition point candidates. The method further includes reporting one or more of the anomaly transition point candidates as an anomaly transition point.

[0012] Any of the above methods can further include, for at least one of the anomaly transition point candidates, generating an explanatory predicate indicative of a human-readable explanation of behavior of the process at the at least one of the anomaly transition point candidates.

[0013] In some embodiments, the data includes a first series of values sequentially indexed according to a corresponding series of reference values. In such embodiments, detecting the one or more abnormal transition point candidates includes, for each of a plurality of the reference values, the following operations. First, determining a first mean and a first variance of a portion of the first series of values, the portion of the first series of values corresponding to the reference value, the reference value being within a predetermined interval immediately preceding the one of the plurality of reference values. Second, determining a second mean and a second variance of another portion of the first series of values, the other portion of the first series of values corresponding to the reference value, the reference value being within a predetermined interval immediately following the one of the plurality of reference values. Third, determining, based on the first mean, the second mean, the first variance, and the second variance, a z-score for the one of the plurality of reference values, the z-score indicating a likelihood that the one of the plurality of reference values represents one of the abnormal transition points. Fourth, identifying the one of the plurality of reference values as representing the one of the abnormal transition point candidates based at least in part on the z-score. The terms“first,”“second,”“third,” and“fourth” do not necessarily imply an order of operations. In some embodiments, the above operations can be performed on multidimensional data, and the results combined to determine abnormal transition points.

[0014] According to another embodiment of the invention, there is provided an electronic device for automatically detecting anomalies in data, the electronic device comprising a data interface and electronic data processing hardware. The device is configured to receive the data using the data interface. The device is further configured to detect, using the electronic data processing hardware, one or more abnormal transition point candidates in the data. Each of the one or more abnormal transition point candidates is indicative of a possible transition to or from an abnormal behavior pattern in a process that generated the data. Each of the abnormal transition point candidates is an independent transition point that is not paired with others of the abnormal transition point candidates. The device is further configured to report one or more of the abnormal transition point candidates as an abnormal transition point using the data interface or another data interface or a user interface. The device can be further configured to, for at least one of the abnormal transition point candidates, generate, using the electronic data processing hardware, an explanatory predicate indicative of a human-readable explanation of behavior of the process at the at least one of the abnormal transition point candidates. The electronic data processing hardware can comprise one or both of: a computer processor operatively coupled to a memory; and a preconfigured logic circuit. BRIEF DESCRIPTION OF DRAWINGS

[0015] The features and advantages of the present invention will become apparent from the following detailed description in conjunction with the drawings, in which:

[0016] Figure 1 Anomaly detection system provided by an embodiment of the present application is schematically shown;

[0017] Figure 2 Various aspects of the z-score calculation and left-right z-score (LRZ) convolution process used in embodiments of the present application are exemplarily shown;

[0018] Figure 3 A left-right z-score (LRZ) convolution process for detecting anomalies based on single-dimensional data provided by an embodiment of the present application is shown;

[0019] Figure 4 A LRZ convolution process for detecting anomalies based on multi-dimensional data provided by another embodiment of the present application is shown;

[0020] Figure 5 An explanation generation process performed as part of anomaly detection provided by an embodiment of the present application is shown;

[0021] Figure 6 An anomaly filtering process performed as part of anomaly detection provided by an embodiment of the present application is shown;

[0022] Figure 7 An automatic anomaly detection system provided by a first exemplary embodiment of the present application is shown;

[0023] Figure 8 An automatic anomaly detection system provided by a second exemplary embodiment of the present application is shown;

[0024] Figure 9 An automatic anomaly detection system provided by a third exemplary embodiment of the present application is shown;

[0025] Figure 10A An anomaly detection system provided by a fourth exemplary embodiment of the present application is shown;

[0026] Figure 10B And Figure 10C Output of the anomaly detection system of Figure 10A is shown graphically;

[0027] Figure 11 A system for performing anomaly detection provided by another embodiment of the present application is shown;

[0028] Figure 12 An electronic device provided by another embodiment of the present application is shown.

[0029] It is noted that in all of the figures, like features are marked by like reference numerals. DETAILED DESCRIPTION

[0030] Automated anomaly detection software can be used as a tool to detect anomalies in data such as performance metadata, for example, to assist a human operator. For example, a human operator can analyze automatically detected anomalies before archiving or discarding the underlying data. The anomalies can indicate potential process changes or events of interest that can require further investigation, adjustment, or correction.

[0031] Anomalies can occur due to changes in the underlying system of the data domain. This can be referred to as a process that generates data that goes through or transitions from a pattern of behavior that is abnormal. Some of these changes are normal and expected, but others are not. According to the present invention, indications of anomalies are automatically generated, and then these indications are accompanied by an explanation or further information. This further information can be used to simplify the task of a user (e.g., an operator or domain expert). For example, based on the explanation, a domain expert can choose to investigate selected ones of the anomalies. The explanation can also increase the likelihood of a final resolution of the anomaly, for example, because the explanation provides a starting point for investigation. An important aspect of a final resolution is to find the root cause of the anomaly. The explanation that is automatically provided can be used to facilitate this activity.

[0032] If there are too many indications of anomalies, a domain expert can have no choice but to start ignoring them, especially in cases where many of the anomalies prove to be false positives (normal data values that are incorrectly classified as anomalies). Therefore, according to embodiments of the present invention, the anomaly detection method and system can be used to accept filtering conditions and adjust the number of reported anomalies based on the filtering conditions. Anomalies that are less likely can not be reported, or such anomalies can be reported less frequently than anomalies that are more likely, according to a prioritized list.

[0033] The data that the anomaly detection software operates on can be various types of data. For simplicity, the main data type considered herein is time series data, or data that has an associated timestamp. However, other types of coordinate data can similarly be processed for anomaly detection. The data for each dimension includes a series of values (e.g., indicating a percentage of usage of a resource) that are indexed according to a corresponding series of reference values, such as but not limited to a timestamp (in the case of time series data).

[0034] Unlike the definition of an anomaly having a start point and an end point (e.g. as indicated by two time stamps), embodiments of the invention define an anomaly as a transition point between non-anomalous data and anomalous data. That is, an anomaly is defined by one time point (for time series data) rather than by two end points. In other words, each anomaly transition point is an independent transition point, not paired with other anomaly transition points, to avoid requiring that the transition point be defined as an end point (or other related point) of an interval in which anomalous behavior is deemed to occur. This definition can make automated anomaly detection more manageable, and determination of anomaly transition points easier to handle. As a result, operation of a computer system executing anomaly detection software is improved.

[0035] Embodiments of the invention provide a new perspective on anomalies, which are not, as traditionally done, of a duration including a start point and an end point, but rather consist of one or a set of transition points, each located between a non-anomalous region and an anomalous region. Furthermore, procedures for identifying such anomaly transition points are provided. In particular, as mentioned above, anomaly transition points (and candidates thereof) are identified as independent points, rather than end points of an anomalous interval (e.g. in time) in the input data. The independent points do not need to be paired with other transition points, e.g. as start and end points of an anomalous interval.

[0036] Furthermore, such anomaly definition is more easily accommodating certain types of anomalies, e.g. those having a well-defined start point, but gradually dissipating, and thus possibly not having a well-defined midpoint. The second case is easier to handle, with potentially less computational complexity, in which case the second anomaly occurs while the system is still reacting to the first one. In a third case, the anomaly is actually a transition to a new normal, non-anomalous state.

[0037] Some other methods of anomaly detection on time series data require that the user explicitly labels anomalous and normal regions in order to be able to train a machine learning model to identify such regions. These methods have the drawback that making such a distinction can be very difficult or impossible to achieve, even if possible, and even if possible, can become infeasible due to the sheer size of the data.

[0038] Anomaly interpretation is considered a new topic. Interpretation generation is becoming more and more important in today’s world, as users are faced with a sea of data that they simply do not have time to process all of. Automatically pre-processing the data allows users to more appropriately focus their attention.

[0039] Some embodiments of the invention filter anomalies based at least in part on how close to each other the anomalies are, rather than (or in addition to) filtering anomalies by assigning scores to the same anomalies and filtering those anomalies whose scores are below a certain domain-specific threshold.

[0040] Embodiments of the invention provide a method and system for automatically detecting anomalies in data, such as time series data. In some embodiments, the method and system also provide annotations or explanatory information (explanatory predicates) for the anomalies. In some embodiments, the method and system also provide a mechanism for filtering some of the automatically detected anomalies, such as by removing less important anomalies. The filtering can be based on statistical scores.

[0041] In some embodiments, the anomaly detection method and system are applied to a specific application domain, such as computer system performance monitoring, or more specifically, database management system performance monitoring. This can be considered as computer system performance metadata monitoring. When a computer runs an application, such as a database management system or an operating system (OS), the computer can generate performance metadata. The performance metadata indicates computer performance aspects, such as processor usage, I / O requests, memory usage, number of interrupts, etc. An anomaly in the performance metadata provides an indication of computer performance anomaly when running the application or OS. In other embodiments, the anomaly detection method and system are applied to one or more different application domains, such as but not limited to processing of financial data, weather data, sensor data, public or private health data, social or political data, scientific data, geological data, communication network data, etc. The output of the anomaly detection method or system can be provided in the form of a set of automatically detected anomaly turning points, which can be sent to domain experts for further consideration or possible actions. For each anomaly turning point, the output can also include a set of one or more explanatory predicates. The output can include computer generated text, graphics, or both. The explanatory predicates can provide information on the cause of the anomaly, or information pointing to the root cause. The explanatory predicates can be ordered based on potential z-scores, so that the most likely discriminators are listed more prominently.

[0042] Figure 1 An anomaly detection system 100 provided by an embodiment of the invention is schematically shown. The system 100 includes a number of modules, including an anomaly detector 110, an explanation generator 120, and an anomaly filter 130. In some embodiments, the anomaly filter can be omitted.

[0043] In some embodiments, the anomaly detection system 100 is implemented using software running on a computer device. Such a software-based system includes computer-readable instructions that, when executed by one or more processors of a computer (e.g., a physical stand-alone computer or a virtual machine), perform a method for detecting anomalies in data. The system can also include a computer device. In addition, a computer device is also provided for implementing the anomaly detection system described above. The system can include one or more functional modules for performing different actions described herein, such as anomaly detection and explanation generation.

[0044] In some embodiments, the anomaly detection system 100 is a system-on-a-chip, ASIC, or FGPA that includes up to three different hardware components, namely the anomaly detector 110, the explanation generator 120, and optionally the anomaly filter 130. Alternatively, the anomaly detection system 100 can include up to three different system-on-a-chip, ASICs, or FGPA, each of the anomaly detector 110, the explanation generator 120, and the anomaly filter 130 is included, which is controlled by a controller (not shown) of the anomaly detection system 100.

[0045] The anomaly detector 110 automatically detects candidate transition points from normal regions to anomalous regions, rather than detecting anomalous regions in data, and vice versa. The detection can be performed using a process described herein as the "Left-window, Right-window z-score (LRZ) convolution" (hereinafter referred to as the "LRZ convolution process"). The anomaly detector receives data, such as time series data (one or more time series), produces K candidates of anomalies, also referred to herein as anomaly transition point candidates. If multiple time series data are provided, they should be aligned. K is typically a variable and depends on the data values and their variations. However, a certain value or range of K can be determined by adjusting operational parameters until at least approximately the desired value or range is reached.

[0046] The explanation generator 120 receives K anomaly transition point candidates from the anomaly detector 110. For each anomaly transition point candidate, the explanation generator is used to provide an ordered set of possible explanations, such as provided as explanatory predicates. The set of explanations can be ordered in a sequence in which the more likely explanations (e.g., determined by z-score or other statistical methods) are preferred over the less likely explanations.

[0047] The anomaly filter 130 receives K anomaly transition point candidates from the anomaly detector 110 and their associated explanations (e.g., a set of explanatory predicates) from the explanation generator 120 as inputs. The anomaly filter 130 is used to systematically filter out some of the anomaly transition point candidates such that K-t confirmed anomaly transition points (and their associated predicate sets) remain. In some cases, no filtering is performed if none of the transition point candidates meet the filtering criteria. The resulting data can then be stored or sent to domain experts for possible actions to be taken. In addition, the data can also be sent to another automated process for automated actions to be taken. The filtering can correspond to a process of retaining some of the anomaly transition point candidates for possible reporting as anomaly transition points and removing other anomaly transition point candidates.

[0048] According to various embodiments of the present invention, the automated anomaly detection method and system operate using a statistical measure called z-score. For example, some or all of the anomaly detector 110, the explanation generator 120, and the anomaly filter 130 can operate using z-scores.

[0049] According to a common definition, a z-score of a data item is a measure of how different that item is from the set of items it belongs to. There is also a second, less common definition of z-score, which is the definition primarily used in the present invention. Consider two sets of data items, each data item having its own mean value. According to this second definition, one can also compute a z-score of the difference between the two means, which is a measure of how different the two sets of data are. For clarity, the term "mean difference z-score" can be used to explicitly refer to the second definition. Although the term "mean value" is used here, other definitions of mean value can potentially be used instead of mean value, such as median or mode.

[0050] Accordingly, the mean difference z-score can indicate the likelihood that a certain reference value or timestamp represents an anomaly transition point. In other words, the mean difference z-score can indicate the likelihood that the data in a predetermined interval immediately preceding the reference value (timestamp) is different from the process behavior pattern that caused the data in a predetermined interval immediately following the reference value (timestamp).

[0051] If a high mean difference z-score, e.g., 5 or above, is observed for two sets of compared data, statistical techniques allow one to conclude that the two sets of data (e.g., two portions of time series data) are significantly different: one set is anomalous and the other is non-anomalous. It is important to note that even if it is not certain which set of data is anomalous, a transition point representing from or to the anomalous set of data is still identified. This technique provides the basis for automated anomaly detection for embodiments of the present invention.

[0052] In various embodiments, the same (or at least comparable) mean difference z-score defines a set of explanatory predicates used to generate for an observed anomaly transition point (or candidate). Each anomaly transition point can be accompanied by one or more such predicates. Each predicate can belong to a different data attribute. These different attributes can be referred to as different dimensions or different dimensions in a multi-series value. For example, for weather station data, the data attributes can include one or more of the following: temperature, pressure, precipitation, humidity, and cloud cover. As another example, for computer processor performance data, the data attributes can include one or more of the following: time spent processing user data, time spent processing system data, processor idle time, processor wait time, time spent on service interrupts, and time spent on service softirqs. Such computer processor performance data can be obtained, for example, by using the Linux "dstat" command.

[0053] In various embodiments, the same (or at least comparable) mean difference z-score defines a filter for less important anomalies. In particular, in some embodiments, if no explanatory predicate can be generated for an anomaly transition point candidate, the anomaly transition point candidate is considered unimportant and is not reported, e.g., by removing the anomaly transition point candidate from a set of anomaly transition point candidates for possible subsequent reporting. In addition, the mean difference z-score can also be used to select the most relevant anomaly transition point candidate from a set of anomaly transition point candidates that are proximate in location (e.g., in time). In particular, according to some embodiments, the anomaly transition point candidate with the highest z-score is selected from a set of anomaly transition point candidates that are proximate in location as the representative of the set. This anomaly transition point candidate can be retained for possible reporting as an anomaly transition point, while the other candidates in the set can be removed as possible anomaly transition points.

[0054] Figure 2 Various aspects of the LRZ convolution process used in various embodiments of the present application are illustratively shown. In particular, anomaly detector 110 uses an LRZ convolution operation. The LRZ convolution process is so named because it uses a left convolution window and a right convolution window, while using a mean difference z-score. The LRZ convolution process refers to the practice of computing a mean difference z-score over multiple (possibly all) reference values or timestamps.

[0055] Figure 2A plot of time series data 205 is shown with time values (time stamps) on the X-axis and values on the Y-axis. In the present example, the time series data 205 represents the percentage of computer processor (CPU) usage within a computer system serving a user "usr" process. In addition, a smoothed (e.g., Kalman filtered) version 210 of the time series data is also shown. A particular reference time point 215 (more generally, also referred to as a reference value) is shown using a vertical line. A window 220 of constant width W to the left of point 215 is referred to as the "left convolution window" (abbreviated as L), while a window 225 of the same width W is referred to as the "right convolution window" (abbreviated as R). In various embodiments, these windows can be of the same width or different widths from each other. In some embodiments, for definiteness, time point 215 can be considered to belong to the right window 225. The following four statistical indicators are defined over the L and R windows. The average of a portion of the time series data 205 that lies within the L window 220 is denoted as μ L , and the variance of this data is denoted as σ 2 L The average of a portion of the time series data 205 that lies within the R window 225 is denoted as μ R , and the variance of this data is denoted as σ 2 R .

[0056] In colloquial terms, the average is the mathematical average of the values. Typically, the square root of the variance is referred to as the standard deviation (σ L or σ R ), which is often used as a measure of the difference between the values. The smaller the standard deviation, the smaller the difference between the values (i.e., the closer the distance). The larger the standard deviation, the smaller the difference between the values (i.e., the farther the distance).

[0057] In various embodiments, the z-score of the mean difference |μ L - μ R | is defined according to the following equation:

[0058]

[0059] The observations under both L and R are considered to be a statistical sample. In some embodiments, according to standard statistical analysis techniques, the sample size (in this case, the window width W) can need to be at least 30 for the Central Limit Theorem (CLT) to be applied adequately. Smaller sample sizes are available, but do not necessarily yield the best results. The z-score is in units of standard deviation. A high z-score (such as 5 or higher) indicates a high probability that the samples in L and R came from different underlying populations. The higher the z-score, the higher the probability that the samples came from different underlying populations. Conversely, a low z-score (such as close to 0) indicates a high probability that the samples in L and R came from the same underlying population, and that the current timestamp can not be a transition point. Different underlying populations can refer to data generated due to the data generation process being in different behavioral modes. The term “timestamp” can refer to Figure 2 a reference point in time 215.

[0060] More accurately, |μ L –μ R | values can be considered to be normally distributed around 0. Thus:

[0061] When z-score = 1, it can be said that 68% of |μ L –μ R | values are within 1 standard deviation of |μ L –μ R |.

[0062] When z-score = 2, it can be said that 95% of values are within 2 standard deviations.

[0063] When z-score = 3, it can be said that 99.7% of values are within 3 standard deviations.

[0064] When z-score = 4, it can be said that 99.994% of values are within 4 standard deviations.

[0065] When z-score = 5, it can be said that 99.9999% of values are within 5 standard deviations.

[0066] It is important to note that the number of standard deviations that the indicated percentage of values falls within is directly represented by the z-score. Thus, a high z-score for a particular timestamp means that the observed mean difference value |μ L –μ R | correspondingly occurs rarely. In other words, a high z-score invalidates the null hypothesis that the samples in L and R came from the same underlying population (generated by the same behavioral mode). This means that the samples in L and R came from different underlying populations (or were produced by different behavioral modes, one being anomalous and the other being non-anomalous).

[0067] To identify anomalous transition points, |μL - μ R A high z-score of | μ L - μ R | implies that the underlying generating process of the signal changes substantially for the timestamp value at the current center of the L and R windows, so essentially the samples in L and R can be considered as different signals. This is the reason why the average of L and R is significantly separated. From this it is inferred that L is the abnormal region and R is not, and vice versa. Optionally, additional tests can be applied to determine which of L and R is likely to be the abnormal region. Further, it is inferred that if the z-score of the mean difference is high, the considered timestamp is a candidate for an abnormal transition point. By performing the LRZ convolution process, i.e. by testing a large number of timestamp values, the transition location can be identified more precisely.

[0068] It is noted that difficulties arise when trying to push the convolution windows to the leftmost or rightmost side of the finite sample set. This is because nothing is observed to the left of the leftmost point or to the right of the rightmost point. However, when the window width W is small compared to the signal as a whole, it is possible to determine abnormal transition point candidates for the main part of the time series. In some embodiments, the observations can be "extended" to the leftmost and rightmost side using fictitious values computed by means of some extrapolation technique. In this case, it is possible to determine abnormalities in the entire signal.

[0069] In some embodiments, the L convolution window and the R convolution window can have different widths W L and W R , respectively. In this case, alternatively, the z-score of the mean difference | μ L - μ R | is defined according to the following equation:

[0070]

[0071] Using different L window widths and R window widths allows to adjust the abnormality detection behavior, e.g. if W L > W R (e.g. W L = 2 W R ), there are more historical samples available as a basis for the decision whether an abnormality has occurred, which tends to result in fewer abnormalities. Typically, it can be desirable that both W L and W R have at least 30 samples.

[0072] Having described the basis of the z-score above, the LRZ convolution process can now be described. The LRZ convolution process receives a time series value of a particular statistic as input (or receives multiple such time series values of different statistics). The LRZ convolution process provides as output zero or more timestamps at which an abnormality is placed (abnormal transition points).

[0073] Figure 3 An LRZ convolution process 300 provided by an embodiment of the present application is shown. In step 310, a current position of a timestamp, for example located at the center of the L convolution window and the R convolution window (more generally, a reference value), is initialized to a value near the beginning of the time series data. In step 320, for the current position of the timestamp, a z-score of the mean difference |μ L - μ R is determined by processing data in the L window and the R window to the left (immediately before) and to the right (immediately after) of the timestamp. In step 330, it is determined whether the z-score of step 320 is greater than a predetermined threshold. If so, the current timestamp is flagged 335 as an anomaly transition point candidate, for example by storing the current timestamp in such a candidate list. In some embodiments, the predetermined threshold can be 5, 10, or another suitable value suitable for the type of input data and the goal of the anomaly detection process. In step 340, the current position of the timestamp is updated (e.g., incremented), and steps 320-340 are then repeated. Steps 320-340 can be repeated until a stopping condition is reached, for example until the timestamp reaches a predetermined value near the end of the time series data. Updating the current position of the timestamp can involve incrementing the current position by one unit or by a predetermined multiple of the unit, where such a unit represents a time interval between consecutive instances of the time series data.

[0074] Figure 3 A simplified process is shown, involving the case where only one dimension of the time series data is analyzed. Figure 4 An LRZ convolution process 400 provided by another embodiment of the present application is shown, in particular a convolution process that analyzes multiple dimensions of the time series data simultaneously. Each dimension of the time series data can correspond to a sequence of timestamp values, or in other words, each dimension can correspond to a different series of values indexed according to the same series of reference values (timestamps). In step 410, the current position of the timestamp is initialized, similar to step 310. In step 420a, for the current position of the timestamp, and for the time series data of the first dimension, a z-score of the mean difference |μ L - μ R is determined by processing data in the L window and the R window to the left and to the right of the timestamp. In step 430a, similar to step 330, it is determined whether the z-score of step 420a is greater than a predetermined threshold. If so, the current timestamp is flagged as an anomaly transition point candidate for the time series data of the first dimension.

[0075] In step 420b, for the current position of the timestamp, and for the time series data of the second dimension, a z-score of the mean difference |μ L - μ Rthe z-score of step 420b. In step 430b, similar to step 330, it is determined whether the z-score of step 420b is greater than a predetermined threshold. If so, the current timestamp is marked as an anomaly transition point candidate for the second dimension of time series data.

[0076] For each additional dimension of time series data to be processed, similar steps 420c, 430c, 420d, 430d, etc. (not shown) can be performed.

[0077] In step 440, for the current timestamp, the number of anomaly transition point candidates resulting from steps 430a, 430b, etc. is counted. It is determined whether the count from step 440 exceeds a predetermined threshold. If so, the timestamp is marked as an anomaly transition point candidate for the entire multi-dimensional time series data. Thus, an anomaly transition point candidate can be determined based on multiple z-scores.

[0078] In step 440, the LRZ convolution process 400 confirms that a particular timestamp is an anomaly transition point candidate because multiple statistics declare the timestamp as an anomaly transition point candidate. In many applications, the collected statistics are moderately to highly correlated, and the anomaly is manifested in multiple statistics at the same time. Of course, if the various statistics in a particular application domain are highly uncorrelated, the “#-of-candidates-threshold” can be set to 1.

[0079] In step 450, the current position of the timestamp is updated (e.g., incremented), and then steps 420a, 430a, 420b, 430b, etc. and steps 440 through 450 are repeated, similar to step 340. This step can be repeated until a stopping condition is reached.

[0080] It is noted that steps 420a, 430a, 420b, 430b, 440, etc. can be performed in different orders. In some embodiments, instead of processing each dimension of time series data and then updating the timestamp, each dimension of time series data can be processed completely separately at all timestamp values, and the results are later combined or processed together.

[0081] In some embodiments, an anomaly transition point or candidate is declared when the average z-score of all statistics is greater than or equal to a threshold. If the average z-score is employed, the threshold can be lower than if an absolute z-score threshold is used. This is because the z-score in certain dimensions that do not exhibit an anomaly can be close to 0 or 1, while the z-score in other dimensions that exhibit an anomaly can be 5 or higher. It is noted that the expected value of the z-score in a region where there is no transition point is 0. This results in an anomaly being declared only when there is a significant change in the underlying data.

[0082] In various embodiments, and restated, the LRZ convolution process processes time series data in terms of time stamps. The time series can have one or more attributes called "statistics". For each processed time stamp value, two fixed-width windows are defined, called "left window" and "right window" (abbreviated as L and R). The mean and variance values of the data points falling under the L window and R window are computed, and from these values, the z-score of the difference between the means is computed. The z-score represents the difference of the values belonging to the L window and R window. If the z-score at a time stamp is large enough to simultaneously make several statistics, then an overall anomaly transition point candidate at that time stamp is identified. The z-score is a known concept in statistics, while the two-dimensional convolution window is well known in digital image processing. The z-score can be defined differently in two different cases, namely for the values in a sequence of values, and for the difference of the means of two sequences. According to embodiments of the present invention, as mentioned above, the definition of the z-score is the second case definition. Also, according to embodiments, it is noted that in the particular processing operation called LRZ convolution process, the z-score is combined with two one-dimensional convolution windows, namely the L window and the R window.

[0083] As mentioned above, according to various embodiments, the explanation generator 120 receives the output from the anomaly detector 110 and generates explanatory predicates for at least some and possibly all of the provided anomaly transition point candidates. The explanation generator receives as input a set of anomaly transition point candidates generated by the anomaly detector, and provides as output a set of explanatory predicates for some or all of the anomaly transition point candidates. Alternatively, the explanation generator can be seen as providing a set of explanatory predicates for all anomaly transition point candidates, but it is noted that the set for one or more anomaly transition point candidates can be empty, as it can contain zero explanatory predicates.

[0084] In conjunction Figure 2 The operation of the explanation generator can be understood again. Figure 5 An explanation generation process 500 provided according to some embodiments is shown. According to this process, in a first step 510, for each anomaly transition point candidate (e.g. at time point 215), the percent change of the mean between L and R is determined as If the percent change is positive, then the mean of the data being processed has decreased at the transition point candidate. If the percent change is negative, then the mean of the data being processed has increased at the transition point candidate. The data being processed refers to the data in the L window and R window. This data is also called statistics. The name of the statistics is denoted with "s".

[0085] In a second step 520, again for each anomaly transition point candidate, one of two explanatory predicates is generated appropriately: "s decrease ", and "s increase In various embodiments, the predicate is generated only if the corresponding z-score exceeds a predetermined threshold. The threshold can be at least 6, for example, it can be around 10. Moreover, the threshold can also be a (potentially adjustable) parameter of the explanation generation process, which value can (but need not) be the same as the threshold used by the LRZ convolution process. A low threshold generates more predicates; a high threshold generates fewer predicates. After all predicates (each corresponding to a different statistic) are generated, they can be ordered by decreasing z-score. Predicates located at the beginning of the list have higher z-scores and stronger explanatory power than predicates located at the end of the list. As described above, steps 510 and 520 are repeated 530 for each anomaly transition point candidate.

[0086] In addition, other types of explanatory predicates can also be provided. A predicate can generally contain a verb and provide information (e.g., quantitative or qualitative) about a subject. Multiple observable objects or predicates can be combined together to generate a new predicate. Predicate sets can be combined together to generate higher-level predicates. For example, if the disk usage, processor usage, and memory usage all increase by a predetermined amount, a predicate can be generated indicating that the monitored computer system is under stress. In some embodiments, the predicates can be provided (e.g., pipelined) to automated actions, in the sense that if a particular predicate is generated, a specified action can be automatically taken.

[0087] As described above, according to various embodiments, the anomaly filter 130 receives as its input the output from the anomaly detector 110 (set of anomaly transition point candidates) as well as the output from the explanation generator 120 (explanatory predicates). The anomaly filter then generates a potentially reduced set of anomaly transition points. In some embodiments, the anomaly filter removes an anomaly transition point candidate from the set as long as such anomaly transition point candidate is not accompanied by any explanatory predicate. This can happen, for example, when the z-score associated with the anomaly transition point candidate is less than a predetermined threshold. The output of the anomaly filter 130 is a set of anomaly transition points and their associated explanatory predicates.

[0088] In various embodiments, and restated, the present embodiments provide for anomaly explanation (explanatory predicates) generation and ordering using z-scores. For each anomaly transition point candidate identified by the anomaly detector, a set of explanatory predicates is generated. Each predicate in the set is for a different statistic, and is based on the z-score of the difference between the mean of that statistic in the "left" window and the "right" window of that transition point candidate. If the z-score is higher than a threshold, a predicate is generated, for example, in the form of "usr is 280% higher than threshold" or "sys is 80% lower than threshold". The set of explanatory predicates is itself ordered by decreasing z-score, so that predicates with stronger explanatory power appear before predicates with weaker explanatory power.

[0089] Figure 6 An anomaly filtering process 600 is shown provided in accordance with some embodiments. In step 610, for each provided anomaly transition point candidate, it is determined whether there are any explanatory predicates associated with the anomaly transition point candidate. If there are no such explanatory predicates, the anomaly transition point candidate is deleted 615 from the set.

[0090] In step 620, additional anomaly transition point candidates are processed in order of increasing timestamp value, as follows. For the current anomaly transition point candidate, it is determined 622 whether there is a prior (i.e., having an earlier timestamp) anomaly transition point candidate within a fixed time interval of a predetermined width W. If there is, a comparison 624 is made between the average z-score of the current anomaly transition point candidate and the average z-score of the prior anomaly transition point candidate. The average z-score of an anomaly transition point candidate is the average z-score of all explanatory predicates of the anomaly transition point candidate. If the current anomaly transition point candidate has a higher average z-score, the prior anomaly transition point candidate is replaced 626 by the current anomaly transition point candidate. If the prior anomaly transition point has a higher average z-score, the current anomaly transition point candidate is deleted 628.

[0091] The rationale behind this filtering step 620 is as follows. Sometimes, due to a non-anomalous region ending and an anomalous region beginning (or vice versa), many adjacent anomalies will be reported. In this case, many adjacent or close-in time stamps qualify for reporting as anomaly transition points. After this filtering step, of such close-in time stamps, only the one with the highest average z-score survives. That is, alternatively, the filtering step 620 can be specified as: from a plurality of anomaly transition point candidates within a given interval, select a single anomaly transition point having the highest z-score of a plurality of anomaly transition points, and delete all of the plurality of anomaly transition points except the single anomaly transition point. Another effect of this filtering step is that no more than one anomaly transition point is reported for a window of width W. Alternatively, a reduced set of anomaly transition points within a given interval can be retained, e.g., two or two but not all. The data of multiple anomalies within the interval can be combined, and the accompanying explanatory predicates can be combined based on such combined data. In such a case, the explanation generation system can be re-used.

[0092] The filtering step 620 reduces the number of reported anomalies, but can miss two closely related but distinct anomalies. A simplifying assumption can be made that two distinct anomalies located within W time units of each other are in fact the same anomaly, and between them, the one with the higher average z-score will be retained.

[0093] In various embodiments, the anomaly filter is an optional subsystem. If omitted or disabled, the output format of the overall procedure remains unchanged. However, more anomalies will be reported after the filter is turned off. In one embodiment, the filter can be enabled when an anomaly in the field of computer system performance monitoring is detected.

[0094] In various embodiments, and restated, the present embodiments provide filtering of potential anomalies using interpretive predicates. Interpretive predicates can be used to filter unimportant anomalies or false positive anomalies using z-scores.

[0095] Filtering can be done in up to three cases, and up to three different corresponding senses. First, during the explanation generation phase, if the z-score of a statistic is below a threshold, the corresponding interpretive predicate is not generated because the explanatory power of that statistic is deemed low. Second, if no predicate can be generated for an anomaly transition point candidate because the z-score of all statistics is below a threshold, the anomaly transition point candidate is not reported at all. This reduces the number of reported anomaly transition points, which can be useful in cases where too many anomaly transition points are reported. Third, multiple adjacent anomaly transition point candidates are replaced with a single anomaly transition point where the average z-score is highest.

[0096] Figure 7 An automated anomaly detection system provided by a first exemplary embodiment of the present invention is shown. The system is capable of detecting and interpreting anomalies discovered using a computer system. The system can run on a computer system different from the monitored computer system, or on the same computer system as the monitored computer system. Arrow directions indicate data flow directions. Computer system 710 can be a standalone, networked, or cloud-based system. The computer system can be part of a database system. A statistics collection module 715 runs on the computer system and collects a predefined set of statistics. A timestamp or sequence of timestamps is associated with each such set. A particular instantiation of module 715 is the Linux "dstat" command, but it can also be a custom program or script. The "dstat" example invocation can collect CPU, virtual memory, IO, and interrupt data from the underlying computer system.

[0097] The statistics collection module 715 creates tabular data and can optionally store it in a file. The data format can be flexible, with two possible formats being CSV (comma-separated values) files and XML. A total of N independent statistics are collected, each being time series data.

[0098] The N statistics are processed by N corresponding filters 720 (e.g., Kalman filters), each of which smooths a different statistic. Alternatively, fewer than N filters can be provided, at least some of which can smooth two or more statistics. Thus, the data processed during anomaly detection can be a filtered version of the original data. Alternatively, for example, as shown in Figure 8 the data can be raw, unfiltered data. The purpose of smoothing is to eliminate short-term spikes and to emphasize or enhance long-term trends in the data. As a result of this filtering, N smoothed statistics will be created. One such smoothed statistic is shown in Figure 2 as data 210. As described above in connection with Figure 1 , the N smoothed statistics are processed by the anomaly detection system 100, e.g., each statistic is time series data. The resulting anomaly transition points and associated explanations can be provided to a domain expert. If the computer system is running a database management system, the domain expert can be a database administrator.

[0099] Some potential advantages and features of the first exemplary embodiment are as follows. First, this embodiment can be used in application domains where the statistics contain noise. If not careful, noise can be mistaken for an anomaly. Smoothing will eliminate such noise, and the resulting system will be less likely to produce false positives (noise that is flagged as an anomaly). Sometimes, a computer system will generate hundreds of statistics, but they can be categorized. For example, if a computer has 8 cores or CPUs, statistics such as "usr," "sys," "idl," "wai," "hiq," and "siq" can be obtained for each core, for a total of 48 statistics that indicate CPU operations. It can be useful to aggregate and average these data, which reduces the statistics to 6, one for each category. A "statistic collection" module can be used for such data cleaning and aggregation. The rest of the system is not affected by the content and degree of preprocessing of the original statistics. As described above, anomaly explanations are provided in order of decreasing z-score. Thus, predicates that are located in the beginning portion of the ordering have stronger explanations than predicates that are located in the end portion.

[0100] In some embodiments, the anomalies themselves can be ordered as follows. For each anomaly transition point, the average z-score of the explanation is computed, and the anomalies are ordered in descending order according to such average z-scores. Anomalies that are located in the beginning portion of the list can be more important than anomalies that are located in the end portion of the list, because the former show more extreme deviations of the data and are thus more salient. A human domain expert can review a limited number of the most important anomalies (e.g., the top three) at each observation period (e.g., each day).

[0101] Figure 8An automatic anomaly detection system provided by a second exemplary embodiment of the invention is shown. This can be considered a simplified version of the first exemplary embodiment described above, and Figure 8 the components in Figure 7 are substantially the same as their counterparts in Figure 7 . Most notably, in this embodiment, the smoothing module (implemented using the N Unscented Kalman Filters 720 in ) is omitted. Instead, the N statistics are processed directly by the anomaly detection system 100 without smoothing.

[0102] The second exemplary embodiment provides an instance where smoothing or processing of the signal is not required. In the inventor’s preliminary testing, both the first and second embodiments reported nearly identical sets of anomaly transition points.

[0103] Some potential advantages and features of the second exemplary embodiment are as follows. This embodiment can be preferred in application domains where the fidelity of the statistical signal must be maintained, or in application domains where a suitable smoothing algorithm has not yet been developed. This embodiment can be preferred in application domains where the statistical data contains short duration or “spike” (i.e., with high frequency signal components) anomalies. Smoothing can erroneously remove such anomalies. Finally, the second exemplary embodiment is simpler due to the lack of filtering.

[0104] Figure 9 An automatic anomaly detection system 900 provided by a third exemplary embodiment of the invention is shown. This can be considered a variation of the second exemplary embodiment described above, and Figure 9 the components in Figure 7 and Figure 8 are substantially the same as their counterparts in . It is noted that filters (e.g., filters 720) can also be included in the third exemplary embodiment.

[0105] The third exemplary embodiment relates to the scalability of the automatic anomaly detection system. In particular, in this embodiment, Support Vector Machines (SVMs) are used to detect anomalies. Thus, it can generally be considered that embodiments of the invention can be provided where the LRZ convolution operation is replaced with another operation for detecting anomalies. This other operation can be, but is not necessarily limited to, an SVM that has been trained to learn to detect anomalies in data.

[0106] According to some embodiments, the SVM operation for anomaly detection is as follows. Note that the description here describes configuring and training SVMs for anomaly detection. Those skilled in the art will readily understand other details of SVMs. In one embodiment, the anomaly detector 910 includes N one-class SVMs, one for each of the N statistics collected by the statistics collection module 715. One-class SVMs can be used because it is only necessary to classify observations as belonging to the "normal" class or not (in which case the observation is anomalous). For example, the observations can be data points at a single time of a time series data, i.e., a numerical indication of the system state at a particular time. Each SVM is trained (or fitted) using window data to the left of the timestamp (e.g., a window of 30 to 50 observations). Each SVM is used to classify whether the next observation is anomalous. If a large number of SVMs classify the next observation as anomalous, then an anomaly is declared. Note that because the width of the anomaly classified by the SVMs is one time unit, these anomalies can be passed as transition points to the explanation generator subsystem 920. In addition, an anomaly filter 930 similar to the anomaly filter 130 of Figure 1

[0107] In some embodiments, if the anomaly detector 110, 910 implements another anomaly detection procedure (e.g., different from the LRZ convolution procedure or SVMs that use "start" and "end" timestamps to identify anomalies), both the start and end points can be included as anomalous transition points in the data provided to the explanation generator subsystem 920. In other embodiments of the application, the anomaly detector 110, 910 can include other various types of classifiers in addition to SVMs, which can be based on machine learning, for example. These classifiers are trained to learn to classify whether the next observation is anomalous.

[0108] Some potential advantages and features of the third example embodiment are as follows. First, it provides extensibility of the automatic anomaly detection system, independent of the LRZ convolution procedure. Second, if domain experts are already accustomed to using an existing anomaly detection program, this existing program can be implemented in the anomaly detector 110, 910, while still benefiting from the explanation generator 120, 920 and (optionally) the anomaly filter 130, 930 of the automatic anomaly detection system 100, 900. Third, it has been observed that SVMs are generally able to detect more anomalies than the LRZ convolution procedure. Further considering this, the anomaly filter 130, 930 was found to be particularly effective in eliminating many anomalies that did not have an explanatory predicate in testing. This can have the advantage of removing false positive detected events.

[0109] Figure 10A to Figure 10C A fourth example embodiment of the application is shown.​Figure 10A A data capture and analysis system is shown that includes two computer systems. A first computer system 1010 runs a database management system (DBMS) on which performance data is captured. According to the present invention, a second computer system 1020 performs data analysis, anomaly detection, anomaly filtering, and explanation generation.

[0110] In this embodiment, two types of data are being captured. The first type is operating system data 1012 (e.g., captured every second). The second type of data is DBMS data 1014 (e.g., captured every 5 seconds). Both of these data types are time series data. The first computer system 1010 provides both of these data types to the second computer system 1020 for processing. The second computer produces an output 1016. Figure 10B and Figure 10C Two sample graphs indicating the output 1016 of the second computer system 1020 performing data analysis and anomaly detection are provided in FIGS. 1 and 2.

[0111] Figure 10B Six statistics are shown, called usr, sys, wai, minpf, int, and csw. These statistics indicate different computer system processes to which a percentage of computing (e.g., processor) resources is attributed. The detected anomaly transition point 1040 is shown using a long arrow. The explanation (explanatory predicate) 1045 for the anomaly is also shown in the box. There are 5 explanatory predicates. For example, the first explanatory predicate states that when the point anomaly was detected, the CPU usage in the "usr" mode dropped by 84% from the previous.

[0112] Figure 10C Four additional statistics are shown, called read, writ, in, and alloc. These statistics indicate different computer system processes to which a percentage of computing (e.g., processor) resources is attributed. In addition, this graph also shows the anomaly transition point 1040 and the explanation (explanatory predicate) 1045. In some embodiments, Figure 10B and Figure 10C All of the statistics of FIGS. 1 and 2 can be shown on the same graph in the user interface. In some embodiments, only the explanatory predicates related to the statistics shown on a particular display graph are shown in the user interface. In some embodiments, explanatory predicates related to certain statistics are shown on the displayed graph even though those statistics are not shown on the graph.

[0113] According to embodiments of the present application, as described above, automatic anomaly detection can be applied in the field of computer system performance monitoring or database management system performance monitoring or both (e.g., performance monitoring of a computer system running a database management system). According to other embodiments of the present application, automatic anomaly detection can be applied in another field, such as in the field of analysis of time series data or other types of data. Examples of such fields include, but are not necessarily limited to, weather sensor systems; devices that electronically communicate with each other, such as Internet of Things (IoT) devices; stock market data; and data collected from instruments used in the healthcare industry.

[0114] It is noted that the present application can be used in specific application fields, such as by adjusting thresholds and behaviors. Different application fields can benefit from different anomaly detection thresholds, as the definition of an anomaly can be quite different in one application field than in another. For example, in the healthcare industry, a relatively small change in body temperature can be considered an abnormal and worrisome anomaly, while a similar magnitude change in stock price can be considered normal. As is clear from the above discussion, the subsystems of the automatic anomaly detection method and system have several parameters that allow the practitioner to fine-tune a particular embodiment to suit the solution field.

[0115] For anomaly detection using the LRZ convolution process, variable parameters include the convolution window width, the number of anomaly statistics (i.e., the number of observations or dimensions that need to be simultaneously anomalous before a candidate transition point is declared anomalous), and the z-score threshold. Each of a larger convolution window width, a larger number of anomaly statistics, and a larger z-score threshold generally results in a smaller number of detected anomalies. For explanation generation, the variable parameter includes the z-score threshold. The value of this threshold can be different from the value used for anomaly detection. A lower threshold generally results in a larger number of explanatory predicates being generated. For anomaly filter, the variable parameter includes the window width used to determine anomaly proximity. This width can be the same as the width W used for the LRZ convolution process, or it can be different. The larger the window width, the fewer the number of detected anomalies.

[0116] Some or all of the parameter values can be set based on some or all of the following information: the number of reported anomalies that an expert user can handle per day; the number of false positive anomalies that are reported; the number of false positive anomalies (missed anomalies); and the ability of the computing platform to perform the anomaly detection task in real time in the case of streaming data.

[0117] Various embodiments of the invention utilize real and / or virtual computer resources. Such computer resources utilize, at a hardware level, a set of one or more microprocessors operably coupled to a corresponding set of memory components, including storage program instructions for execution by the microprocessors. The computing resources can be used to provide virtual computing resources at one or more virtualization levels. For example, one or more virtual machines can be provided using one or more given general-purpose computer hardware platforms. The computer hardware, such as processor resources, memory, etc., can also be virtualized in order to provide resources from which other virtual machines are built. A set of computing resources that can be allocated for providing various computing resources can be viewed as providing a distributed computing system, the internal architecture of which can be configured in various ways, while these computing resources are used to implement various computing components of the system.

[0118] As noted above, in some embodiments, the LRZ convolution process flags a timestamp as an abnormal transition point if there is at least a certain number of abnormal transition point candidates at that timestamp, and each candidate is from a different statistic. In alternative embodiments, an anomaly can be declared if the average, maximum, or minimum z-score (at a given timestamp) across multiple statistics reaches or exceeds a predetermined threshold.

[0119] In various embodiments, detecting anomalies can be based on a combination of multiple statistics (or dimensions of time series data) at a time (e.g., as described above in connection with Figure 4 This can be beneficial when the multiple statistics are moderately to highly correlated. For example, in the application domain of computer system monitoring, performance data such as CPU usage, disk usage, memory usage, and network usage are moderately to highly correlated. For example, when disk usage increases, CPU usage typically also increases because it needs CPU to process the additional disk data. If the individual statistics are highly uncorrelated, it can be preferable to declare an anomaly when the maximum z-score exceeds a threshold. If the individual statistics are highly correlated, it can be preferable to declare an anomaly when the minimum z-score exceeds a threshold.

[0120] Some embodiments are used to handle missing portions of input data, e.g., as illustrated below. In certain applications such as weather sensing, time series data can have gaps (missing values) due to extreme weather conditions, sensor malfunction, battery depletion, etc. In such cases, an interpolation technique can be employed to generate missing data values prior to automated anomaly detection. Such interpolation is not expected to significantly degrade anomaly detection quality if the time interval between missing values is small enough compared to the entire set of observations.

[0121] As noted above, extensibility enables changing the anomaly detection process, e.g., by using an SVM instead of the LRZ convolution process, or another anomaly detection process. In particular application domains, users can be satisfied with existing anomaly detection algorithms that they have developed and tuned. The explanation generation and anomaly filtering described herein can still be leveraged. As noted previously, for point anomalies, no changes to the data are required prior to providing the data to the explanation generator subsystem. If the anomaly has a "start" and "end" timestamp, these should be included as potential anomaly transition points input so that the explanation generation and anomaly filtering processes further process.

[0122] Embodiments of the invention include an electronic device, such as a computer system, for performing anomaly detection. The electronic device has a processor operably coupled to a memory, where the memory holds program instructions for execution by the processor for performing automatic anomaly detection. The electronic device can be separate from the computer system being monitored for performance. According to other embodiments, a hardware implementation can be provided instead of a computer software (i.e., a processor executing program instructions stored in memory) implementation. For example, using a field-programmable gate array (FPGA), the automatic anomaly detection program can be encoded into a silicon chip. Such embodiments run much faster than software implementations. At the same time, algorithm parameters in a hardware-only embodiment can be more difficult to adjust than in a software implementation.

[0123] According to some embodiments, multiple window widths can be employed, as described below. In certain application domains, time series data has a clear periodicity, and while there are peaks and troughs in each period, these do not necessarily constitute anomalies. For example, consider temperature recorded at a weather station. On a given weather season, the temperature each day has a reasonable periodicity, with temperatures during the day typically higher than at night. As the season changes, the difference can increase, decrease, or remain the same. In addition, there is a yearly cycle, after which a much more complex pattern repeats. Another example is the healthcare industry, when it comes to measuring a person's heartbeat. For such complex patterns, no single window width is sufficient to predict anomalies. The automatic anomaly detection is extended directly to use two or more different window widths to process the data. For each window width, detected anomalies can be reported to a domain expert who, after necessary analysis, can determine the correct window width. In addition, different window widths can be useful for extracting different types of anomalies.

[0124] Figure 11 A schematic diagram of an electronic device 1100 that can perform any or all of the operations of the methods and features described herein, explicitly or implicitly, is shown, according to different embodiments of the invention. The electronic device can include components as described in connection withFigure 1 Various aspects of the described anomaly detection system, such as anomaly detection, explanation generation, and anomaly filtering. These aspects can be provided, for example, using software stored in memory 1120 and / or mass storage 1130 and executed by processor 1100.

[0125] As shown, the device includes processor(s) 1110, memory 1120, mass storage 1130, I / O interface 1140, network interface 1150, and transceiver 1160, all of which are communicatively coupled via bi-directional bus 1170. According to certain embodiments, any or all of the described elements can be utilized as desired. Further, device 1100 can contain multiple instances of certain elements, such as multiple processors, memories or transceivers. In addition, hardware elements can be directly coupled to other elements without the use of the bus. Additionally or alternatively, other electronic devices, such as integrated circuits, can be employed instead of or in combination with the processor(s) and memory to perform the required logic operations.

[0126] Memory 1120 can include any type of non-transitory memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination of these or the like. Mass storage element 1130 can include any type of non-transitory storage device, such as a solid-state hard drive, hard disk drive, disk drive, optical drive, USB drive, or any computer program product for storing data and machine-executable program code. According to certain embodiments, memory 1120 or mass storage 1130 can have recorded thereon instructions and statements, which can be executed by processor 1110, for performing any of the method operations described above. Additionally or alternatively, hardware processing components, such as FPGAs or other hardware logic circuits, can be provided and configured to perform such operations.

[0127] In some embodiments, the electronic device can be a virtual machine provided by a computing cluster or cloud service provider, for example, executing instructions encoded in software of the anomaly detection system. Figure 1

[0128] Figure 12 ​An electronic device 1200 is shown provided in accordance with another embodiment of the present application. The electronic device includes a data interface 1210, processing hardware 1220, and optionally another data interface or user interface 1230. The processing hardware can include an anomaly detector 1222, an explanation generator 1224, and optionally an anomaly filter 1226. The anomaly detector 1222 performs the same functions as the anomaly detector 110 of Figure 1 、 Figure 7 and Figure 8 or the anomaly detector 910 of Figure 9 . For example, the explanation generator 1224 performs the same functions as the explanation generator 120 of Figure 1 、 Figure 7 and Figure 8 or the explanation generator 920 of Figure 9 . The anomaly filter 1226 performs the same functions as the anomaly filter 130 of Figure 1 、 Figure 7 and Figure 8 or the anomaly filter 930 of Figure 9 . The data interface 1210, the processing hardware 1220, or a combination thereof can also include a statistics collection module 1240. The statistics collection module 715 is connected to the computer system 710 to collect data (e.g., performance metadata) from the computer system. The data is collected and possibly stored or organized so that it can be passed forward for further processing. For example, the statistics collection module can be implemented by a computer processor executing program instructions stored in memory. The processing hardware 1220 can also include one or more input filters 1245, such as Kalman filters. The data interface 1210 can be a serial data interface or a parallel data interface or a network port, or an interface that uses a computer file system or an internal computer data bus, etc. The processing hardware 1220 can include a computer processor executing program instructions stored in memory, or a hardware processing component, or both. The user interface 1230 can include a visual display.

[0129] It is to be understood that even though numerous characteristics and embodiments of the present application have been set forth in the foregoing disclosure, the details can be varied without departing from the application, and it is therefore understood that it is intended to cover all alternatives, modifications, and equivalents falling within the true scope of the application. Especially, computer program products or program elements, program memories or memories for use with various devices, or the like, storing machine-readable signals, which are used or otherwise yield the operations and functionality described in the embodiments of the present application, are intended to come within the scope of the present application.

[0130] The actions associated with the methods described herein can be implemented as coded instructions in a computer program product. In other words, the computer program product is a computer readable medium on which the software code is recorded so as to execute the methods when loaded into the memory and executed on the microprocessor of the wireless communication device.

[0131] Furthermore, each operation of the methods can be performed on any computing device, for example, a personal computer, a server, a PDA, etc., and from one or more program elements, modules or objects generated from, for example, any programming language, such as C++, Java, etc., or one or more parts of said one or more. Moreover, each operation or the files or objects that implement each of the operations, etc., can be performed by special-purpose hardware or circuitry designed for that purpose or by a circuit module designed for that purpose.

[0132] It is clear that the above-described embodiments of the application are exemplary and can be implemented in various ways. Such present or future changes shall not be considered as a departure from the spirit and scope of the application, and all such modifications apparent to those skilled in the art are intended to be included within the scope of the following claims.

Claims

1. A method for performance monitoring of a computer system (710) in an electronic device (1100), characterized by, The method comprises: obtaining performance metadata indicative of usage of the computer system or of more computing resources; detecting one or more anomalous transition point candidates in the performance metadata, each of the one or more anomalous transition point candidates being indicative of a possible transition to or from an anomalous behavior pattern in a process by which the computer system generates the performance metadata, wherein each of the anomalous transition point candidates is an independent transition point that is not paired with others of the anomalous transition point candidates; and reporting one or more of the anomalous transition point candidates as anomalous transition points; for at least one of the anomalous transition point candidates, generating an explanatory predicate indicative of a human-readable explanation of behavior of the process at the at least one of the anomalous transition point candidates; the performance metadata comprises a first series of values indexed sequentially according to a corresponding series of reference values; detecting the one or more anomalous transition point candidates comprises: for each of a plurality of the reference values: determining a first mean and a first variance of a portion of the first series of values, the portion of the first series of values corresponding to the reference value, the reference value being within a predetermined interval immediately preceding the one of the plurality of reference values; determining a second mean and a second variance of another portion of the first series of values, the other portion of the first series of values corresponding to the reference value, the reference value being within the predetermined interval immediately following the one of the plurality of reference values; based on the first mean, the second mean, the first variance, and the second variance, determining a z-score for the one of the plurality of reference values, the z-score being indicative of a likelihood that the one of the plurality of reference values represents one of the anomalous transition points; and identifying the one of the plurality of reference values as representing the one of the anomalous transition point candidates based at least in part on the z-score.

2. The method of claim 1, wherein, the explanatory predicate is based on a difference between the first mean and the second mean.

3. The method of claim 1, wherein, the series of reference values is a series of timestamps at which corresponding values in the first series of values occur.

4. The method of claim 1, wherein: the performance metadata further comprises a second series of values indexed sequentially according to the series of reference values; detecting the one or more anomalous transition point candidates further comprises: for each of the plurality of the reference values: determining a third mean and a third variance of a portion of the second series of values, the portion of the second series of values corresponding to the reference value, the reference value being within the predetermined interval immediately preceding the one of the plurality of reference values; determining a fourth mean and a fourth variance of another portion of the second series of values, the other portion of the second series of values corresponding to the reference value, the reference value being within the predetermined interval immediately following the one of the plurality of reference values; determining a second z-score for the one of the plurality of reference values based on the third average, the fourth average, the third variance, and the fourth variance, the second z-score indicating another likelihood that the one of the plurality of reference values represents one of the anomalous transition points; and identifying the one of the plurality of reference values as representing the one of the anomalous transition point candidates based at least in part on the z-score and the second z-score.

5. The method of claim 4, wherein, the explanatory predicate is based on a difference between the first average and the second average, and is further based on a difference between the third average and the fourth average.

6. The method of claim 1, wherein, Further comprising: generating the explanatory predicate for each of the anomalous transition point candidates when a likelihood that the one of the plurality of reference values represents one of the anomalous transition points exceeds a predetermined threshold, and refraining from generating the explanatory predicate otherwise.

7. The method of claim 6, wherein, Further comprising: retaining the one of the anomalous transition point candidates for possible reporting as one of the anomalous transition points only when the explanatory predicate has been generated.

8. The method according to any one of claims 1 to 7, characterized in that, Further comprising: identifying two or more of the anomalous transition point candidates that correspond to reference values within a predetermined interval; identifying one of the two or more anomalous transition point candidates that has a highest likelihood of representing one of the anomalous transition points; and retaining the identified one of the two or more anomalous transition point candidates for possible reporting as one of the anomalous transition points, and removing each other of the two or more anomalous transition point candidates as a possible anomalous transition point of the anomalous transition points.

9. The method according to any one of claims 1 to 7, characterized in that, the performance metadata is raw performance metadata, or the data is filtered performance metadata obtained from the raw performance metadata.

10. The method according to any one of claims 1 to 7, characterized in that, Further comprising: adjusting one or more anomaly detection operation parameters to adjust a number of the reported anomalous transition points.

11. A method for automatically detecting anomalies in performance metadata in an electronic device, comprising: The method comprises: detecting one or more anomalous transition point candidates in the performance metadata, each of the one or more anomalous transition point candidates indicating a possible transition to or from an anomalous behavior pattern in a process that generated the performance metadata, wherein each of the anomalous transition point candidates is an independent transition point that is not paired with others of the anomalous transition point candidates; and reporting one or more of the anomalous transition point candidates as anomalous transition points; for at least one of the anomalous transition point candidates, generating an explanatory predicate indicating a human-readable explanation of behavior of the process at the at least one of the anomalous transition point candidates; the performance metadata comprises a first series of values sequentially indexed according to a corresponding series of reference values; detecting the one or more anomalous transition point candidates comprises: for each of a plurality of the reference values: determining a first mean and a first variance for a portion of the first series of values corresponding to the reference value, the reference value being within a predetermined interval immediately preceding the one of the plurality of reference values; determining a second mean and a second variance for another portion of the first series of values corresponding to the reference value, the reference value being within a predetermined interval immediately following the one of the plurality of reference values; based on the first mean, the second mean, the first variance, and the second variance, determining a z-score for the one of the plurality of reference values, the z-score indicating a likelihood that the one of the plurality of reference values represents one of the abnormal transition points; and identifying the one of the plurality of reference values as representing the one of the abnormal transition point candidates based at least in part on the z-score.

12. An electronic device (1100) for automatically detecting anomalies in performance metadata, characterized in that, The electronic device includes a data interface (1210) and electronic data processing hardware (1220), and is to: receive the performance metadata using the data interface; detect, using the electronic data processing hardware, one or more abnormal transition point candidates in the performance metadata, each of the one or more abnormal transition point candidates indicating a possible transition to or from an abnormal behavior pattern in a process that generated the performance metadata, wherein each of the abnormal transition point candidates is an independent transition point that is not paired with others of the abnormal transition point candidates; and report one or more of the abnormal transition point candidates as an abnormal transition point using the data interface or another data interface or a user interface; for at least one of the abnormal transition point candidates, generate, using the electronic data processing hardware, an explanatory predicate indicating a human-readable explanation of behavior of the process at the at least one of the abnormal transition point candidates; the performance metadata includes a first series of values sequentially indexed according to a corresponding series of reference values; detecting the one or more abnormal transition point candidates includes: for each of a plurality of the reference values: determining a first mean and a first variance for a portion of the first series of values corresponding to the reference value, the reference value being within a predetermined interval immediately preceding the one of the plurality of reference values; determining a second mean and a second variance for another portion of the first series of values corresponding to the reference value, the reference value being within a predetermined interval immediately following the one of the plurality of reference values; based on the first mean, the second mean, the first variance, and the second variance, determining a z-score for the one of the plurality of reference values, the z-score indicating a likelihood that the one of the plurality of reference values represents one of the abnormal transition points; and identifying the one of the plurality of reference values as representing the one of the abnormal transition point candidates based at least in part on the z-score.

Images