Anomaly Detection

Abstract

According to an embodiment of the present invention, a method accurately detects anomalies or outliers in a time series data set. A method for identifying whether a particular data element of a time series data set is an outlier includes predicting a value of the particular data element and obtaining a threshold value, the threshold value defining whether an actual value of the data element is an outlier relative to the predicted value. In one aspect of an embodiment of the present invention, the threshold value is generated based on historical error values ​​associated with data elements that precede the particular data element in time in the time series data set.

Description

Background Art

[0001] Embodiments of the present invention relate to anomaly detection, and in particular to anomaly detection using a data modeling method. Embodiments of the present invention also relate to a computer program product, which includes a computer-readable program code that enables a processor of a system to implement this method. Embodiments of the present invention also relate to a system for detecting anomalies. Such a system can be used to identify anomalies in any data found in a variety of different fields. Such data can include, for example, health data (e.g., for detecting abnormal test results), security data (e.g., for detecting fraud / intrusion), or communication data (e.g., for detecting transmission errors).

[0002] Anomaly detection aims to identify data points or data patterns that deviate from the expected behavior of that data point or pattern (e.g., anomalies). In different fields, anomalies can be alternatively labeled as outliers, inconsistent observations, exceptions, aberrations, surprises, uniqueness, or contaminants.

[0003] Anomaly detection is useful in various application areas (e.g., health, fraud / intrusion security detection systems) and is particularly beneficial for 5G network operators to avoid or detect performance degradation in a timely manner, which may otherwise lead to service interruptions and increased maintenance costs. The goal is to detect degradation in a proactive (e.g., prepare before the degradation occurs) or reactive (e.g., react after the degradation occurs) manner to allow corrective actions to be applied at an early stage to minimize the impact of the degradation and reduce costs.

[0004] There are a plethora of anomaly detection techniques which can be categorized based on the type of strategy adopted: i) instance-based methods, a popular formulation for outlier detection is to find anomalies in multidimensional data by their distance from neighboring points, and (ii) explicit generalization methods, another popular approach, where a generalization model is created in advance which models the normal behavior of the monitored instances and also uses the deviation between the expected normal behavior and the actual behavior as an error measure for anomaly detection.

[0005] The most common prior art techniques employed in practical systems employ threshold-based methods, which can be categorized as parametric and non-parametric methods. Parametric methods make assumptions about the underlying distribution of the data set and determine the threshold based on that distribution. However, this assumption is often impractical and violated in practice. In addition, non-parametric methods avoid making such assumptions, but determine the threshold in an ad hoc manner. However, both of these methods are generally unrealistic, not suitable for changing patterns in the incoming data set, and generally require a lot of effort in adjusting the threshold.

[0006] More sophisticated anomaly detection techniques employ machine learning methods, which can be divided into three categories: supervised, semi-supervised, and unsupervised. Supervised learning anomaly detection requires a dataset in which each instance is labeled, and typically it involves training a classifier on a training set. Semi-supervised algorithms, for example, build a model representing normal behavior based on an input normal training dataset, following the model to calculate the likelihood that a test dataset will be generated by the model. Unsupervised models do not require a labeled dataset and operate under the assumption that most data points are normal, returning the remaining data points as outliers.

[0007] However, most methods such as statistical and probabilistic models are typically applicable to univariate datasets, where a single metric is monitored at a time. This can be extended to multiple metrics by building a model for each metric; however, this will not directly account for any correlations between the metrics. Therefore, these methods cannot be easily extended to multivariate analysis, where correlations between metrics can be used to identify potential abnormal behavior. In addition, existing methods typically rely on datasets containing ground truth labels, where anomalies are specifically pointed to data points. This is difficult to collect in real-world situations because labeled data is expensive and requires expert knowledge, which may be affected by human errors in the labeled data. In addition, the initial model may not generalize to new types of anomalies unless retrained, requiring expert knowledge for the entire duration of the deployment of the anomaly detection model. This makes these methods impractical to deploy in dynamic environments such as 5G networks. Summary of the invention

[0008] Embodiments of the present invention seek to provide a method for determining whether a data element having a value of a time series data set is an outlier.

[0009] Embodiments of the present invention also seek to provide a computer program product comprising computer program code for implementing the method when executed on a processor of a data processing system.

[0010] The embodiment of the present invention also attempts to provide a system suitable for executing the computer program code.

[0011] Embodiments of the present invention also seek to provide a system for determining whether a data element having a value of a time series data set is an outlier.

[0012] According to an embodiment of the present invention, a computer-implemented method for determining whether a data element with a value of a time series data set is an outlier is provided. The method includes obtaining prediction data for predicting the value of the data element from first data of the time series data set that precedes the data element in time, and using the prediction data to predict the predicted value of the data element. The method also includes obtaining historical error values ​​of the time series data set, each historical error value representing the difference between the value of a second data element of the time series data set that precedes the data element in time and the predicted value of the second data element. The method also includes obtaining a threshold value based on one or more historical error values, the threshold value being used to define the value of the data element that is considered to be an outlier relative to the predicted value of the data element, and determining whether the data element is an outlier based on the threshold value, the predicted value, and the value of the data element.

[0013] The use of historical error values ​​in the generation of thresholds increases the accuracy of identifying abnormal data values. Specifically, it is easy to see whether the current error value has deviated from the expected error value (e.g., deviated from the historical data value) to identify whether the error value associated with the data element falls within a normal or expected range.

[0014] Therefore, using historical error values ​​improves the accuracy and precision of identifying anomalous data elements of a time series data set.

[0015] The historical error values ​​may each be a squared error value. The squared error value is the square of the difference between the predicted value of a data element and the actual value of the data element. A mean squared error value may be used as a historical error value for a data element including a plurality of values.

[0016] Using the square of the historical error values ​​to calculate the threshold results in improved accuracy and precision in identifying outlier data based on the benchmark. This effect is particularly significant when the square of the error value of the data element under investigation is compared to the threshold to determine whether the data element under investigation is an outlier.

[0017] In some embodiments of the invention, each historical error value is scaled to a value between 0 and 1 (e.g., normalized). This ensures that a particular historical error value does not disproportionately affect the calculation of the threshold. To improve consistency, the error value of the (investigated) data element compared to the threshold may also be scaled between 0 and 1 (e.g., using the same scaling factor as the historical value).

[0018] In a further embodiment of the invention, the standard deviation of the historical error values ​​is used to calculate or otherwise obtain the threshold value. Since the standard deviation represents the normal or expected behavior of the system, using the standard deviation of the historical error values ​​to generate the threshold value allows accurate identification of outliers.

[0019] The principle of Chebyshev's inequality can be used to determine an appropriate threshold for identifying anomalous data elements based on the standard deviation. This improves the reliability of detecting anomalous data elements because no assumptions are made about the underlying distribution of historical error values. This thereby increases the number of applications in which the outlier detection method can be used (e.g., where a non-normal distribution is not expected).

[0020] In certain embodiments of the present invention, patterns in historical error data may be utilized to improve the accuracy of identifying outlier data elements. It has been recognized that there may be periodicity in historical error values ​​because error values ​​at the same time of the hour / day / week (or other period) are similar. Therefore, using historical error values ​​associated with periodic time periods allows for more accurate identification of outlier data elements because this periodicity can be considered part of expected normal behavior. In other words, the behavior of a time series data set may change periodically so that similar behavior (matching the periodicity of the time series data set) may be considered normal. In various examples, the predetermined time period may be an hour, a day, or a week because a time series data set is more likely to have periodic behavior with such a time period.

[0021] The step of predicting a predicted value of a data element may include predicting a corresponding plurality of potential predicted values ​​of the data element using the prediction data and a plurality of different prediction models. The same step may also include designating the potential predicted value closest to the value of the data element as the predicted value.

[0022] If none of the multiple models have managed to accurately predict the value of the data element under study, it is more likely that it is an anomaly or outlier. Therefore, by using a predicted value (out of multiple possible predicted values) that is closest to the actual / measured value of the data element, the accuracy and reliability of identifying anomalous data elements can be improved.

[0023] Benchmark tests have shown that using multiple prediction models to generate predictions is particularly effective. In particular, it improves the precision and recall of detecting anomalous data elements.

[0024] According to another embodiment of the present invention, there is provided a computer program product for determining whether a data element having a value of a time series data set is an outlier. The computer program product comprises a computer-readable storage medium having program instructions contained therein, the program instructions being executable by a processing unit to cause the processing unit to perform a method according to an embodiment of the present invention, such as those previously described.

[0025] A system is also provided, comprising at least one processor and a computer program product according to an embodiment of the invention. The at least one processor is adapted to execute computer program code of the computer program product.

[0026] Therefore, a system for determining whether a data element with a value of a time series data set is an outlier is provided. The system includes a first obtaining unit, which is suitable for obtaining prediction data from first data of the time series data set that is before the data element in time to predict the value of the data element. The system also includes a prediction unit, which is suitable for using the prediction data to predict the predicted value of the data element. The system also includes a second obtaining unit, which is suitable for obtaining historical error values ​​of the time series data set, each historical error value represents the difference between the value of a second data element of the time series data set that is before the data element in time and the predicted value of the second data element. The second obtaining unit can also be suitable for obtaining a threshold value based on one or more historical error values, and the threshold value is used to define the value of the data element that is considered to be an outlier relative to the predicted value of the data element. The system also includes a determination unit, which is suitable for determining whether the data element is an outlier based on the threshold value, the identified predicted value and the value of the data element. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, in which:

[0028] Figure 1 Describes a graphical representation of an example distributed system in which aspects of the illustrative embodiments may be implemented;

[0029] Figure 2 is a block diagram of an example system in which aspects of the illustrative embodiments may be implemented;

[0030] Figure 3 is a flow chart of a method according to an embodiment of the present invention;

[0031] Figure 4 According to an embodiment of the present invention Figure 3 a flow chart of a portion of a method;

[0032] Figure 5 According to another embodiment of the present invention Figure 3 a flow chart of a portion of a method;

[0033] Figures 6A-6D is a diagram illustrating the effectiveness of methods according to different embodiments of the present invention;

[0034] Figure 7 A system according to an embodiment of the present invention is shown;

[0035] Figure 8 is a block diagram of an example system in which aspects of the illustrative embodiments may be implemented; and

[0036] Fig. 9 The structure of the method or system according to the preferred embodiment of the present invention is shown. DETAILED DESCRIPTION

[0037] It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to represent the same or similar components.

[0038] In the context of the present application, where the embodiments of the present invention constitute methods, it is understood that such methods are processes for being performed by a computer, e.g., computer-implementable or computer-implemented methods. Therefore, the various steps of the method reflect various parts of a computer program, e.g., various parts of one or more algorithms.

[0039] In addition, in the context of the present application, a system may be a single device or a collection of distributed devices suitable for performing one or more embodiments of the method of the present invention. For example, a system may be a personal computer (PC), a server, or a collection of PCs and / or servers connected via a network such as a local area network, the Internet, etc., so as to collaboratively perform at least one embodiment of the method of the present invention.

[0040] Embodiments of the present invention provide an explicit generalization model based approach to discover anomalies in a timely manner by leveraging a wide range of statistical and probabilistic methods and deep learning models to create predictions. The prediction closest to the actual value is selected at each timestamp, so that the anomaly detection solution only warns about true positive anomalies and has a minimum number of false negatives.

[0041] The method of the embodiment of the present invention also utilizes a non-traditional dynamic sliding window mechanism to determine a dynamic threshold for fitting each analyzed time series, wherein the dynamic window contains the rescaled square error of each point to ensure the highest accuracy.

[0042] Other aspects make the method of the embodiment of the present invention more effective in practice than other anomaly detection techniques. Typically, anomaly detection algorithms often require users to select thresholds that are difficult to determine and provide gold labels for testing, which can be expensive and prone to errors. This is avoided by the anomaly detection method of the embodiment of the present invention, which calculates dynamic thresholds based on past prediction errors, and because a semi-supervised learning method is used, no labels are required for training. In addition, the LSTM model of the embodiment of the present invention can receive multiple features as input, which makes the embodiment of the present invention multivariate compared to a univariate anomaly detector because the embodiment of the present invention can use a single model that can capture anomalies across multiple features.

[0043] A concept for accurately detecting anomalies or outliers of a time series data set is proposed. Specifically, the proposed concept relies on a common underlying method for identifying whether a particular data element of a time series data set is an outlier. The common method includes predicting a value for the particular data element and obtaining a threshold value that defines whether the actual value of the data element is an outlier relative to the predicted value.

[0044] In some examples, an error value is obtained for a particular data element, the error value representing the difference between the actual value and the predicted value of the data element. The error value can be compared with a threshold to determine whether the actual value and the associated data element are outliers. This provides a highly accurate but adaptable method for determining whether a data element is an outlier.

[0045] In an embodiment of the present invention, the prediction data used to predict the value of a data element includes a predetermined number of other data elements of the time series data set that precede the data element in time.

[0046] According to an embodiment of the present invention, the historical error value is used to identify whether the following data element is an outlier. Here, the historical error value is an error value associated with a data element before (e.g., earlier in time than) the studied data element in the time series data set. Specifically, the historical error value is used to generate a threshold value for defining whether a data element is an outlier.

[0047] Embodiments of the present invention are based on the recognition that historical error values ​​can be used to identify whether a current data element is an outlier, where each of the historical error values ​​represents the difference between a predicted value and an actual value of a previous data element in a time series data set. In particular, it can be determined whether a current error value (being the error value associated with the current data element being evaluated or studied) is within a normal range or whether it is an outlier. This allows for an improved assessment of whether a data element is an outlier relative to previous data elements of the time series data set.

[0048] Therefore, embodiments of the present invention propose using historical error values ​​to generate a threshold value for determining whether a (current) data element is an outlier.

[0049] According to an embodiment of the present invention, multiple prediction models are used to generate predicted values ​​for the data element under study. The predicted value generated by the prediction model that generates the predicted value that best matches the actual value of the data element, or the predicted value of the prediction model that best predicts the entire time series data set including the data element, is selected as the predicted value.

[0050] Therefore, in an embodiment of the present invention, multiple potential values ​​of the predicted value are generated, and the potential predicted value that is closest to the actual value of the data element (under investigation) is selected as the predicted value.

[0051] If none of the multiple models have managed to accurately predict the value of the data element under study, it is more likely that it is an anomaly or outlier. Therefore, by using a predicted value (out of multiple possible predicted values) that is closest to the actual / measured value of the data element, the accuracy and reliability of identifying anomalous data elements can be improved.

[0052] The method / system of the embodiment of the present invention adopts the concept of generating thresholds based on historical error values ​​and generating predicted values ​​using multiple prediction models to provide a particularly effective, precise and accurate method for identifying whether a data element is an outlier.

[0053] However, the method of an embodiment of the present invention adopts the concept of using multiple prediction models to generate prediction values, which can themselves improve the accuracy of identifying outliers. Therefore, a method is also envisioned in which there is no need to use historical error values ​​to generate thresholds.

[0054] Thus, a computer-implemented method for determining whether a data element having a value of a time series data set is an outlier may be provided. The method comprises: obtaining prediction data from first data of the time series data set that temporally precedes the data element for predicting the value of the data element; predicting a corresponding plurality of predicted values ​​of the data element using the prediction data and a plurality of different prediction models; identifying the predicted value that is closest to the value of the data element from the plurality of predicted values; obtaining a threshold value for defining a value of the data element that is considered to be an outlier relative to the predicted value of the data element; and determining whether the data element is an outlier based on the threshold value, the identified predicted value, and the value of the data element.

[0055] The predicted data may include a predetermined number of other data elements of the time series data set that precede the data element in time.

[0056] The step of obtaining the threshold value may include obtaining historical error values ​​of the time series data set, each historical error value representing the difference between a value of a second data of the time series data set before the data element in time and a corresponding predicted value of the second data of the time series data set. The step of obtaining the threshold value may also include obtaining the threshold value based on one or more historical error values.

[0057] Preferably, each historical error value is a square of an error value which is the square of the difference between a value of second data of the time series data set temporally preceding the data element and a corresponding predicted value of the second data of the time series data set.

[0058] The step of obtaining the threshold value may include scaling each historical error value to a value between 0 and 1 based on a maximum value of the historical error values.

[0059] The step of obtaining the threshold value may include calculating a standard deviation of historical error values, and obtaining the threshold value based on the standard deviation of the historical error values.

[0060] The step of obtaining the threshold value may further include multiplying the standard deviation of the historical error values ​​by a predetermined number to obtain the threshold value. Preferably, the step of obtaining the threshold value includes calculating the predetermined number using Chebyshev's inequality.

[0061] In an embodiment of the present invention, the second data corresponds to data of a time series data set obtained in a plurality of predetermined time periods before the data element. Preferably, the predetermined time period is one hour, one day or one week.

[0062] In at least one embodiment of the invention, the data elements are data points of a time series data set.

[0063] A computer program product for determining whether a data element having a value of a time series data set is an outlier may also be provided. The computer program product includes a computer-readable storage medium having program instructions. The program instructions are executable by a processing unit to cause the processing unit to perform a method, the method comprising: obtaining prediction data for predicting the value of the data element from first data of the time series data set that temporally precedes the data element, and using the prediction data and a plurality of different prediction models to predict a plurality of corresponding predicted values ​​of the data element; identifying the predicted value that is closest to the value of the data element from the plurality of predicted values; obtaining a threshold for defining the value of the data element that is considered to be an outlier relative to the predicted value of the data element; and determining whether the data element is an outlier based on the threshold, the identified predicted value, and the value of the data.

[0064] Also proposed is a computer system comprising at least one processor and a computer program product as previously described, wherein the at least one processor is adapted to execute computer program code of the computer program product.

[0065] A system for determining whether a data element having a value of a time series data set is an outlier may also be provided. The system includes a first obtaining unit adapted to obtain prediction data from first data of the time series data set that temporally precedes the data element for predicting the value of the data element. The system also includes a prediction unit adapted to predict a plurality of corresponding prediction values ​​of the data element using the prediction data and a plurality of different prediction models; and to identify the prediction value that is closest to the value of the data element from the plurality of prediction values. The system also includes a second obtaining unit adapted to obtain a threshold value for defining a value of a data element that is considered to be an outlier relative to the prediction value of the data element; and a determination unit adapted to determine whether the data element is an outlier based on the threshold value, the identified prediction value, and the value of the data element.

[0066] The illustrative embodiments of the present invention can be used in many different types of data processing environments. To provide a context for describing the elements and functions of the illustrative embodiments, the following is provided. Figure 1 and Figure 2 As an example environment in which aspects of the illustrative embodiments may be implemented. It should be understood that Figure 1 and Figure 2 This is merely an example and is not intended to assert or imply any limitation with respect to the environments in which aspects or embodiments of the invention may be implemented. Many modifications to the described environment may be made without departing from the scope of the invention.

[0067] Figure 1 A graphical representation of an exemplary distributed transaction processing storage system in which aspects of the exemplary embodiments may be implemented is depicted. Distributed system 100 may include a network of computers in which aspects of the illustrative embodiments may be implemented. Distributed system 100 includes at least one network 102, which is a medium for providing communication links between various devices and computers connected together within distributed data processing system 100. Network 102 may include connections such as wired, wireless communication links, or fiber optic cables.

[0068] In the depicted example, first server 104 and second server 106 are connected to network 102 along with storage unit 108. In addition, clients 110, 112, and 114 are also connected to network 102. Clients 110, 112, and 114 may be, for example, personal computers, network computers, etc. In the depicted example, first server 104 provides data, such as boot files, operating system images, and applications to clients 110, 112, and 114. In the depicted example, clients 110, 112, and 114 are clients of first server 104. Distributed system 100 may include additional servers, clients, and other devices not shown.

[0069] In the depicted example, distributed system 100 is the Internet, where network 102 represents a worldwide collection of networks and gateways that use the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite to communicate with each other. The core of the Internet is a backbone of high-speed data communication lines between major nodes or host computers, including thousands of commercial, government, educational, and other computer systems that route data and messages. Of course, distributed system 100 can also be implemented to include multiple different types of networks, such as intranets, local area networks (LANs), wide area networks (WANs), etc. As described above, Figure 1 It is intended as an example and not as an architectural limitation to the different embodiments of the present invention, and thus, Figure 1 The specific elements shown are not to be construed as limiting the environments in which illustrative embodiments of the invention may be implemented.

[0070] Figure 2 2 is a block diagram of an example system 200 in which aspects of the illustrative embodiments of the present invention may be implemented. System 200 is such as Figure 1 An example of a computer of client 110 in which computer usable code or instructions implementing the processes of the illustrative embodiments of the present invention may be located.

[0071] In the depicted example, system 200 employs a hub architecture that includes a north bridge and memory controller hub (NB / MCH) 202 and a south bridge and input / output (I / O) controller hub (SB / ICH) 204. Processing unit 206, main memory 208, and graphics processor 210 are connected to NB / MCH 202. Graphics processor 210 may be connected to NB / MCH 202 via an accelerated graphics port (AGP).

[0072] In the depicted example, a local area network (LAN) adapter 212 is connected to the SB / ICH 204. An audio adapter 216, a keyboard and mouse adapter 220, a modem 222, a read-only memory (ROM) 224, a hard disk drive (HDD) 226, a CD-ROM drive 230, a universal serial bus (USB) port and other communication ports 232, and a PCI / PCIe device 234 are connected to the SB / ICH 204 via a first bus 238 and a second bus 240. PCI / PCIe devices may include, for example, Ethernet adapters, add-on cards, and PC cards for notebook computers. PCI uses a card bus controller, while PCIe does not. ROM 224 may be, for example, a flash basic input / output system (BIOS).

[0073] HDD 226 and CD-ROM drive 230 are connected to SB / ICH 204 via second bus 240. HDD 226 and CD-ROM drive 230 may use, for example, an integrated drive electronics (IDE) or serial advanced technology attachment (SATA) interface. Super I / O (SIO) device 236 may be connected to SB / ICH 204.

[0074] An operating system runs on processing unit 206. The operating system coordinates and provides Figure 2 The operating system 200 of the system 200 can be controlled by the operating system 200. As a client, the operating system can be a commercially available operating system. An object-oriented programming system, such as a JAVA programming system, can run with the operating system and provide calls from a JAVA program or an application program executed on the system 200 to the operating system.

[0075] As a server, system 200 may be, for example, an IBM ESERVER SYSTEM P computer system running an Advanced Interactive Executive (AIX) operating system or a LINUX operating system. System 200 may be a symmetric multiprocessor (SMP) system including a plurality of processors in processing unit 206. Alternatively, a single processor system may be employed.

[0076] Instructions for the operating system, programming system, and applications or programs are located on storage devices such as HDD 226 and may be loaded into main memory 208 for execution by processing unit 206. Similarly, one or more message handling programs according to embodiments of the present invention may be suitable for storage by storage devices and / or main memory 208.

[0077] The processes of the illustrative embodiments of the present invention may be performed by processing unit 206 using computer usable program code that may be located in a memory such as main memory 208 , ROM 224 , or in one or more peripheral devices 226 and 230 .

[0078] Such as Figure 2 The bus system shown as the first bus 238 or the second bus 240 may include one or more buses. Of course, the bus system may be implemented using any type of communication structure or architecture that provides for data transmission between different components or devices attached to the structure or architecture. Figure 2 A communication unit such as a modem 222 or a network adapter 212 may include one or more devices for transmitting and receiving data. The memory may be, for example, the main memory 208, the ROM 224, or a memory such as a memory card. Figure 2 The cache found in the NB / MCH 202 in.

[0079] Those of ordinary skill in the art will understand that Figure 1 and Figure 2 The hardware in can vary depending on the implementation. Figure 1 and Figure 2 In addition to or in place of the hardware described in the drawings, other internal hardware or peripheral devices such as flash memory, equivalent nonvolatile memory, or optical disk drives may be used. Furthermore, the processing of the exemplary embodiments may be applied to multi-processor data processing systems other than the previously mentioned systems without departing from the spirit and scope of the invention.

[0080] Furthermore, system 200 may take the form of any of a variety of different data processing systems, including a client computing device, a server computing device, a tablet computer, a laptop computer, a phone or other communication device, a personal digital assistant (PDA), etc. In some illustrative examples, system 200 may be a portable computing device that is configured with flash memory to provide non-volatile memory for storing, for example, operating system files and / or user-generated data. Thus, system 200 may be essentially any known or later developed data processing system, without architectural limitation.

[0081] As used herein, the term "data element" is used to refer to a part or a portion of a time series data set, such as one or more data points of a data set. A single data element may contain a group of data points or a single data point. A data element may be associated with one or more timestamps, each of which may represent a single time instance or a time period (e.g., a series of consecutive time instances). Each data element may be associated with a single value or multiple values, e.g., each data element may be one-dimensional or multi-dimensional.

[0082] The value or "actual value" of a data element refers to the value actually stored by or contained in the data element. This is different from the "predicted value" of a data element, which instead refers to a prediction of the actual value (e.g., not the actual value itself) and is typically determined based on the values ​​of other data elements in the data set.

[0083] The term "outlier" can be used interchangeably with the term "anomaly".

[0084] Figure 3 A flow chart of a method 30 for determining whether a data element (eg, "the studied data element" or "the current data element") is an outlier according to an embodiment of the present invention is shown. Different examples of the present inventive concept utilize and improve upon this embodiment.

[0085] Method 30 begins at step 31, which includes obtaining prediction data for predicting the value of a data element. In particular, step 31 includes obtaining prediction data from a first data of the time series data set that precedes the data element in time. Thus, step 31 obtains prediction data from data in the time series data set that is earlier than the data element under investigation.

[0086] For example, step 31 may include obtaining a plurality of data elements that precede the data element under investigation in time in the data sequence. This may be referred to as a "look-back" operation. The number of data elements obtained may vary depending on the application, but preferably includes no fewer than 3 data elements and / or no more than 20 data elements. Preferably, the data used to predict the value of the (under investigation) data element includes the data element that immediately precedes (and in time) the data element under investigation.

[0087] Thus, the predicted data obtained in step 31 may include multiple data elements of the time series data set, each data element of the first data preceding in time the data element under investigation. Similarly, the first data from which the predicted data is obtained may include multiple data elements of the time series data set, each data element of the first data preceding in time the data element under investigation.

[0088] Then, method 30 moves to step 32, which includes predicting one or more values ​​of the (under investigation) data element based on the prediction data. Thus, step 32 generates a predicted value of the under investigation data element that is different from the actual value (e.g., recorded or stored value) of the data element.

[0089] Step 32 preferably comprises applying the prediction model to the prediction data, thereby predicting one or more values ​​of the data element under investigation. Other embodiments of step 32 will be described later.

[0090] Any (one or more) suitable prediction models may be used in step 32, and known prediction models for predicting data values ​​include long-short term memory (LSTM), autoregressive integrated moving average (ARIMA), and triple exponential smoothing, also known as Holt-Winters (HW). The choice of prediction model may vary depending on the characteristics of the time series data set. For example, deep neural network prediction methods, such as LSTM, may provide the best results given large training data (e.g., a large amount of prediction data), while ARIMA and HW may provide better predictions given a small data set. It will be appreciated that other models may be used.

[0091] The forecasting model may be a time series forecasting (TSF) model. The output of a TSF model is a one-step ahead forecast, which identifies / predicts the value to be expected at the next timestamp, for example, for the data element under study.

[0092] After step 32, method 30 proceeds to step 33, which includes obtaining a threshold value for defining the value of a data element that is considered an outlier. Thus, the obtained threshold value can be used to determine whether the actual value of the data element is an outlier (at least with reference to the predicted value of the data element).

[0093] After step 33, the method 30 proceeds to step 34, which includes determining whether the (investigated) data element is an outlier based on the threshold, the predicted value and the (actual) value of the data element. Thus, it can be determined whether the data element is an outlier, for example, based at least on a comparison of the predicted value and the (actual) value of the data element with reference to the threshold.

[0094] In particular, step 34 may include calculating an error value using the predicted value and the (actual) value of the data element under investigation. The error value may be compared to a threshold value to determine whether the data element is an outlier (e.g., an anomaly). For example, if the error value of the data element under investigation is above the threshold value, it may be determined that the data element is an outlier. Similarly, if the error value of the data element under investigation is below the threshold value, it may be determined that the data element is not an outlier.

[0095] As an example, the difference (magnitude) between the (actual) value and the predicted value of the data element under investigation can be used as the error value. In another example, a specific error calculation (such as RMS error calculation, error square calculation or scaled error square calculation) is performed using the (actual) value and the predicted value of the data element under investigation to generate the error value.

[0096] Thus, in general, an "error value" represents the difference between a predicted value and an actual value of a data element. Those skilled in the art will know many methods of obtaining an error value representing such a difference.

[0097] The threshold value may indicate an upper limit of error values ​​that are considered to be within a normal or non-outlying range. Thus, the threshold value may indicate a maximum allowable error value that is considered not to be an outlier. In this way, an error value greater than the threshold value indicates that the data element associated with the error value is an outlier.

[0098] In other examples, the thresholds may indicate upper and lower limits of a range of (actual) values ​​of data elements that are not considered outliers, wherein any value falling outside the range is considered an outlier.

[0099] In another example, step 34 includes adding a threshold to the predicted value of the data element under investigation to define an upper limit of a range that is not considered an outlier, and subtracting the threshold from the same predicted value to define a lower limit of a range that is not considered an outlier. Any value that falls outside the upper and lower boundaries may be marked as an outlier or anomaly. In this way, the threshold may define a range of acceptable, non-outlier values ​​for the data element relative to the predicted value.

[0100] After identifying whether the data element is abnormal in step 34, the method may include the step of signaling that the data element under investigation is abnormal (e.g., triggering an alarm, generating a specific data set, etc.). The signaling may be used to mark or identify the abnormal data element. If an abnormal data element is identified, the method using abnormal data element detection may choose to ignore the abnormal data element or request that the abnormal data element be resent (e.g., in the case where the abnormality is caused by noise, etc.). Other uses of the anomaly detection method described herein will be apparent to those skilled in the art.

[0101] It is contemplated that method 30 may be applied iteratively to different data elements of a time series data set to identify whether different data elements are outliers. Each time method 30 is performed, for example, a new threshold may be calculated for each different data element under investigation. Thus, for example, the method may be iteratively performed for each data element (e.g., by the system), wherein a different threshold is determined for each data element (e.g., the threshold is dynamic).

[0102] It will be apparent that embodiments of the present invention may be applicable to determining whether a multi-dimensional (eg, including more than one value) data element is an outlier.

[0103] In such an embodiment, step 32 may include predicting a corresponding (e.g., identical) number of predicted values ​​for the data element as there is an (actual) value for the data element. Similarly, step 34 may include calculating an error value using the predicted and (actual) values ​​of the data element for comparison with a threshold value. Thus, a data element may be associated with a plurality of (actual) values ​​and a corresponding plurality of predicted values, from which a single error value may be calculated for comparison with a threshold value.

[0104] Other methods of adapting the proposed concepts to multivariate data will be apparent to those skilled in the art. For example, in embodiments of the present invention, each value of a multidimensional data element under investigation may be individually evaluated to determine whether it is an outlier (e.g., using the methods described herein). In the event that one or more values ​​of a multidimensional data element are identified as outliers, the entire data element may be considered an outlier / anomaly.

[0105] Therefore, those skilled in the art may easily modify the embodiments of the present invention for multi-dimensional data elements associated with more than one value.

[0106] Figure 4 An embodiment of a step 33 for obtaining a threshold value and a step 34 for determining whether the data element under investigation is an outlier is shown.

[0107] The illustrated embodiment of step 33 comprises a plurality of individual sub-steps.

[0108] The first sub-step 41 of step 33 includes obtaining historical error values ​​for the time series data set. Each historical error value represents a difference between at least one value of a second data element of the time series data set that precedes the data element under investigation in time (e.g., before or earlier than the data element under investigation in the time series data set) and at least one corresponding predicted value of the second data element. In other words, the historical error value represents a difference between (one or more) predicted values ​​and (one or more) corresponding actual values ​​of a previous data element in the time series data set. The historical error values ​​may optionally be labeled as previous error values.

[0109] Thus, a plurality of "second data elements" of the time series data set are data elements that precede the data element under investigation in time, each of which contributes to a corresponding historical error. Preferably, these second data elements are data elements that precede the (investigated) data element in the time series data set. For example, the second data elements may include the data element that immediately precedes the data element under investigation.

[0110] Sub-step 41 may comprise obtaining a predicted value for each second data element in the same manner as the predicted value for the data element under investigation. That is, data elements of the data set preceding a particular second data element of the time series data set may be used to predict the value of the second data element.

[0111] Sub-step 41 may comprise obtaining the historical error value, for example from a stored set of historical error values. Thus, the stored set of historical error values ​​may contain error values ​​for data elements of the time series data set that temporally precede the data element under investigation.

[0112] The number of historical error values ​​obtained in sub-step 41 may vary according to the embodiment. In one embodiment of the present invention, the number of historical error values ​​obtained is not less than 5, in another embodiment of the present invention, the number of historical error values ​​is not less than 10, in other preferred embodiments of the present invention, the number of historical error values ​​is not less than 20 or 50, and in other embodiments of the present invention, the number of historical error values ​​is not less than 100 or 200. In other embodiments of the present invention, all previous data elements (for the data element under study) are associated with historical error values, and all error values ​​are obtained in sub-step 41. The greater the number of historical error values, the higher the accuracy of identifying outliers, because the threshold value will more accurately represent the predicted historical error tolerance (e.g., more indicative of the characteristics of the data set).

[0113] A second subsequent sub-step 42 of step 33 comprises obtaining a threshold value based on historical error values.

[0114] In this manner, a threshold for determining whether a data element under investigation is an outlier is determined based on previous / historical error values ​​of data elements that temporally precede the data element under investigation in the time series data set.

[0115] Preferably, each historical error value obtained in sub-step 41 is a squared error value which is the square of the difference between the (actual) value and the predicted value of a corresponding second data element preceding the data element in time of the time series data set.

[0116] For data elements including multiple values, such as multidimensional data elements, the historical error value may be the mean square error of the second data elements. Thus, each second data element may be associated with multiple (actual) values ​​and corresponding multiple predicted values, wherein a single error value - the historical error value - is calculated using the multiple actual values ​​and predicted values. Other methods applicable to multivariate data will be apparent to those skilled in the art.

[0117] Using the square of the error value in the calculation of the threshold provides a more accurate and precise method of identifying whether a data element is an outlier.

[0118] In other examples, the historical error value may be the difference between the (actual) value and the predicted value of a corresponding second data element of the time series data set that precedes the data element under investigation in time. Other examples of suitable error values ​​for use as the historical error value are obvious to those skilled in the art (e.g., RMS value, logarithmic error, etc.), and methods for calculating the same are also obvious to those skilled in the art. Preferably, the historical error value is based on the magnitude of the difference, or includes only positive values ​​(such as square values), for example, it is unsigned or signless.

[0119] In an embodiment of the present invention, step 33 may further include sub-step 43, which, for example, scales each historical error value to a value between 0 and 1 based on the maximum value of the historical error values ​​obtained in step 41. Therefore, sub-step 43 may include determining the maximum value of the historical error values ​​obtained in step 41 and scaling each historical value based on the maximum value (for example, dividing each historical error value by the maximum value of the historical error values).

[0120] In another example, each historical error value is scaled based on the maximum of any historical error values ​​and the corresponding error value of the data element under investigation (e.g., the error value of the data element calculated in the same manner as each historical error value) in sub-step 43. This ensures consistency in the anomaly identification process.

[0121] In some embodiments of the present invention, the number of historical error values ​​is equal to the number of data elements in the time series data set that precede the data element under investigation in time. Therefore, each data element in the time series data set that precedes the data element under investigation can provide a historical error value. This increases the accuracy of the anomaly detection mechanism at the expense of processing efficiency and cost.

[0122] In the illustrated embodiment, sub-step 42 for obtaining the threshold value includes performing sub-step 44 and sub-step 45 in sequence.

[0123] Sub-step 44 includes calculating the standard deviation of the historical error values ​​obtained in step 41, which is performed using a conventional standard deviation calculation method.

[0124] Sub-step 45 comprises obtaining a threshold value based on the calculated standard deviation. In particular, sub-step 45 may comprise multiplying the standard deviation by a predetermined value and basing the threshold value on the result.

[0125] In the case where the historical error values ​​are only positive numbers (eg, each error value is the magnitude of the error or the square of the error value), sub-step 45 may include multiplying the standard deviation by a predetermined value (eg, 3 or 5) to directly obtain the threshold value.

[0126] When the historical error values ​​include both positive and negative numbers (e.g., when the error value is not the square of the error value or the magnitude of the error value), the average of the historical error values ​​may also be used to define the threshold value. In particular, the product of a predetermined value (e.g., 3 or 5) and the standard deviation may be added to the average of the historical error values ​​to define the threshold value.

[0127] This provides a highly accurate and adaptable dynamic threshold.

[0128] Preferably, the predetermined value is not less than 3, for example not less than 5. As the predetermined value increases, the false positive rate of detecting outliers decreases, and the false negative rate of detecting outliers increases. The predetermined value of 5 has been shown to provide particularly accurate and reliable outlier detection. However, if the potential (historical) error can be represented by a normal distribution or a semi-normal distribution or is assumed to be a normal distribution or a semi-normal distribution, the predetermined value can be defined as 3.

[0129] Those skilled in the art will well appreciate that the predetermined value used to calculate the threshold value may depend on the definition of the outlier data element. For example, in one embodiment of the present invention, a data element is considered an outlier if it has an error value that does not fall within a range within which a predetermined percentage of all error values ​​are expected to fall. The predetermined percentage may be, for example, 95%, 99%, 99.5%, 99.7%, etc. The user may provide the predetermined percentage from which the predetermined value is calculated using a standard procedure.

[0130] Thus, the predetermined value, and therefore the threshold, may depend on the percentage of error values ​​that are expected to be outliers (eg, 5%, 1%, 0.5%, 0.3%, etc.).

[0131] In other words, a data element may be considered an outlier if it is associated with an error value that does not fall within a range within which a predetermined percentage of error values ​​(for any data element in the time series data set) are expected to fall.

[0132] For half-normally distributed error values, it can be assumed that 95% of all error values ​​fall within 2 standard deviations of 0, and 99.7% of error values ​​fall within 3 standard deviations of 0. For normally distributed error values, it can be assumed that 95% of all error values ​​fall within 2 standard deviations of the mean (of historical error values), and 99.7% of error values ​​fall within 3 standard deviations of the mean (of historical error values).

[0133] Considering a scenario where each historical error value is the square of the error value, it can be assumed that the historical error values ​​have a normal distribution, and an outlier is defined as a data element whose associated error value does not fall within the normal expected limits of 99.7% of the data elements. In this example, the predetermined value would be 3, and the threshold would be 3 times the standard deviation.

[0134] It can be seen that the predetermined value may therefore depend on the probability of a data element having a value outside or within normal limits.

[0135] However, to ensure that the method is not constrained by the underlying distribution of historical errors, Chebyshev's inequality can be used. Compared to the 68-95-99 rule (also known as the empirical rule that applies only to normal distributions), Chebyshev's inequality guarantees that for a wide range of probability distributions, no more than a certain portion of the values ​​can be more than a certain distance away from the mean or 0 (only for positive error values).

[0136] Therefore, the principle of Chebyshev's inequality can be used to calculate a predetermined value to determine how many standard deviations away a distance value must be to be considered an outlier.

[0137] This may be based on defining what constitutes an abnormal error value. For example, an error value may be defined as an abnormal error value if the error value is greater than a threshold below which at least a predetermined percentage of expected error values ​​will fall. The predetermined percentage P1 may vary depending on the application (e.g., from a user input value), such as 95%, 99%, or 99.5%.

[0138] In one embodiment of the present invention, where k is a predetermined number and p1 is a predetermined percentage (expressed as a decimal), the predetermined number may be calculated by performing the following steps:

[0139]

[0140] The predetermined percentage P1 represents the percentage of error values ​​that are considered to be within the normal range, e.g., not associated with outlying data elements. Thus, Formula 1-p1 represents the percentage of error values ​​(e.g., the entire data set) that are considered to be outside the normal range (e.g., associated with an abnormal element). The percentage value may vary depending on the application.

[0141] This significantly improves the identification of appropriate thresholds for determining whether a data element is an outlier, thereby improving the accuracy and precision of identifying anomalous data elements. Methods that employ Chebyshev's inequality are more general than those that assume a normal distribution of error values.

[0142] In other embodiments of the present invention, sub-step 42 may use other algorithms to generate the threshold value. For example, sub-step 42 may include determining the mean / median / mode of the historical error values ​​and using that value to define the threshold value. For example, the mean / median / mode may be multiplied by a predetermined value (e.g., not greater than 2, such as approximately 1.5) to provide the threshold value.

[0143] However, a preferred embodiment of the present invention includes utilizing sub-step 43 and sub-step 42 including said sub-steps 44, 45, in which case the threshold is calculated based on the standard deviation of the scaled historical error values. This preferred embodiment provides extremely accurate and precise identification of outlying data elements.

[0144] Step 34 for determining whether the (investigated) data element is an outlier may depend on the calculation used in step 33 for generating the threshold value, in other words, the method of step 34 may complement the method of step 33 for generating the threshold value.

[0145] In an exemplary embodiment of the invention, an error value for the data element under investigation is calculated in the same manner as each historical error value for a second data element that temporally precedes the data element under investigation in the time series data set. The error value is compared to a threshold value generated using the historical error values.

[0146] For example, consider the case where the threshold value calculated in step 33 is based on the square of the error value of historical data (e.g., a second data element that temporally precedes the data element under investigation). Step 34 may include a corresponding sub-step 46 of calculating the square of the error value of the data element under investigation and a corresponding sub-step 47 of determining whether the data element is an outlier based on a comparison between the square of the error value (of the data element) and the threshold value.

[0147] In another example, consider a scenario in which step 33 includes a sub-step 43 of scaling the historical error value. Step 34 may include a corresponding sub-step 48 of scaling the error value of the data element (e.g., so that it is scaled to correspond to the historical error value). In such an embodiment, sub-step 48 preferably includes scaling the error value of the data element under investigation using the same maximum value of the scaled historical error value used to scale the historical error value. Thus, the historical error value and the error value under investigation may be normalized in the same manner.

[0148] In this way, the error value for the data element under investigation may correspond to the historical error value used to calculate the threshold value because it is calculated in a similar manner and scaled to correspond to the historical error value.

[0149] It is contemplated that the above embodiments may be applied iteratively to different data elements of a time series data set. In this manner, a different threshold may be calculated for each data element under investigation, for example, at each time step of the time series data set, so that each data element may be evaluated individually to determine whether it is an outlier.

[0150] In some embodiments of the invention, the calculated error value of the data element under investigation is stored for future iterations of the method, for example, to serve as a historical error value in future iterations. Thus, the calculated error value of the data element may be appended to a set of stored historical error values.

[0151] In a preferred embodiment of the present invention, the historical error values ​​obtained in step 41 are associated with the second data element corresponding to a specific time window. As previously mentioned, the data elements of the time series data set are associated with a specific time point or time period (e.g., they are associated with at least one timestamp). Therefore, each historical error value may also be associated with a specific time point or time period (e.g., they are also associated with at least one timestamp). Therefore, the historical error values ​​associated with a specific time window may be selected, for example, relative to the timestamp of the data element under investigation.

[0152] Preferably, the timestamp associated with the historical error value falls within a fixed time window that is located at a predetermined time length from the timestamp associated with the data element under investigation. For example, step 41 may include obtaining historical error values ​​that fall within a 10-minute window centered at a time 1 hour before the timestamp of the data element under investigation (e.g., ±5 minutes).

[0153] In an embodiment of the present invention, each historical error value obtained in step 41 may be associated with a time window that is located a plurality of predetermined time periods prior to the time associated with the data element under investigation.

[0154] As an example, consider the case where the predetermined time period is one hour, the length of the fixed time window is 10 minutes, and the data element under investigation is associated with a timestamp of 10:00:00. In this case, the timestamp associated with the historical error value obtained in step 41 may fall within any of the following windows: 09:00:00 (±0:05:00); 08:00:00 (±0:05:00); 07:00:00 (±0:05:00); and so on.

[0155] The length of the fixed time window may depend on the length of a predetermined time length from the timestamp associated with the data element under investigation. For example, the length of the time window may be a multiple (eg, 0.01x) of the predetermined time length from the timestamp.

[0156] The length of the predetermined time period is preferably hours, days or weeks. This allows exploitation of patterns in the time series data set that are typically associated with such time periods. This improves the accuracy and reliability of detecting outliers by taking into account periodic patterns in the time series data set.

[0157] Embodiments of the invention may also include the step of identifying periodic patterns in the historical error values ​​for the entire time series data set. This may be performed, for example, using a pattern recognition algorithm or procedure or a frequency analysis method (e.g., using a Fourier transform). Identifying periodic patterns in the historical error values ​​allows obtaining historical data values ​​associated with a periodic time window (e.g., within which the timestamps of the data elements under investigation fall).

[0158] In other embodiments of the present invention, the historical error value obtained corresponds to the error value of the data element in the time series data set that immediately precedes the data element under investigation (e.g., without any intervening data elements). Thus, for example, the most recent or last data element in the time series data set that immediately follows the second data element is the data element under investigation.

[0159] Embodiments of the present invention implement extreme value analysis by calculating a dynamic threshold for each data element of a time series data set. Such embodiments may include the step of determining the distance between the actual value of a data element and the predicted value of the data element, and reporting the actual value of the data element as an anomaly when the distance is above the dynamic threshold.

[0160] In such an embodiment, the distance is preferably the squared error between the actual value and the predicted value normalized (e.g., scaled) between 0 and 1. The dynamic threshold may be calculated separately for each data element (e.g., at different time steps of the time series data set) based on historical error values ​​associated with previous data elements of the time series data set. Even more preferably, the dynamic threshold is based on the historical squared error that has been normalized (e.g., scaled) between 0 and 1.

[0161] This provides a particularly accurate and precise method for identifying anomalous data elements or when a data element is associated with one or more outliers.

[0162] Figure 5 An embodiment of step 32 of method 30 is shown for predicting the value of a (under investigation) data element based on prediction data. The method is preferably combined with reference Figure 4The described embodiments are used together, however, it is contemplated that Figure 3 The embodiments are used together with the separate Figure 5 The method can improve the accuracy of identifying abnormal data elements.

[0163] Step 32 includes a sub-step 51 of predicting a corresponding plurality of predicted values ​​of the data element using the prediction data and a plurality of different prediction models. Thus, the plurality of prediction models are each used to predict a corresponding value of the data element.

[0164] Suitable prediction models for predicting the value of a data element are well known to those skilled in the art, such as those previously described. Typically, a prediction model takes a plurality of prior data elements as input, and processes these data elements to predict the value of a subsequent data element. Thus, the prediction data used by the prediction model preferably includes a plurality of data elements that precede the data element under investigation in time.

[0165] Of course, if the data element under study is multi-dimensional, and therefore includes more than one data value, then each predictive model may predict a corresponding set of predicted values ​​for that data element.

[0166] Step 32 then proceeds to sub-step 52 of identifying a prediction model that provides a prediction value closest to the value of the data element from the plurality of prediction values. In subsequent steps of method 30, the prediction value closest to the value of the data element is used in determining whether the data element is an outlier.

[0167] Preferably, sub-step 52 comprises calculating the root mean square (RMS) error between each of a plurality of predicted values ​​of the data element and the (actual) value of the data element. The predicted value associated with the lowest RMS error is selected as the predicted value for the future step.

[0168] In another example, step 32 instead includes using multiple prediction models to predict the total time series data set prior to the data element under investigation. The prediction model that generates a time series data set that is closest to the actual time series data set (e.g., as determined using RMS analysis) is selected to generate a predicted value for the data element under investigation.

[0169] Thus, step 32 may include utilizing a deep learning mechanism to identify one of a plurality of prediction models that best predicts the value of a data element or best models the overall time series data set.

[0170] Embodiments of the present invention recognize that if none of the multiple prediction models have managed to accurately predict the value of the data element under study, it is more likely that the data element is an anomaly or outlier. Therefore, by using the predicted value (of the multiple possible predicted values) that is closest to the actual / measured value of the data element, the accuracy and reliability of identifying anomalous data elements can be improved.

[0171] Furthermore, using multiple prediction models and selecting the best of these means that embodiments do not need to rely on a particular time series prediction model - such as a prediction model - which would otherwise affect the maximum efficacy of the method for identifying anomalous data elements. In particular, different time series data sets may be modeled better or more accurately by certain prediction models than by other prediction models. By using multiple prediction models, the most appropriate model may be identified, thereby reducing the likelihood of misidentifying a data element as an outlier.

[0172] A particularly advantageous embodiment of the invention employs reference Figure 4 Step 33 described and referenced Figure 5 Step 32 described, however, it will be apparent that these two concepts (e.g., Figure 4 and Figure 5 ) can be considered separate from each other.

[0173] Therefore, a method including reference Figure 3 and Figure 5 The steps described (for example, and not necessarily including reference to Figure 4 The method of the steps described above.

[0174] Fig. 6A 6 is a graph showing the effectiveness of the proposed embodiment of the present invention compared to the related art. Benchmarking was performed on each data element of a plurality of different data sets using six different methods of identifying abnormal data elements. Five of these methods (61, 62, 63, 65, 66) are consistent with the proposed embodiment of the present invention. The sixth method 64 corresponds to a method of the related art.

[0175] According to an embodiment of the present invention, the first method 61 comprises executing as shown in reference Figure 3 and Figure 4 The method described (eg, using historical error values ​​to calculate thresholds) wherein the prediction model used in step 32 for predicting the value of a data element is an ARIMA model.

[0176] According to an embodiment of the present invention, the second method 62 includes executing as shown in reference Figure 3 , Figure 4 and Figure 5 The method described, therefore, the second method 62 uses the historical error value to calculate the threshold. Step 32 of method 62 includes using the prediction data and a plurality of different prediction models to predict a corresponding plurality of potential prediction values ​​of the data element; and designating the potential prediction value closest to the value of the data element as the prediction value.

[0177] According to an embodiment of the present invention, the third method 63 further includes executing reference Figure 3 , Figure 4 and Figure 5 The method described, therefore, the third method 63 uses the historical error value to calculate the threshold value. Step 32 of method 63 includes using the prediction data and a plurality of different prediction models to predict a corresponding plurality of potential prediction values ​​of a plurality of data elements of the data set; and selecting a prediction model that generates a predicted value that most closely matches the (actual) value of the associated plurality of data elements of the data set to generate a predicted value for the data element.

[0178] According to an embodiment of the present invention, the fourth method 65 includes executing as shown in reference Figure 3 and Figure 4 The method described (eg, using historical error values ​​to calculate thresholds), wherein the prediction model used in step 32 for predicting data element values ​​is a single LSTM (LSTM-S) prediction model.

[0179] According to an embodiment of the present invention, the fifth method 66 includes performing the following steps: Figure 3 and Figure 4 The method described (eg, using historical error values ​​to calculate thresholds), wherein the prediction model used in step 32 for predicting the value of a data element is a deep LSTM (LSTM-D) prediction model.

[0180] According to the related art, method 64 includes performing outlier detection using a conventional EGADS model.

[0181] Fig. 6A The number of data sets for which each of the six methods 61-66 received a ranking score between 1 and 6 is shown, where a ranking of 1 represents the best method for identifying outliers in a particular data set for a given data set and a ranking of 6 represents the worst method for identifying outliers in a particular data set for a given data set.

[0182] To evaluate the relative ranking of the methods, the F1 score of each model was calculated, where the method associated with the highest F1 score was assigned rank 1, and the method with the lowest F1 score was assigned rank 6, with other ranks, e.g., ranks 2–5, distributed in between accordingly.

[0183] In other words, for each of the multiple benchmark data sets, methods 61-66 are applied to each data element of the data set, and then ranked based on a ranking evaluation method. Here, the ranking evaluation method includes calculating the F1 score of each method, and ranking the methods based on the size of the F1 score.

[0184] The Y-axis 6Y indicates the number of data sets for which the method received a particular ranking. The X-axis 6X represents the ranking.

[0185] The F1 score is defined by the following equation:

[0186]

[0187] Where “precision” is defined as the number of true anomalies found in a given dataset divided by the total number of anomalies found in that dataset, and the “recall” of a technique is defined as the ratio of the number of true anomalies found in that dataset to the total number of true anomalies in the dataset.

[0188] The datasets tested have a known number of anomalies (eg, they are benchmark datasets) so that the efficacy of a particular method can be easily determined.

[0189] It will be clearly seen that methods according to embodiments of the present invention are ranked higher on average than methods according to the prior art.

[0190] Figures 6B to 6D Other graphs are shown that illustrate the relative ranking of the above methods performed on the same multiple data sets with reference to other ranking evaluation methods.

[0191] As previously described, for each of the plurality of data sets, methods 61-66 are applied and ranked based on the ranking evaluation method. The total number of times a method receives a particular ranking is summed and reported in Figures 6A-6D Therefore, Figures 6A-6D The overall ranking of each method according to the ranking of different methods is shown.

[0192] exist Figure 6B In , the relative ranking of the models is evaluated by calculating the accuracy of each model, where the method associated with the highest F1 score is assigned a rank of 1, while the method with the lowest F1 score is assigned a rank of 6, and the other ranks are distributed among the remaining methods accordingly. "Accuracy" is defined as the number of true anomalies found in a given dataset divided by the total number of anomalies found in that dataset.

[0193] exist Figure 6C In , the relative ranking of the models is evaluated by calculating the recall of each model, where the method associated with the highest F1 score is assigned rank 1, while the method with the lowest F1 score is assigned rank 6, and the other ranks are distributed among the remaining methods accordingly.

[0194] exist Fig.6D In , the relative ranking of the models is evaluated by calculating the conventional ED score for each model, where the method associated with the highest F1 score is assigned rank 1, while the method with the lowest F1 score is assigned rank 6, and the other ranks are distributed among the remaining methods accordingly.

[0195] Figures 6A-6DIt is clearly shown that the embodiments of the present invention are consistently ranked higher than related art methods for identifying outliers. For example, Fig.6D It is shown how the related art methods deviate from performing well to performing poorly, which is avoided by using embodiments of the present invention.

[0196] In some embodiments of the present invention, a system may be provided, the system comprising: Figures 3 to 5 A processing apparatus as hereinbefore described.

[0197] As an example, Figure 7 As shown, an embodiment of the present invention may include a computer system 70, which may form part of a networked system 7, and the components of the computer system / server 70 may include, but are not limited to, one or more processing devices, such as a processor or processing unit 71, a system memory 74, and a bus 90 that couples various system components including the system memory 74 to the processing unit 71.

[0198] Bus 90 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, these architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0199] The computer system / server 70 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by the computer system / server 70, and it includes volatile and non-volatile media, removable and non-removable media.

[0200] The system memory 74 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 75 and / or cache memory 76. The computer system / server 70 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 77 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, and commonly referred to as a "hard drive"). Although not shown, a disk drive for reading from and writing to a removable, non-volatile disk (e.g., a "floppy disk") may be provided, as well as an optical drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media. In this case, each may be connected to the bus 90 via one or more data media interfaces. As will be further depicted and described below, the memory 74 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0201] A program / utility 78 having a set (at least one) of program modules 79, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in the memory 74 as an example and not by way of limitation. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. The program modules 79 generally perform the functions and / or methods of the embodiments of the present invention described herein.

[0202] The computer system / server 70 may also communicate with one or more external devices 80, such as a keyboard, pointing device, display 85, etc.; one or more devices that enable a user to interact with the computer system / server 70; and / or any device that enables the computer system / server 70 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 72, however, the computer system / server 70 may communicate with one or more networks, such as a local area network (LAN), a general purpose wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 73. As shown, the network adapter 73 communicates with other components of the computer system / server 70 via a bus 90, and it should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system / server 70, examples of which include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0203] Figure 8A block diagram of a system 800 for determining whether a data element having a value in a time series data set 805 is an outlier is shown.

[0204] The system 800 comprises a first obtaining unit 810 adapted to obtain prediction data from first data of the time series dataset 805 temporally preceding the data element. Thus, the first obtaining unit may be adapted to extract prediction data for predicting the value of the data element from the time series dataset.

[0205] The system further comprises a prediction unit 820 adapted to predict a value of a data element. In particular, the prediction unit uses the prediction data to predict a value of the data element. Thus, the prediction unit generates a predicted value of the data element.

[0206] In some embodiments of the invention, the prediction unit 820 is adapted to use the prediction data and a plurality of different prediction models to predict a corresponding plurality of predicted values ​​of the data element; and to identify the predicted value closest to the value of the data element from the plurality of predicted values. Thus, the prediction unit may use a plurality of different prediction models / methods to predict corresponding predicted values, and select the predicted value closest to the (actual) value of the data element as the predicted value for use by future units of the system.

[0207] The system 800 further comprises a second obtaining unit 830. The second obtaining unit is adapted to obtain a threshold value, said threshold value being used to define 226 the value of a data element considered to be an outlier relative to a predicted value of the data element.

[0208] In some embodiments of the present invention, the second obtaining unit 830 is suitable for obtaining (multiple) historical error values ​​of a time series data set, each historical error value representing the difference between a value of a second data element of the time series data set that is chronologically previous to the data element and a predicted value of the second data element; and obtaining a threshold value based on the one or more historical error values, wherein the threshold value is used to define the value of the data element that is considered to be an outlier relative to the predicted value of the data element.

[0209] The obtained historical error value may be obtained from a storage set 809 of historical error values ​​(e.g., selected from the storage set). In other examples, the historical error value is calculated using (one or more) prediction models to predict the value of the second data element, and the error value is calculated using the predicted value and the (actual) value of the second data element. Thus, the historical error value may be calculated based on the second data element found in the time series data set 805.

[0210] The system 800 further comprises a determination unit 840 adapted to determine whether a data element is an outlier based on a threshold value (from the second obtaining unit 830), the identified predicted value (from the prediction unit 820) and the (actual) value of the data element. The (actual) value of the data element can be obtained directly from the time series data set 805.

[0211] In this manner, system 800 can determine whether a data element of a time series data set is an outlier.

[0212] Fig. 9 A general architecture 900 or block diagram of a method / system for identifying whether a data element of a time series data set 901 is an outlier according to a preferred embodiment of the present invention is shown.

[0213] The data set 901 may be an offline data set (eg, stored data) or a real-time stream.

[0214] In block 902, the method / system may obtain a data element from a data set. The method / system may also apply a plurality of prediction models 903 to the data elements of the data set to generate a corresponding plurality of predicted values ​​for the data elements. As previously explained, the prediction models are applied to data elements in the data set 901 that are temporally preceding the data element (obtained in block 902) in the data set.

[0215] The method / system may include merging prediction models to identify the best simulation data element and / or prediction model for the time series data set in block 904. This may be performed by identifying a prediction model that generates a predicted value for the data element that is closest to the actual value of the data element.

[0216] In block 905, a threshold value for the data element may be generated from the data set 901. In a particular embodiment, a historical error value is obtained from the data set and used to generate the threshold value (as previously described). In a further example, block 905 may include generating the threshold value using the historical error value for the data element associated with a particular time window. The particular time window may be temporally located or associated with a predetermined time period prior to a time associated with the data element under investigation.

[0217] In block 906, it is determined whether the data element (obtained in step 902) is an outlier. This may be performed by calculating an error value representing the difference between the actual value of the data element and the predicted value obtainable from the identified predictive model of block 904. The error value may be compared to the threshold value obtained in block 905 to determine whether the data element is an outlier.

[0218] therefore, Fig. 9is a block diagram of an embodiment of the present invention, in which it is identified whether a data element (under investigation) is an anomaly / outlier. As described later, each block may represent a module or unit of a processing system, or a step of a method according to an embodiment.

[0219] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the present invention.

[0220] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0221] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0222] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0223] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0224] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0225] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0226] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0227] The description of various embodiments of the present invention has been given for the purpose of illustration, but it is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for determining whether a data element having a value of a time series data set is an outlier, the method comprising: Obtaining prediction data for predicting a value of the data element from first data that precedes the data element in time in the time series data set; using the prediction data to predict a predicted value of the data element; obtaining an error value for the data element representing a difference between the value and the predicted value for the data element; Obtaining historical error values ​​of the time series data set, each historical error value representing a difference between a value of a second data element of the time series data set that temporally precedes the data element and a predicted value of the second data element, wherein the second data element corresponds to a data element of the time series data set obtained at a specific predetermined time period before the data element; Obtaining a threshold value of the error value of the data element based on one or more of the historical error values, the threshold value being used to define an error value of the data element that is considered an outlier, wherein obtaining the threshold value comprises: determining the predetermined number based on a percentage of error values ​​that are expected to be outliers; multiplying a statistical measure of the historical error values ​​by the predetermined number to generate a result, wherein the statistical measure comprises one of a mean, a median, a mode, and a standard deviation; and determining the threshold based on the result, wherein a different threshold is determined for each data element in the time series data set; and A determination is made whether the data element is an outlier based on a comparison of the threshold and the error value of the data element. 2 . The computer-implemented method of claim 1 , wherein the forecast data comprises a predetermined number of other data elements of the time series data set that precede the data element in time.

3. A computer-implemented method as described in claim 1, wherein each historical error value is the square of an error value, wherein the square of the error value is the square of the difference between the value of the second data element that precedes the data element in time in the time series data set and the corresponding predicted value of the second data element of the time series data set. 4 . The computer-implemented method of claim 1 , wherein obtaining a threshold value comprises scaling each historical error value to a value between 0 and 1 based on a maximum value of the historical error values. 5 . The computer-implemented method of claim 1 , wherein obtaining the threshold comprises calculating the predetermined number using Chebyshev's inequality. The computer-implemented method of claim 1 , wherein the predetermined time period is an hour, a day, or a week.

7. The computer-implemented method of claim 1 , wherein predicting a predicted value of the data element comprises: using the prediction data and a plurality of different prediction models to predict a corresponding plurality of potential prediction values ​​for the data elements; as well as The potential predicted value that is closest to the value of the data element is designated as the predicted value.

8. The computer-implemented method of claim 1, wherein the data elements are data points of the time series data set.

9. A computer program product for determining whether a data element having a value of a time series data set is an outlier, the computer program product comprising a computer readable storage medium having program instructions contained therein, the program instructions being executable by a processing unit to cause the processing unit to perform a method comprising: Obtaining prediction data for predicting a value of the data element from first data that precedes the data element in time in the time series data set; using the prediction data to predict a predicted value of the data element; obtaining an error value for the data element representing a difference between the value and the predicted value for the data element; Obtaining historical error values ​​of the time series data set, each historical error value representing a difference between a value of a second data element of the time series data set that temporally precedes the data element and a predicted value of the second data element, wherein the second data element corresponds to a data element of the time series data set obtained at a specific predetermined time period before the data element; Obtaining a threshold value of the error value of the data element based on one or more of the historical error values, the threshold value being used to define an error value of the data element that is considered an outlier, wherein obtaining the threshold value comprises: determining the predetermined number based on a percentage of error values ​​that are expected to be outliers; multiplying a statistical measure of the historical error values ​​by the predetermined number to generate a result, wherein the statistical measure comprises one of a mean, a median, a mode, and a standard deviation; and determining the threshold based on the result, wherein a different threshold is determined for each data element in the time series data set; and A determination is made whether the data element is an outlier based on a comparison of the threshold and the error value of the data element.

10. A computer program product as described in claim 9, wherein each historical error value is the square of an error value, wherein the square of the error value is the square of the difference between the value of the second data element that precedes the data element in time in the time series data set and the corresponding predicted value of the second data element of the time series data set.

11. The computer program product of claim 9, wherein obtaining the threshold value comprises scaling each historical error value to a value between 0 and 1 based on a maximum value of the historical error values.

12. The computer program product of claim 9, wherein predicting a predicted value of the data element comprises: using the prediction data and a plurality of different prediction models to predict a corresponding plurality of potential prediction values ​​for the data elements; as well as The potential predicted value that is closest to the value of the data element is designated as the predicted value.

13. A computer system for determining whether a data element having a value of a time series data set is an outlier, the computer system comprising: At least one processor configured to: Obtaining prediction data for predicting a value of the data element from first data that precedes the data element in time in the time series data set; using the prediction data to predict a predicted value of the data element; obtaining an error value for the data element representing a difference between the value and the predicted value for the data element; Obtaining historical error values ​​of the time series data set, each historical error value representing a difference between a value of a second data element of the time series data set that temporally precedes the data element and a predicted value of the second data element, wherein the second data element corresponds to a data element of the time series data set obtained at a specific predetermined time period before the data element; Obtaining a threshold value of the error value of the data element based on one or more of the historical error values, the threshold value being used to define an error value of the data element that is considered an outlier, wherein obtaining the threshold value comprises: determining the predetermined number based on a percentage of error values ​​that are expected to be outliers; multiplying a statistical measure of the historical error values ​​by the predetermined number to generate a result, wherein the statistical measure comprises one of a mean, a median, a mode, and a standard deviation; and determining the threshold based on the result, wherein a different threshold is determined for each data element in the time series data set; and A determination is made whether the data element is an outlier based on a comparison of the threshold and the error value of the data element.

14. A computer system as described in claim 13, wherein each historical error value is the square of an error value, and the square of the error value is the square of the difference between the value of the second data element that precedes the data element in time of the time series data set and the corresponding predicted value of the second data element of the time series data set.

15. The computer system of claim 13, wherein predicting a predicted value of the data element comprises: using the prediction data and a plurality of different prediction models to predict a corresponding plurality of potential prediction values ​​for the data elements; as well as The potential predicted value that is closest to the value of the data element is designated as the predicted value.

16. A system for determining whether a data element having a value of a time series data set is an outlier, the system comprising: At least one processor, comprising: a first obtaining unit adapted to obtain prediction data for predicting a value of the data element from first data that temporally precedes the data element in the time series data set; a prediction unit adapted to predict a predicted value of the data element using the prediction data; The second acquisition unit is suitable for: obtaining an error value for the data element representing a difference between the value and the predicted value for the data element; Obtaining historical error values ​​of the time series data set, each historical error value representing a difference between a value of a second data element of the time series data set that temporally precedes the data element and a predicted value, wherein the second data element corresponds to a data element of the time series data set obtained during a specific predetermined period of time before the data element; and Obtaining a threshold value of the error value of the data element based on one or more of the historical error values, the threshold value being used to define an error value of the data element that is considered an outlier, wherein obtaining the threshold value comprises: determining the predetermined number based on a percentage of error values ​​that are expected to be outliers; multiplying a statistical measure of the historical error values ​​by the predetermined number to generate a result, wherein the statistical measure comprises one of a mean, a median, a mode, and a standard deviation; and determining the threshold based on the result, wherein a different threshold is determined for each data element in the time series data set; and A determination unit is adapted to determine whether the data element is an outlier based on a comparison between the threshold and an error value of the data element.

17. A system according to claim 16, wherein each historical error value is the square of an error value, wherein the square of the error value is the square of the difference between the value of the second data element that precedes the data element in time in the time series data set and the corresponding predicted value of the second data element of the time series data set.

18. The system of claim 16, wherein predicting the predicted value of the data element comprises: using the prediction data and a plurality of different prediction models to predict a corresponding plurality of potential prediction values ​​for the data elements; as well as The potential predicted value that is closest to the value of the data element is designated as the predicted value.

Images