Automatic parameter adjustment for anomaly detection systems

Abstract

A parameter adjustment system can obtain a set of different value combinations for one or more parameters associated with an anomaly detection system and obtain one or more time series monitored by the anomaly detection system. In implementations, each data point of the one or more time series can be labeled with a label indicating presence, absence, or undetermined of an anomaly. The parameter adjustment system can execute the anomaly detection system assigned with each value combination on a subset of data points of the one or more time series to obtain predicted labels for the subset of data points, compute a performance score for the anomaly detection system assigned with the corresponding value combination based on a predetermined evaluation metric, and select a parameter combination corresponding to a highest performance score as a recommended parameter combination for the anomaly detection system.

Description

BACKGROUND

[0001] With the explosive development of computer technology, many computer systems have been developed and used in various application domains for monitoring and control purposes. For example, an anomaly detection system can be deployed and configured to monitor performance metrics (such as percentage of CPU usage, percentage of memory usage, etc.) of a plurality of servers in a cloud computing architecture and detect any occurrence of various types of irregularities or anomalies (e.g., sudden jumps or spikes in traffic volume, failure of a certain server, etc.) in the cloud computing architecture. To accurately detect the occurrence of anomalies, a human expert familiar with the anomaly detection algorithm is usually recruited to manually or semi-manually configure or tune parameters of the anomaly detection system to set the optimal configuration of parameters for the anomaly detection algorithm in the anomaly detection system. However, due to the large amount of time series and data points collected for performance metrics, it is impractical, if not impossible, to involve an expert to configure and tune the optimal parameters for each performance metric. Without the involvement of a human expert, the performance (e.g., accuracy) of the anomaly detection system can be affected and cannot be guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0002] The detailed description is set forth with reference to the accompanying drawings. In the drawings, the left-most digit(s) of each reference number identifies the first figure in which the reference number appears. The use of the same reference numbers in different figures indicates similar or identical items.

[0003] Figure 1 An example environment in which a parameter tuning system can be used is shown.

[0004] Figure 2 An example anomaly detection system is shown.

[0005] Figure 3 An example parameter tuning system is shown.

[0006] Figure 4 An example parameter tuning method is shown. DETAILED DESCRIPTION

[0007] SUMMARY

[0008] As described above, existing anomaly detection systems require a human expert to manually or semi-manually tune parameters of the anomaly detection system to set the optimal configuration of parameters needed to detect anomalies in time series data associated with everyday life operations such as performance of servers provided in the cloud. This is impractical and inefficient for a human expert to be involved in manually setting the optimal parameters for each metric, thus limiting the possibility of anomaly detection systems to scale up to monitor a large number of machines provided in computer systems.

[0009] The present disclosure describes an example parameter adjustment system. The parameter adjustment system can automatically adjust and configure parameters associated with an anomaly detection system to obtain an optimal configuration of parameters that can be used for anomaly detection in the anomaly detection system, and further adaptively adjust the parameters as the anomaly detection system collects new time series data.

[0010] In implementations, the parameter adjustment system can obtain a set of different value combinations for one or more parameters associated with an anomaly detection system, and obtain one or more time series monitored by the anomaly detection system. In implementations, each data point of the one or more time series can be labeled with a label indicating presence, absence, or undetermined of an anomaly. In implementations, the parameter adjustment system can assign a respective value combination of the set of different value combinations to the one or more parameters associated with the anomaly detection system, and apply the anomaly detection system assigned with the respective value combination to a subset of data points of the one or more time series to obtain, for each value combination, predicted labels for the subset of data points. In implementations, the parameter adjustment system can then compute a performance score for the anomaly detection system assigned with the respective value combination based at least in part on a predetermined evaluation metric, and select, from the set of different parameter combinations, a parameter combination corresponding to a highest performance score of the anomaly detection system as a recommended parameter combination for the one or more parameters associated with the anomaly detection system. In implementations, the parameter adjustment system can compute the performance score based on the predicted labels and labels provided by a user.

[0011] In implementations, the functions described herein to be performed by the parameter adjustment system can be performed by a plurality of separate units or services. For example, a retrieval service can obtain a set of different value combinations for one or more parameters associated with an anomaly detection system, and obtain one or more time series monitored by the anomaly detection system, while a detection service can assign a respective value combination of the set of different value combinations to the one or more parameters associated with the anomaly detection system, and apply the anomaly detection system assigned with the respective value combination to a subset of data points of the one or more time series to obtain, for each value combination, predicted labels for the subset of data points. An evaluation service can compute a performance score for the anomaly detection system assigned with the respective value combination based at least in part on a predetermined evaluation metric, and select, from the set of different parameter combinations, a parameter combination corresponding to a highest performance score of the anomaly detection system as a recommended parameter combination for the one or more parameters associated with the anomaly detection system.

[0012] Further, while in examples described herein the parameter adjustment system can be implemented as a combination of software and hardware installed in a single device, in other examples the parameter adjustment system can be implemented and distributed among multiple devices, or as a service provided in one or more computing devices and / or in a cloud computing architecture over a network.

[0013] This application describes a number of different embodiments and implementations. The following sections describe an exemplary framework suitable for practicing various implementations. Next, this application describes exemplary systems, devices, and processes for implementing a parameter adjustment system.

[0014] Exemplary Environment

[0015] Figure 1 An exemplary environment 100 that can be used to implement a parameter adjustment system is shown. The environment 100 can include a parameter adjustment system 102. In this example, the parameter adjustment system 102 is described as existing as a separate entity. In some instances, the parameter adjustment system 102 can include one or more servers 104. In other instances, the parameter adjustment system 102 can be included as part of, or distributed among, one or more servers 104 that communicate data with each other via a network 106. In implementations, a first one of the one or more servers 104 can include part of the functionality of the parameter adjustment system 102, while the functionality of the parameter adjustment system 102 can be included in a second one of the one or more servers 104. Further, in some implementations, some or all of the functionality of the parameter adjustment system 102 can be included in a cloud computing system or architecture, and can be provided as a service for determining or recommending a suitable or optimal parameter configuration for anomaly detection.

[0016] In implementations, the parameter adjustment system 102 can be part of a client device 108, e.g., a software and / or hardware component of the client device 108. In some instances, the parameter adjustment system 102 can include the client device 108.

[0017] In implementations, the environment 100 can also include an anomaly detection system 110. In implementations, the parameter adjustment system 102 can be included in the anomaly detection system 110 and provide services to the anomaly detection system 110. In some implementations, some or all of the functionality of the parameter adjustment system 102 can be included and provided in one or more servers 104, client devices 108, and / or anomaly detection systems 110 that communicate with each other via the network 106.

[0018] The client device 108 can be implemented as any of various computing devices, including but not limited to a desktop computer, a notebook or laptop computer, a handheld device, a netbook, an Internet appliance, a tablet computer or tablet touch computer, a mobile device (e.g., a mobile phone, a personal digital assistant, a smart phone, etc.), etc., or a combination thereof.

[0019] The network 106 can be a wireless or a wired network, or a combination thereof. The network 106 can be a collection of individual networks that are interconnected and act as a single large network (e.g., the Internet or an intranet). Examples of such individual networks include, but are not limited to, a telephone network, a cable network, a local area network (LAN), a wide area network (WAN), and a metropolitan area network (MAN). Further, the individual networks can be wireless or wired networks, or a combination thereof. The wired networks can include electrical carriers connections (such as communication cables, etc.) and / or optical carriers or connections (such as fiber optic connections, etc.). The wireless networks can include, for example, WiFi networks, other radio frequency networks (e.g., Bluetooth® Zigbee, etc.), etc.

[0020] In an implementation, the parameter adjustment system 102 can receive, from the anomaly detection system 110, one or more parameters to adjust and the time series data with the labeled data points. The parameter adjustment system 102 can then generate an adjustment space based on different value combinations of the one or more parameters to adjust and the labeled data points of the time series data, and perform anomaly detection on the labeled data points using the different value combinations of the one or more parameters. The parameter adjustment system 102 can evaluate respective performances of the different value combinations of the one or more parameters based on a predetermined evaluation metric, and determine a particular value combination to recommend to the anomaly detection system as a recommended value combination of the one or more parameters.

[0021] Example anomaly detection system

[0022] Figure 2 The anomaly detection system 110 is shown in more detail. In an implementation, the anomaly detection system 110 can include, but is not limited to, one or more processors 202, input / output (I / O) interfaces 204 and / or network interfaces 206, and a memory 208.

[0023] ​In implementations, the processor 202 can be configured to execute instructions stored in the memory 208 and / or received from the input / output interface 204 and / or the network interface 206. In implementations, the processor 202 can be embodied as one or more hardware processors including, for example, a microprocessor, an application-specific instruction-set processor, a physical processing unit (PPU), a central processing unit (CPU), a graphics processing unit, a digital signal processor, a tensor processing unit, etc. Additionally or alternatively, the functionality described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip system (SOC), a complex programmable logic device (CPLD), etc.

[0024] The memory 208 can include computer-readable media in the form of volatile memory, such as random-access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory 208 is an example of computer-readable media.

[0025] Computer-readable media can include volatile or non-volatile memory, removable or non-removable media, implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other internal storage technology, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. As defined herein, computer-readable media does not include any transitory medium, such as modulated data signals and carrier waves.

[0026] Although only hardware components are described in this example in the anomaly detection system 110, in other instances, the anomaly detection system 110 can also include other hardware components and / or other software components, such as program modules 210 for executing instructions stored in the memory 208 for performing various operations, and program data 212 for storing data associated with anomaly detection, such as one or more time series of data points, parameter values associated with anomaly detection, etc.

[0027] By way of example, and not limitation, program modules 210 can include a data pre-processing module 214 configured to inspect and clean the input time series, such as inspecting time stamps of data points in the input time series, inspecting any missing data points, performing interpolation to fill in missing data points, and the like. In embodiments, program modules 210 can also include a classification module 216 configured to separate data of at least some portions of the input time series into different types based on respective data patterns included in the at least some portions of the input time series. In embodiments, program modules 210 can also include a transformation module 218 configured to process the input time series by, for example, denoising and smoothing the input time series and decomposing the input time series into different components including, but not limited to, a trend component, a seasonal component, and a residual component, and the like. In embodiments, program modules 210 can also include a detection module 220 configured to detect and determine whether an anomaly occurs in the input time series by applying one or more statistical hypothesis tests, such as T-test, F-test, and / or MK-test, to the different components obtained by the transformation module 218.

[0028] Example parameter adjustment system

[0029] Figure 3 An example parameter adjustment system 102 is shown in more detail. In embodiments, parameter adjustment system 102 can include, but is not limited to, one or more processors 302, input / output (I / O) interfaces 304, network interfaces 306, and memory 308. In embodiments, memory 308 can include computer-readable media as described in the above description. In embodiments, parameter adjustment system 102 can also include an evaluator 310 and an adjuster 312. Additionally, in some embodiments, parameter adjustment system 102 can also include more other devices. In embodiments, some functions of parameter adjustment system 102 can be implemented using hardware, such as ASICs (i.e., application specific integrated circuits), FPGAs (i.e., field programmable gate arrays), and / or other hardware. In this example, parameter adjustment system 102 is described as existing as a separate entity. In some cases, some or all functions of parameter adjustment system 102 can be included in anomaly detection system 110, one or more servers 104, and / or client devices 108.

[0030] Although only hardware components are described in the parameter adjustment system 102 in this example, in other instances, the parameter adjustment system 102 can also include other hardware components and / or other software components, such as program modules 314 for executing instructions stored in the memory 308 for performing various operations, and program data 316 for storing data associated with parameter adjustment, such as one or more time series of labeled data points, combinations of values of parameters associated with the anomaly detection system 110, etc.

[0031] Exemplary method

[0032] Figure 4 A schematic diagram depicting an exemplary method of parameter adjustment is shown. Figure 4 The method of Figure 1 may, but need not be, implemented in the environment of Figure 2 and using the system of 3 For ease of explanation, the method 400 is described with reference to Figures 1 to 3 However, the method 400 can instead be implemented in other environments and / or using other systems.

[0033] The method 400 is described in the general context of computer- executable instructions. Generally, computer-executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, and the like that perform particular functions or implement particular abstract data types. Furthermore, each of the example methods can be implemented as a set of blocks in a logic flowchart which represents a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof. The order in which the methods are described is not intended to be construed as a limitation, and any number of the described methods can be combined in any order to implement the method, or an alternate method. Additionally, individual blocks can be omitted from the method without departing from the spirit and scope of the subject matter described herein. In the context of software, the blocks represent computer- instructions that, when executed by one or more processors, perform the recited operations. In the context of hardware, some or all of the blocks can represent special-purpose integrated circuits (ASICs) or physical components that perform the recited operations.

[0034] Referring back to Figure 4 , at block 402, the parameter adjustment system 102 can receive information of one or more parameters to be adjusted for the anomaly detection system and one or more time series of labels.

[0035] In implementations, the parameter adjustment system 102 can be connected or associated with one or more anomaly detection systems (e.g., the anomaly detection system 110), and can communicate data with the one or more anomaly detection systems through the network 106. In implementations, when one or more parameters associated with the anomaly detection system 110 need to be configured or adjusted, the parameter adjustment system 102 can receive information of the one or more parameters to be adjusted from the anomaly detection system 110 for automatic parameter adjustment. In addition, the parameter adjustment system 102 can receive one or more labeled time series including labeled data points from the anomaly detection system 102. The one or more labeled time series can correspond to performance data of a computer system (such as percentage of CPU usage, percentage of memory usage, etc.) periodically collected and monitored by the anomaly detection system 110.

[0036] In implementations, the one or more parameters to be adjusted can include, but are not limited to, parameters associated with one or more statistical hypothesis tests used for anomaly detection. Examples of the parameters associated with the statistical tests can include length of time series (including window size of data points to be tested), direction of statistical test (e.g., one-sided test or two-sided test), significance level of statistical test, etc. In implementations, the one or more statistical hypothesis tests can include, but are not limited to, T-test, F-test, and MK-test, etc. For ease of description and without loss of generality, the significance level of the one or more statistical hypothesis tests involved in anomaly detection is used as an example of the one or more parameters to be adjusted hereinafter. However, the types of parameters associated with the anomaly detection system, such as length of time series (including window size of data points to be tested), direction of statistical test (e.g., one-sided test or two-sided test), etc. can also be adjusted and configured in a similar manner.

[0037] In implementations, the information of the one or more parameters to be adjusted can include initial values or default values of the one or more parameters used in the anomaly detection system 110, and these initial values or default values of the one or more parameters can not be optimal or can be outdated, and thus need to be adjusted or improved.

[0038] In implementations, prior to sending information of one or more parameters to be adjusted to the parameter adjustment system 102, the anomaly detection system 110 can first automatically separate data in at least some portions of the one or more time series into different types (e.g., such as spike, level shift, mean change, etc. anomaly types) by the classification module 216 using an active learning-based method based on respective data patterns in the at least some portions of the one or more time series and / or by user selection. Then, the anomaly detection system 110 can select representative samples from respective data of each type. In implementations, each representative sample can include a predetermined number of data points in the time series. Then, the anomaly detection system 110 can transform each representative sample of each type by the transformation module 218 and perform anomaly detection (e.g., apply corresponding one or more statistical hypothesis tests) on these representative samples of each type by the detection module 220 to generate anomaly scores for these representative samples.

[0039] By way of example, and not limitation, the anomaly scores of these representative samples can be related to probability values (i.e., p-values) of these representative samples generated for one or more statistical hypothesis tests. By way of example, and not limitation, the anomaly score of a representative sample under a statistical test can be a negative logarithm of a p-value generated for the representative sample under the statistical test with base 10. In statistical hypothesis testing, a p-value or probability value is defined as a probability of obtaining a test result at least as extreme as the result actually observed during the statistical hypothesis test. In implementations, the anomaly detection system 110 can compare the p-values of the representative samples generated for one or more statistical hypothesis tests with respective significance levels of the one or more statistical hypothesis tests to determine whether an anomaly occurs in each of the representative samples.

[0040] In implementations, the anomaly detection system 110 can provide the representative sample to one or more users and request feedback from the one or more users to label at least some of the data points in the representative sample as an anomaly point and / or a non-anomaly point. In some implementations, the anomaly detection system 110 can also provide the anomaly scores of the representative sample to the one or more users as further reference. Based on the feedback from the one or more users, the representative sample can be labeled accordingly. For example, the one or more users can label whether there is an anomaly in the data points in the representative sample. In implementations, each data point of the representative sample can be labeled with a label indicating the presence, absence, or undetermined of an anomaly. Specifically, a data point determined by a user as an anomaly point (or a data point with an anomaly) is labeled with a label indicating the presence of an anomaly (e.g., anomaly). A data point determined by a user as a non-anomaly point (or a data point without an anomaly) is labeled with a label indicating the absence of an anomaly (e.g., non-anomaly). A data point that is not determined by any user as an anomaly point or a non-anomaly point is labeled with a label indicating that the anomaly is undetermined or has not been determined (e.g., undetermined).

[0041] In implementations, the anomaly detection system 110 can select a portion of the representative sample and the corresponding anomaly scores (e.g., p-values) and send the selected portion of the representative sample and the corresponding anomaly scores as at least a portion of the data associated with the one or more parameters to be adjusted to the parameter adjustment system 102.

[0042] At block 404, the parameter adjustment system 102 can generate a set of different value combinations for the one or more parameters associated with the anomaly detection system based on the received information of the one or more parameters to be adjusted.

[0043] In implementations, in response to receiving the information of the one or more parameters to be adjusted from the anomaly detection system 110, the parameter adjustment system 102 can obtain the respective candidate values of the one or more parameters from the received information of the one or more parameters to be adjusted. In this example, the one or more parameters are described as one or more significance levels of one or more statistical hypothesis tests, and thus the respective candidate values of the one or more parameters can include the respective p-values of the data points labeled with a label indicating the presence of an anomaly obtained under the one or more statistical hypothesis tests.

[0044] In implementations, the parameter adjustment system 102 can combine the respective candidate values of the one or more parameters to form a set of different value combinations of the one or more parameters to be adjusted. For example, if N data points are detected and labeled as anomaly points or normal points under a first statistical test T1, where the corresponding p-values are generated for the first statistical test T1 are M data points are detected and labeled as anomaly points or normal points under a second statistical test T2, where the corresponding p-values are generated for the second statistical test T2 are In implementations, the parameter adjustment system 102 can combine the respective candidate values of the one or more parameters to form a set of different value combinations of the one or more parameters to be adjusted. For example, if N data points are detected and labeled as anomaly points or normal points under a first statistical test T1, where the corresponding p-values are generated for the first statistical test T1 are detected and labeled as abnormal, where the corresponding p-value generated for the second statistical test T2 is and K data points under the third statistical test T3 detected and labeled as having an anomaly, where the corresponding p-value generated for the third statistical test T3 is The set of different value combinations of the one or more parameters to be adjusted (in this example, there are three parameters, i.e., the significance levels of the first, second, and third statistical hypothesis tests) generated by the parameter adjustment system 102 can include N x M x K different value combinations, i.e., where 1≤n≤N, 1≤m≤M, and 1≤k≤K. N, M, and K are positive integers greater than or equal to 1.

[0045] In embodiments, the parameter adjustment system 102 can further generate an adjustment space from the set of different value combinations of the one or more parameters to be adjusted and the one or more labeled time series. For example, after generating the set of different value combinations of the one or more parameters to be adjusted, the parameter adjustment system 102 can further generate an adjustment space using the set of different value combinations of the one or more parameters to be adjusted and the one or more labeled time series. In embodiments, the adjustment space can include a parameter space consisting of the set of different value combinations and a data space consisting of the one or more labeled time series.

[0046] At block 406, the parameter adjustment system 102 can send the set of different value combinations of the one or more parameters to be adjusted to the anomaly detection system to perform anomaly detection on the one or more labeled time series based on each value combination of the one or more parameters to be adjusted.

[0047] In embodiments, the parameter adjustment system 102 can send the set of different value combinations to the anomaly detection system 110 to cause the anomaly detection system 110 to perform anomaly detection on the data points of the one or more labeled time series based on each value combination (in this example, the combination of candidate values for the significance levels of the statistical hypothesis tests) and obtain the prediction results or labels for the data points of the one or more labeled time series.

[0048] At block 408, the parameter adjustment system 102 can receive the prediction results or labels for the data points of the one or more time series from the anomaly detection system.

[0049] In an implementation, after the anomaly detection system 110 performs anomaly detection on the data points of the one or more labeled time series based on each value combination of the set of different value combinations of the one or more parameters to be adjusted, the anomaly detection system 110 can return the prediction results or labels of the data points of the one or more labeled time series to the parameter adjustment system 102. In an implementation, each prediction result or label can include a label indicating whether an anomaly is predicted to exist or not exist.

[0050] At block 410, the parameter adjustment system 102 can calculate a respective performance score for each value combination of the set of different value combinations of the one or more parameters to be adjusted.

[0051] In an implementation, upon receiving the prediction results or labels of the data points of the one or more time series from the anomaly detection system, the parameter adjustment system 102 or the evaluator 310 of the parameter adjustment system 102 can evaluate a respective performance score for each value combination of the set of different value combinations of the one or more parameters to be adjusted based on the prediction results or labels made by the one or more users on the data points of the one or more labeled time series and the user labels of the data points of the one or more labeled time series. In an implementation, the parameter adjustment system 102 can calculate an enhanced confusion matrix as shown in Table 1 below.

[0052] Table 1: Enhanced confusion matrix

[0053] Predicted as abnormal Predicted as not abnormal Labeled as abnormal TP FN Labeled as not abnormal FN N ]]> TP N ]]> Not labeled FP TN

[0054] In an implementation, the parameter adjustment system 102 can also employ a lenient mode of anomaly matching between the predicted labels and the user labels of the data points of the one or more labeled time series. For example, a data window of a predetermined size (i.e., a predetermined integer such as 2, 3, 4,... etc.) can be used to accommodate a pattern anomaly in which multiple consecutive data points are labeled as anomalies. This corresponds to a situation in which a data point labeled with a predicted anomaly is closed by several data points (i.e., earlier or later) compared to a data point labeled as an anomaly by a user. In this case, the parameter adjustment system 102 can still consider matching a data point labeled with a predicted anomaly to a data point labeled with a user anomaly, and thus count this data point to TP.

[0055] In an implementation, based on the enhanced confusion matrix and the lenient mode, the parameter adjustment system 102 can calculate a predetermined evaluation metric for each value combination. In an implementation, the predetermined evaluation metric can include, but is not limited to, a recall score and a precision score. Specifically, the recall score and the precision score for each value combination can be calculated using the following equations, respectively:

[0056]

[0057]

[0058] In obtaining the recall score and the precision score for each value combination, the parameter adjustment system 102 can obtain a performance score for each value combination based on a predetermined evaluation metric. In embodiments, the predetermined evaluation metric can include, for example, an F β score, which can be defined as follows:

[0059]

[0060] where β is a real number and is chosen such that the importance of the recall score is considered β times that of the precision score.

[0061] In embodiments, the parameter adjustment system 102 can choose β to be 1, such that the F β score becomes an F1 score in which the recall score and the precision score are equally important.

[0062] At block 412, the parameter adjustment system 102 can select or determine, from the set of different value combinations, a particular value combination as the value combination for the one or more parameters to be adjusted to recommend to the anomaly detection system, based on the calculated performance score for each value combination in the set of different value combinations for the one or more parameters to be adjusted.

[0063] In embodiments, after calculating the performance score for each value combination in the set of different value combinations for the one or more parameters to be adjusted, the parameter adjustment system 102 or the adjuster 312 of the parameter adjustment system 102 can select the value combination that produces the highest performance score as the value combination for the one or more parameters to be adjusted to recommend to the anomaly detection system 110.

[0064] In embodiments, if the recommended value combination is not needed, for example, the performance score is low, or one or more users are not satisfied with the performance of the anomaly detection system 102 assigned with the recommended value combination, the parameter adjustment system 102 can request the one or more users to provide further feedback on the one or more time series, for example, to label additional data points of the one or more time series as being abnormal points or non-abnormal points, etc. The parameter adjustment system 102 can then iteratively perform the operations as described above (e.g., blocks 402-412) to obtain a more optimal or better value combination for the one or more parameters to be adjusted.

[0065] At block 414, the parameter adjustment system 102 can wait for one or more newly labeled time series from the anomaly detection system or new data points of the one or more labeled time series, and automatically perform a new iteration to adjust the one or more parameters of the anomaly detection system.

[0066] In implementations, the parameter adjustment system 102 can automatically perform a new iteration to adjust the one or more parameters if it receives one or more new time series or new data points for the one or more labeled time series from the anomaly detection system, i.e., repeat the parameter adjustment operations as described above in blocks 404-412.

[0067] In implementations, additionally or alternatively, the parameter adjustment system 102 can receive one or more labeled time series and information of one or more parameters associated with another anomaly detection system from the other anomaly detection system. The adjustment parameter system can then perform the parameter adjustment operations as described above in blocks 404-412 for the anomaly detection system.

[0068] Although the above method blocks are described as being performed in a particular order, in some implementations, some or all of the method blocks can be performed in other orders or in parallel.

[0069] CONCLUSION

[0070] Although the implementations have been described with a structural features and / or method action specific language, it is to be understood that the claims are not necessarily limited to the specific features or actions described. Rather, the specific features and actions are disclosed as illustrative forms of implementing the claimed subject matter. Some or all of the operations can additionally or alternatively be implemented by one or more ASICs, FPGAs, or other hardware.

[0071] The disclosure can be further understood using the following clauses.

[0072] Clause 1 : A method implemented by one or more computing devices, the method comprising: generating a set of different value combinations for one or more parameters associated with an anomaly detection system; obtaining one or more time series, each data point of the one or more time series labeled with a label indicating presence, absence, or undetermined of an anomaly; assigning a respective value combination of the set of different value combinations to the one or more parameters associated with the anomaly detection system, and applying the anomaly detection system assigned with the respective value combination to a subset of data points of the one or more time series to obtain, for each value combination, predicted labels for the subset of data points; calculating a performance score for the anomaly detection system assigned with the respective value combination based at least in part on a predetermined evaluation metric; and selecting, from the set of different parameter combinations, a parameter combination corresponding to a highest performance score of the anomaly detection system as a recommended parameter combination for the one or more parameters associated with the anomaly detection system.

[0073] Clause 2: The method of clause 1, wherein the one or more parameters associated with the anomaly detection system comprise one or more parameters used in one or more statistical hypothesis tests for detecting one or more anomaly types in the anomaly detection system.

[0074] Clause 3: The method of clause 2, wherein the one or more parameters used in the one or more statistical hypothesis tests comprise one or more significance levels used in the one or more statistical hypothesis tests.

[0075] Clause 4: The method of clause 3, wherein the set of different value combinations of the one or more parameters associated with the anomaly detection system comprises respective probability values generated for a plurality of data points labeled with a label indicating presence of an anomaly.

[0076] Clause 5: The method of clause 1, further comprising: obtaining a new data point for the one or more time series, the new data point comprising at least one data point labeled with a label indicating presence of an anomaly; computing a probability value for the at least one data point; using the computed probability value for the at least one data point as a value for a parameter of the one or more parameters associated with the anomaly detection system; and incorporating the computed probability value into the set of different value combinations to form a new set of different value combinations for the one or more parameters associated with the anomaly detection system.

[0077] Clause 6: The method of clause 5, further comprising: assigning a respective value combination in the new set of different value combinations to the one or more parameters associated with the anomaly detection system, and applying the anomaly detection system assigned with the respective value combination to a new subset of data points of the one or more time series to obtain predicted labels for the new subset of data points for each value combination in the new set of different value combinations; computing a performance score for the anomaly detection system assigned with the respective value combination based at least in part on the predetermined evaluation metric; and selecting, from the new set of different parameter combinations, a new parameter combination corresponding to a new highest performance score for the anomaly detection system as a new recommended parameter combination for the one or more parameters associated with the anomaly detection system.

[0078] Clause 7: The method of clause 1, wherein the performance score for the anomaly detection system comprises an F-1 score computed based on a recall score and a precision score included in the evaluation metric.

[0079] Clause 8: The method described in Clause 7, wherein the precision score depends on the number of correct anomaly predictions, the number of incorrect anomaly predictions, and the number of test data points with labels indicating that anomalies have not yet been identified.

[0080] Clause 9: The method described in Clause 1 further includes receiving feedback from one or more users to update labels associated with multiple data points of the one or more time series.

[0081] Clause 10: One or more computer-readable media storing executable instructions, which, when executed by one or more processors, cause the one or more processors to perform actions including: generating a set of distinct value combinations for one or more parameters associated with an anomaly detection system; obtaining one or more time series, each data point of the one or more time series being labeled with a label indicating the presence, absence, or undetermined nature of an anomaly; assigning a corresponding value combination from the set of distinct value combinations to the one or more parameters associated with the anomaly detection system, and applying the anomaly detection system assigned the corresponding value combination to a subset of data points of the one or more time series to obtain a predicted label for each value combination of the subset of data points; calculating a performance score of the anomaly detection system assigned the corresponding value combination based at least in part on a predetermined evaluation metric; and selecting from the set of distinct parameter combinations a parameter combination corresponding to the highest performance score of the anomaly detection system as a recommended parameter combination for the one or more parameters associated with the anomaly detection system.

[0082] Clause 11: One or more computer-readable media as described in Clause 10, wherein the one or more parameters associated with the anomaly detection system include one or more parameters used in one or more statistical hypothesis tests for detecting one or more anomaly types in the anomaly detection system.

[0083] Clause 12: One or more computer-readable media as described in Clause 11, wherein the one or more parameters used in the one or more statistical hypothesis tests include one or more significance levels used in the one or more statistical hypothesis tests.

[0084] Clause 13: One or more computer-readable media as described in Clause 12, wherein the set of different combinations of values ​​of the one or more parameters associated with the anomaly detection system includes corresponding probability values ​​generated for a plurality of data points marked with labels indicating the presence of an anomaly.

[0085] Clause 14: The one or more computer-readable media of clause 10, wherein the actions further comprise: obtaining new data points for the one or more time series, the new data points including at least one data point labeled with a label indicating presence of an anomaly; computing a probability value for the at least one data point; using the computed probability value for the at least one data point as a value for a parameter of the one or more parameters associated with the anomaly detection system; and incorporating the computed probability value into the set of different value combinations to form a new set of different value combinations for the one or more parameters associated with the anomaly detection system.

[0086] Clause 15: The one or more computer-readable media of clause 14, wherein the actions further comprise: assigning respective value combinations of the new set of different value combinations to the one or more parameters associated with the anomaly detection system, and applying the anomaly detection system assigned with the respective value combination to a new subset of data points of the one or more time series to obtain predicted labels for the new subset of data points for each value combination of the new set of different value combinations; computing a performance score for the anomaly detection system assigned with the respective value combination based at least in part on the predetermined evaluation metric; and selecting, from the new set of different parameter combinations, a new parameter combination corresponding to a new highest performance score for the anomaly detection system as a new recommended parameter combination for the one or more parameters associated with the anomaly detection system.

[0087] Clause 16: The one or more computer-readable media of clause 10, wherein the performance score for the anomaly detection system comprises an F-l score computed based on a recall score and a precision score included in the evaluation metric.

[0088] Clause 17: The one or more computer-readable media of clause 16, wherein the precision score depends on a number of correct anomaly predictions, a number of incorrect anomaly predictions, and a number of test data points having a label indicating that an anomaly has not been determined.

[0089] Clause 18: The one or more computer-readable media of clause 10, wherein the actions further comprise receiving feedback from one or more users to update labels associated with a plurality of data points of the one or more time series.

[0090] Clause 19: A system comprising: one or more processors; and memory storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform actions comprising: generating a set of different value combinations for one or more parameters associated with an anomaly detection system; obtaining one or more time series, each data point of the one or more time series labeled with a label indicating presence, absence, or not yet determined of an anomaly; assigning respective value combinations of the set of different value combinations to the one or more parameters associated with the anomaly detection system, and applying the anomaly detection system assigned with the respective value combinations to a subset of data points of the one or more time series to obtain, for each value combination, predicted labels for the subset of data points; computing performance scores for the anomaly detection system assigned with the respective value combinations based at least in part on a predetermined evaluation metric; and selecting, from the set of different parameter combinations, a parameter combination corresponding to a highest performance score for the anomaly detection system as a recommended parameter combination for the one or more parameters associated with the anomaly detection system.

[0091] Clause 20: The system of clause 19, wherein the actions further comprise: obtaining new data points of the one or more time series, the new data points including at least one data point labeled with a label indicating presence of an anomaly; computing a probability value for the at least one data point; using the computed probability value for the at least one data point as a value for a parameter of the one or more parameters associated with the anomaly detection system; incorporating the computed probability value into the set of different value combinations to form a new set of different value combinations for the one or more parameters associated with the anomaly detection system; assigning respective value combinations of the new set of different value combinations to the one or more parameters associated with the anomaly detection system, and applying the anomaly detection system assigned with the respective value combinations to a new subset of data points of the one or more time series to obtain, for each value combination of the new set of different value combinations, predicted labels for the new subset of data points; computing performance scores for the anomaly detection system assigned with the respective value combinations based at least in part on the predetermined evaluation metric; and selecting, from the new set of different parameter combinations, a new parameter combination corresponding to a new highest performance score for the anomaly detection system as a new recommended parameter combination for the one or more parameters associated with the anomaly detection system.

Claims

1. A method implemented by one or more computing devices, characterized in that, The method includes: Generate a set of different combinations of values ​​for one or more parameters associated with an anomaly detection system, said one or more parameters including: parameters associated with one or more statistical hypothesis tests for anomaly detection; Obtain one or more time series, each data point of the one or more time series being labeled with a label indicating the presence, absence, or undetermined nature of an anomaly; The corresponding value combination from the set of different value combinations is assigned to the one or more parameters associated with the anomaly detection system, and the anomaly detection system assigned the corresponding value combination is applied to a subset of the data points of the one or more time series to obtain a predicted label for the subset of data points for each value combination; The performance score of the anomaly detection system, assigned the corresponding value combinations, is calculated at least in part based on a predetermined evaluation metric; and From the set of different value combinations, select the parameter combination corresponding to the highest performance score of the anomaly detection system as the recommended parameter combination for the one or more parameters associated with the anomaly detection system.

2. The method according to claim 1, characterized in that, in, The one or more parameters associated with the anomaly detection system include one or more parameters used in one or more statistical hypothesis tests for detecting one or more anomaly types in the anomaly detection system.

3. The method according to claim 2, characterized in that, The one or more parameters used in the one or more statistical hypothesis tests include: one or more significance levels used in the one or more statistical hypothesis tests.

4. The method according to claim 3, characterized in that, The set of different combinations of values ​​for one or more parameters associated with the anomaly detection system includes corresponding probability values ​​generated for multiple data points marked with labels indicating the presence of an anomaly.

5. The method according to claim 1, characterized in that, Also includes: Obtain new data points for the one or more time series, the new data points including at least one data point marked with a label indicating the presence of an anomaly; Calculate the probability value of the at least one data point; The probability value calculated using the at least one data point is used as the parameter value among the one or more parameters associated with the anomaly detection system; as well as The calculated probability values ​​are combined into the set of different value combinations to form a new set of different value combinations for the one or more parameters associated with the anomaly detection system.

6. The method according to claim 5, characterized in that, Also includes: Assigning the corresponding value combinations from the new set of different value combinations to the one or more parameters associated with the anomaly detection system; and, The anomaly detection system, which is assigned the corresponding value combinations, is applied to a new subset of the data points of the one or more time series to obtain a predicted label for the new subset of data points for each of the new set of different value combinations. The performance score of the anomaly detection system, which is assigned the corresponding combination of values, is calculated at least in part based on the predetermined evaluation metric. as well as From the set of new combinations of different values, select a new combination of parameters that corresponds to the new highest performance score of the anomaly detection system as a new recommended combination of parameters associated with the anomaly detection system.

7. The method according to claim 1, characterized in that, The performance score of the anomaly detection system includes an F-1 score, which is calculated based on the recall score and precision score included in the evaluation metric.

8. The method according to claim 7, characterized in that, The accuracy score depends on the number of correct anomaly predictions, the number of incorrect anomaly predictions, and the number of test data points with labels indicating that anomalies have not yet been identified.

9. The method according to claim 1, characterized in that, Also includes: Receive feedback from one or more users to update the labels associated with multiple data points of the one or more time series.

10. One or more computer-readable media storing executable instructions, characterized in that, When executed by one or more processors, the executable instructions cause the one or more processors to perform actions including the following: Generate a set of different combinations of values ​​for one or more parameters associated with an anomaly detection system, said one or more parameters including: parameters associated with one or more statistical hypothesis tests for anomaly detection; Obtain one or more time series, each data point of the one or more time series being labeled with a label indicating the presence, absence, or undetermined nature of an anomaly; The corresponding value combination from the set of different value combinations is assigned to the one or more parameters associated with the anomaly detection system, and the anomaly detection system assigned the corresponding value combination is applied to a subset of the data points of the one or more time series to obtain a predicted label for the subset of data points for each value combination; The performance score of the anomaly detection system, assigned the corresponding value combinations, is calculated at least in part based on a predetermined evaluation metric; and From the set of different value combinations, select the parameter combination corresponding to the highest performance score of the anomaly detection system as the recommended parameter combination for the one or more parameters associated with the anomaly detection system.

11. One or more computer-readable media according to claim 10, characterized in that, The one or more parameters associated with the anomaly detection system include one or more parameters used in one or more statistical hypothesis tests for detecting one or more anomaly types in the anomaly detection system.

12. One or more computer-readable media according to claim 11, characterized in that, The one or more parameters used in the one or more statistical hypothesis tests include one or more significance levels used in the one or more statistical hypothesis tests.

13. One or more computer-readable media according to claim 12, characterized in that, The set of different combinations of values ​​for one or more parameters associated with the anomaly detection system includes corresponding probability values ​​generated for multiple data points marked with labels indicating the presence of an anomaly.

14. One or more computer-readable media according to claim 10, characterized in that, The actions mentioned above also include: Obtain new data points for the one or more time series, the new data points including at least one data point marked with a label indicating the presence of an anomaly; Calculate the probability value of the at least one data point; The probability value calculated using the at least one data point is used as the value of one or more parameters associated with the anomaly detection system; and The calculated probability values ​​are combined into the set of different value combinations to form a new set of different value combinations for the one or more parameters associated with the anomaly detection system.

15. One or more computer-readable media according to claim 14, characterized in that, The actions mentioned above also include: The corresponding value combinations from the new set of different value combinations are assigned to the one or more parameters associated with the anomaly detection system, and the anomaly detection system, which is assigned the corresponding value combinations, is applied to a new subset of the data points of the one or more time series to obtain a predicted label for the new subset of the data points for each of the new set of different value combinations. The performance score of the anomaly detection system, assigned the corresponding value combinations, is calculated at least in part based on the predetermined evaluation metric; and From the set of new combinations of different values, select a new combination of parameters that corresponds to the new highest performance score of the anomaly detection system as a new recommended combination of parameters associated with the anomaly detection system.

16. One or more computer-readable media according to claim 10, characterized in that, The performance score of the anomaly detection system includes an F-1 score calculated based on the recall score and precision score included in the evaluation metric.

17. One or more computer-readable media according to claim 16, characterized in that, The accuracy score depends on the number of correct anomaly predictions, the number of incorrect anomaly predictions, and the number of test data points with labels indicating that anomalies have not yet been identified.

18. One or more computer-readable media according to claim 10, characterized in that, The actions described also include receiving feedback from one or more users to update labels associated with multiple data points of the one or more time series.

19. A system, characterized in that, include: One or more processors; as well as A memory storing executable instructions that, when executed by one or more processors, cause the one or more processors to perform actions including: Generate a set of different combinations of values ​​for one or more parameters associated with an anomaly detection system, said one or more parameters including: parameters associated with one or more statistical hypothesis tests for anomaly detection; Obtain one or more time series, each data point of the one or more time series being labeled with a label indicating the presence, absence, or undetermined nature of an anomaly; The corresponding value combination from the set of different value combinations is assigned to the one or more parameters associated with the anomaly detection system, and the anomaly detection system assigned the corresponding value combination is applied to a subset of the data points of the one or more time series to obtain a predicted label for the subset of data points for each value combination; The performance score of the anomaly detection system, assigned the corresponding value combinations, is calculated at least in part based on a predetermined evaluation metric; and From the set of different value combinations, select the parameter combination corresponding to the highest performance score of the anomaly detection system as the recommended parameter combination for the one or more parameters associated with the anomaly detection system.

20. The system according to claim 19, characterized in that, The actions mentioned above also include: Obtain new data points for the one or more time series, the new data points including at least one data point marked with a label indicating the presence of an anomaly; Calculate the probability value of the at least one data point; The probability value calculated using the at least one data point is used as the value of one or more parameters associated with the anomaly detection system; The calculated probability values ​​are combined into the set of different value combinations to form a new set of different value combinations for the one or more parameters associated with the anomaly detection system; The corresponding value combinations from the new set of different value combinations are assigned to the one or more parameters associated with the anomaly detection system, and the anomaly detection system, which is assigned the corresponding value combinations, is applied to a new subset of the data points of the one or more time series to obtain a predicted label for the new subset of the data points for each of the new set of different value combinations. The performance score of the anomaly detection system, assigned the corresponding value combinations, is calculated at least in part based on the predetermined evaluation metric; and From the set of new combinations of different values, select a new combination of parameters that corresponds to the new highest performance score of the anomaly detection system as a new recommended combination of parameters associated with the anomaly detection system.

Images