Dynamic configuration of anomaly detection

Abstract

The disclosed embodiments generate multiple anomaly detector configurations and compare the results generated by these anomaly detectors with a reference result set. The reference result set is generated by a trained model. The correlation between each result generated by the anomaly detector and the result set is compared to select an anomaly detector configuration that provides results most similar to those of the trained model. Then, in some embodiments, data defining the selected configuration is transmitted to a product facility. The product facility instantiates the defined anomaly detector and uses the instantiated detector to analyze local events. In some other embodiments, the defined anomaly detector is instantiated by the same system that selected the anomaly detector; therefore, in these embodiments, the anomaly detector configuration is not transferred from one system to another.

Description

Background Technology

[0001] Anomaly detection techniques offer numerous advantages. In particular, anomaly detection can identify new or unique events in unlabeled data. Conversely, supervised classification in machine learning excels at detecting the recurrence of known events, but relies on labeled examples of those events to learn the model. For instance, in the security field, anomaly detection can identify unfamiliar program behavior and reveal previously unseen malware types, while supervised classifiers are better suited for detecting known malware types.

[0002] Despite its advantages, anomaly detection can be difficult to use in practice. First, while a range of anomaly detection algorithms exist, they can be tuned by hyperparameters to define what is “new” by a particular detector. Predicting which specific configuration of an anomaly detector will produce useful results can be challenging. Second, and more fundamentally, “new” is not the same as “useful.” Returning to the security example, many legitimate programs exhibit rare but non-malicious behavior. Both of these issues lead to a difficult barrier to identify combinations of anomaly detection algorithms and parameters that detect “new” events that provide useful observations for the task at hand. Therefore, improved anomaly detection methods are needed. Attached Figure Description

[0003] In the accompanying drawings, the same numerical symbols may describe similar components in different views, and the drawings are not necessarily drawn to scale. The same numerical symbols with different letter suffixes may represent different instances of similar components. The accompanying drawings illustrate, by way of example and not limitation, the various embodiments discussed in this document.

[0004] Figure 1 This is an overview diagram of an anomaly detector definition system implemented in at least one of the disclosed embodiments.

[0005] Figures 2A to 2C It is a data flow diagram illustrating the data flow in one or more of the disclosed embodiments.

[0006] Figure 3 Example data structures implemented in one or more of the disclosed embodiments are shown.

[0007] Figure 4 An example data stream is shown in the anomaly detector in at least one of the disclosed embodiments.

[0008] Figure 5 This is an example message portion implemented in one or more of the disclosed embodiments.

[0009] Figure 6 Example message portions are shown that can be implemented in one or more of the disclosed embodiments.

[0010] Figure 7 This is a flowchart used to define the anomaly detector.

[0011] Figure 8 This is a flowchart used to define the anomaly detector.

[0012] Figure 9 This is a flowchart of the process for configuring the anomaly detector.

[0013] Figure 10 A block diagram of an example machine implemented in one or more of the disclosed embodiments is illustrated. Detailed Implementation

[0014] The following description and accompanying drawings fully illustrate specific embodiments to enable those skilled in the art to practice them. Other embodiments may be combined with structural, logical, electrical, procedural, and other variations. Parts and features of some embodiments may be included in, or replaced by, parts and features of other embodiments. The embodiments set forth in the claims cover all available equivalents of these claims.

[0015] The disclosed embodiments describe an environment that provides continuous improvement in anomaly detector performance. As discussed further below, an anomaly detector creation hub generates multiple anomaly detector configurations and instantiates these anomaly detectors. Each anomaly detector analyzes the test dataset, where each anomaly detector ranks and / or classifies events. In some embodiments, events are ranked to indicate a relative level of anomalousness or the relevance of each event. Thus, in some embodiments, the first-ranked event is the most anomalous event relative to other events included in the test data. Events are also ranked via a trained classifier. The classifier is trained using annotated training data, which may have been annotated with human input and / or a combination of human and machine input.

[0016] In some embodiments, the ordering of events also conveys the probability that the ordered events belong to each of a plurality of event types. In some embodiments, the ordering includes a plurality of probabilities for each event. Each of the plurality of probabilities conveys the probability that the event belongs to one of the plurality of event types.

[0017] The ranking and / or classification generated by the trained classifier is considered a reference ranking or classification. The reference ranking is then compared to the rankings / classifications generated by each of the instantiated detectors. This comparison may include an indication of correlation, which is the correlation between the reference ranking / classification and the rankings / classifications generated by the instantiated anomaly detectors. The anomaly detector with the highest correlation to the reference ranking / classification can be selected for distribution to one or more product facilities.

[0018] After an anomaly detector configuration is selected, the data defining the configuration is distributed to the product facilities. For example, in some cases, the anomaly detector creation hub is maintained by the software vendor. After the software vendor selects the anomaly detector configuration, the vendor distributes the data defining the configuration to the product facilities at the customer's location. Each product facility can then instantiate the selected detector in its local environment at the customer's location. The instantiated detector then operates on the data generated within the local customer environment.

[0019] In some disclosed embodiments, the product facility is also used to provide training data back to the anomaly detector creation hub. This additional training data is used to evaluate additional variations in the anomaly detector configuration. In some embodiments, the additional training data is annotated to further train the trained model described above, thereby generating a second reference ranking or a second reference classification. The additional variations in the anomaly detector configuration are then evaluated against the second reference ranking or second reference classification. As described above, the configuration most relevant to the improved second reference ranking or second reference classification is selected, and data defining the same configuration is redistributed to the product facility. This cycle can be repeated throughout the lifetime of a particular product and / or anomaly detector application.

[0020] Figure 1 This is a general view of an anomaly detector definition system 100 implemented in at least one of the disclosed embodiments. System 100 includes an anomaly detector creation hub 102. The anomaly detector creation hub 102 communicates with four product facilities 104a-104d. Product facilities 104a-104d represent facility locations or deployment locations for a product. The product facilities utilize anomaly detectors to detect anomalous behavior occurring within the local environment of the respective product facility (e.g., any of 104a-104d).

[0021] A hub 102 is shown that distributes anomaly detector definition data 106a-106d to each of the product facilities 104a-104d. The anomaly detector definition data 106a-106d can define anomaly detection algorithms and one or more hyperparameter values ​​for those algorithms. Product facilities 104a-104d deploy the defined anomaly detectors to their respective products. Product facilities 104a-104d also collect event data that defines one or more computer system events. In various embodiments, computer system events identify process creation events, packets transmitted over a network, file operations (such as file creation, deletion, modification, registry editing), logon / logout events, or other types of events. Each of these events also identifies one or more parameters associated with that event. For example, in some embodiments, a process creation event identifies a filename associated with executable code instantiated when a process is created. In at least some embodiments, the input parameters of the "createprocess()" function are also defined as part of the event.

[0022] Then, product facility 104-104d will store at least a portion of the corresponding event data (in... Figure 1 The training data (shown as training data 108a-108d) is transmitted to the anomaly detector creation hub 102. The anomaly detector creation hub 102 can utilize the training data 108a-108d to improve the anomaly detector. This may result in additional or updated anomaly detector definition data 106a-106d being redistributed to each of the product facilities 104-104d. Note that, although as described above… Figure 1 This demonstrates the separation between hub 102 and product facilities 104-104d. However, some other embodiments do not necessarily implement this separation. For example, in some embodiments, a single system or hardware processor is configured to select an anomaly detector from various anomaly detector configurations and then detect anomalous behavior (or sort computer events) based on the selected anomaly detector, as further described below. In these embodiments, no configuration information defining the selected anomaly detector is transferred from one computer system (or hardware processor) to another.

[0023] Figure 2A-2B A data flow diagram illustrating the data flow in one or more of the disclosed embodiments is shown. Figure 2ASeveral anomaly detectors 204a-204d are shown. Anomaly detectors 204a-204d represent the configuration of the anomaly detector. This configuration includes one or more specifications for the algorithm used by the anomaly detector, one or more hyperparameter values ​​for the algorithm, or specifications for the feature selection scheme and / or feature transformation scheme used by the anomaly detector.

[0024] Each of the anomaly detectors 204a-204d reads data from the computer event data store 202 and processes the data to identify anomalous events within the data. Figure 2A A trained classifier 205 is also shown. The trained classifier 205 also reads data from the computer event data store 202 and sorts the events based on the relevance of each event within the data. Indications of anomalous data generated by each of the anomaly detectors 204a-204d are shown in... Figure 2A The values ​​are shown as 206a-206d respectively. The indicators of anomalous data generated by the trained classifier are shown as 208a-208d in Figure 2.

[0025] Figure 2A Correlators 210a-210d are also shown. Each correlator in the correlators receives the output from the trained classifier 205 as input. The output of classifier 205 is shown via data streams 208a-208d, respectively. Each correlator in the correlators 210a-210d also receives the output from one of the anomaly detectors 204a-204d as input. Each correlator then compares its two corresponding inputs and outputs a signal (shown as 212a-212d) to comparator 208.

[0026] Data flow 200 continues via external reference "A" to Figure 2B , Figure 2B The diagram illustrates comparator 208 providing data to anomaly detector selection component 220. Anomaly detector selection component 220 selects one or more anomaly detectors from anomaly detectors 204a-204d based on correlation scores provided by correlators 210a-210d. Anomaly detector selection component 220 then provides product facility 104 with data 222 defining the selected anomaly detectors.

[0027] The product facility then uses a defined anomaly detector to detect anomalous events in the local environment. For example, two computer systems 224 and 226 represent the local environment of product facility 104. Product facility 104 monitors the activities of the two computer systems 224 and 226. For example, product facility program 104 monitors system log files, event logs, or similar activity information generated by the two computer systems 224 and 226, and detects anomalous behavior in one or more of the computer systems 224 and 226 via a selected anomaly detector. The product facility then generates an event 228 indicating the anomalous behavior.

[0028] Figure 2C An example implementation of an anomaly detector, implemented in one or more of the disclosed embodiments, is shown. Figure 2C The example anomaly detector 204a shown in the figure can be Figure 2A Any one or more of the anomaly detectors 204a-204d illustrated in the figure. Figure 2C The example anomaly detector 204a shown is considered an ensemble anomaly detector because it comprises multiple individual anomaly detection algorithms. Anomaly detectors 240a-240c are shown. Each anomaly detector in anomaly detectors 240a-240c relies on individually learned feature data and / or model information to perform anomaly detection on data from... Figure 2A The event data storage 202 shown is used for anomaly detection or sorting of computer events. Each anomaly detector in anomaly detectors 240a-240c is configured via the following... Figure 4 The described training process is performed individually. Each anomaly detector, including anomaly detectors 240a-240c in the integrated anomaly detector 204a, generates a sorting of corresponding computer events, shown as sorts 240a-240c. A combiner component 244 of the integrated anomaly detector 204a combines the multiple sortings generated by the multiple anomaly detectors of the integrated anomaly detector 204a to generate a single sorting 246 as the output of the integrated anomaly detector 204a. The combiner component 244 is controlled via combiner configuration data 245. For example, in various embodiments, when combining sorts 242a-242c, the combiner configuration data 245 specifies one or more weights among the weights associated with each sort in sorts 242a-242c, and specifies the algorithm used for the combination (e.g., sort average, sort weighted average, or other algorithm).

[0029] Figure 3 Example data structures implemented in one or more of the disclosed embodiments are shown. Figure 3An event table 300, a sorting table 310, and a mitigation table 320 are shown. In some embodiments, the event table 300 is included in a computer event data storage 202, as described above. Figure 2A The event table, as discussed, stores information about events ordered by embodiments of this disclosure. Event table 300 includes an event identifier 302, an event type 303, an event time field 304, and a parameter count field 305. The event table also includes a variable number of field pairs. Each pair includes a parameter name / identifier field 306 and a parameter value field 307. The number of pairs is defined by the parameter count field 305. The event identifier field 302 uniquely identifies the event to be ordered by the disclosed embodiments. The event type 303 identifies the type of event. For example, the event type field 303 distinguishes between process creation events, network packet transmission or reception events, file operation events, or other types of events. The event time field 304 identifies the time when the event occurred. The parameter id / name field 306 identifies a specific parameter of the event. For example, for a process creation event, the parameter id / name field 306 may identify the process name or file name used to create the process. The parameter value field 307 identifies the value of the identified parameter.

[0030] The sorting table 310 includes a sorting identifier field 312, an event identifier field 314, and a position indicator field 316. The sorting identifier field 312 uniquely identifies a particular sort. For example, in some embodiments, each anomaly detector in anomaly detectors 204a-204d and the trained classifier 205 generate a separate sorting of events. Therefore, in at least some embodiments, each of these separate sortings is assigned a different sorting identifier. The event identifier field 314 may be cross-referenced with the event identifier field 302 and identifies a specific event within a particular sorting. In some embodiments, the position / score field 316 identifies the position of the identified event (e.g., identified via field 314) within a sorting (e.g., identified via field 312). Some embodiments operate on scores or probabilities rather than sorting. For example, while sorting indicates the relevance of a particular event relative to other events, conversely, some embodiments may determine the probability that a particular event is classified as a particular type of event. Therefore, in these embodiments, field 316 does not store the order of the corresponding measurements indicating relevance; instead, field 316 stores an indication of the likelihood that a particular event is of a particular type (e.g., the probability that an event is of a particular type). Event type 318 indicates the type of event. In some embodiments, the probability that the event identified by event id field 314 is of the type of event indicated by event type field 318 is stored in position / score field 316.

[0031] Mitigation table 320 includes an event identifier / type field 322 and a mitigation action field 324. The event identifier field 322 uniquely identifies an event and can be cross-referenced with any of the fields 302 and 314. In some embodiments, field 322 identifies the type of the event. Therefore, in these embodiments, all events of a common type are mitigated via a common set of one or more mitigation actions defined in field 324 and discussed further below. If field 322 stores the event type, then field 322 can be cross-referenced with field 318, as discussed above.

[0032] Mitigation Action field 324 defines one or more mitigation actions associated with an event (identified via field 322). In some embodiments, mitigation action field 324 identifies one or more executable code segments that perform the mitigation action, or identifies parameters passed to the executable code. For example, mitigation action field 324 may define portions of the event identified via field 322 that are passed as input parameters to the executable code performing the mitigation. Mitigation actions may include, for example, resetting the computer that generated the event, generating an email or text message to a defined distribution list, wherein the email or text describes the event that occurred. Other mitigation actions may include adjusting one or more system parameters. For example, a mitigation action increases the amount of available swap space. Another mitigation action increases the redundancy level of logging or tracking diagnostic utilities.

[0033] Figure 4 Example data streams present in an anomaly detector in at least one of the disclosed embodiments are shown. In various embodiments, regarding... Figure 4 The data stream under discussion is 400. Figure 2A Anomaly detectors 204a-204d are implemented. Data stream 400 demonstrates the operation of a single anomaly detector, such as any one or more anomaly detectors 240a-240c, and / or any anomaly detector in anomaly detectors 204a-204d that are not integrated anomaly detectors. Such as the above regarding... Figure 2C The illustrated integrated anomaly detector may include multiple copies of components 425, 440, 430, and 445 discussed below, each copy for each individual anomaly detector included in the integrated anomaly detector. Each anomaly detector included in the integrated anomaly detector will be trained separately via independent copies of components 410, 415, and 420, resulting in at least one separate copy 425, which is used by each corresponding individual anomaly detector at runtime.

[0034] In at least one embodiment, Figure 4The components of the anomaly detector system shown represent parts of the anomaly detector configuration. Figure 4 The diagram illustrates training data 405, a feature selection / transformation component 410, and an anomaly detection training algorithm 415. The feature selection / transformation component 410 processes the training data 405 and provides the processed data to the anomaly detection training algorithm 415. Based on the training data and one or more hyperparameters 420, the anomaly detection training algorithm 415 generates data defining model 425. Examples of information stored in model 425 include feature weights in OC-SVM, tree information when using an isolated forest, distribution parameters for a mixture model, or a forgetting factor for a time series model.

[0035] Then, the anomaly detection algorithm 430 relies on the learned features 425 to analyze the additional event data 435. In at least some embodiments, the event data 435 may be stored or conform to the above-mentioned... Figure 3 The format of the event table 300 under discussion. In some embodiments, the event data is also processed by the feature selection / transformation component 440. Configuration parameters 442 defining the operation of the feature selection / transformation component 410 are passed to the feature selection / transformation component 440 to ensure that its operation is similar to that of the feature selection / transformation component 410.

[0036] Then, in some embodiments, the anomaly detection algorithm 430 generates a sorting 445 of the additional event data 435. The additional event data 435 is sorted from the most relevant events to the least anomalous events. In some embodiments, the sorting 445 is performed via the above-mentioned... Figure 3 The sorting table 310 under discussion is represented. The anomaly detection algorithm 430 implements one or more of the following algorithms: single-class support vector machine (SVM), k-means, or anomaly detection based on local outlier factors.

[0037] In some other embodiments, the anomaly detection algorithm 40 classifies the event type of the event. For example, in some embodiments, the anomaly detection algorithm 430 determines the probability that a particular event is a particular event type. In some embodiments, probabilities are generated for multiple event types. Thus, multiple probabilities are generated based on a particular event, where each probability indicates the probability that the particular event is a particular type of event.

[0038] In some embodiments, the data defining the anomaly detector includes specifications for model 435 and anomaly detection algorithm 430. For example, in some aspects, the specification for anomaly detection algorithm 430 includes instructions for implementing the algorithm. In various embodiments, the instructions may be specified in source code, intermediate code, or machine code. In some embodiments, the specification identifies a pre-existing algorithm for anomaly detection. For example, a pre-existing algorithm may be identified via a predetermined identifier mapped to a pre-existing algorithm.

[0039] Figure 5 This is an example message portion implemented in one or more of the disclosed embodiments. In some embodiments, one or more fields described below with respect to message portion 500 are transmitted from creation hub 102 to at least one of product facilities 104a-104d. For example, message portion 500 may be included in one or more data streams 106a-106d from creation hub 102 to product facilities 104a-104d. In some embodiments, message portion 500 is transmitted as part of data stream 222 from anomaly detector selection component 220 to product facility 104, as described above with respect to... Figure 2B The subject of discussion.

[0040] Message section 500 defines the anomaly detector configuration. In other words, in at least some embodiments, message section 500 includes information sufficient to enable the product facility to instantiate the anomaly detector and operate the anomaly detector to sort events occurring within the computer system. In some cases, the anomaly detector is an integrated anomaly detector, which includes multiple individual anomaly detector algorithms (e.g., 430) and / or learned feature data / model data (e.g., 425). In some embodiments, the integrated anomaly detector includes multiple anomaly detectors that share a common anomaly detection algorithm (e.g., 430) but utilize different learned feature / model data (e.g., 425) to configure the anomaly detector algorithms to operate differently. In some embodiments, the two anomaly detectors may differ in feature selection / transformation methods used to process event data (e.g., 435) before the event data is provided to the anomaly detector (e.g., 430).

[0041] The message section 500 includes a number field 504 for the number of integrated anomaly detectors. The number field 504 specifies the number of anomaly detector algorithms used to generate the single output ordering of the integrated anomaly detectors. In some embodiments, the number field 504 specifies a single anomaly detector. In this case, it is not necessary to combine the algorithms described above. Figure 2C The order of description. Conversely, the operation of the anomaly detector specified by message section 500 is similar to that of anomaly detector 430, as described below. Figure 4 The subject of discussion.

[0042] Some fields of message section 500 are repeated for each anomaly detector included in the integrated anomaly detector specified by field 504. These fields include algorithm specification field 506, model definition field 508, hyperparameter field 509, and field 510, where field 510 stores data defining feature selection and / or feature transformation control parameters. Algorithm specification field 506 defines anomaly detection algorithm (e.g., 430). As described above, some embodiments define anomaly detection algorithms via instructions for implementing the algorithm (e.g., intermediate or binary code implementing the algorithm). Other embodiments identify anomaly detection algorithms via a unique identifier, which identifies the algorithm via a predetermined mapping from identifier to algorithm. Model definition field 508 stores data defining the trained model. In embodiments that communicate anomaly detection configuration defining the trained anomaly detector, model definition field 508 is included in message section 500. Some other embodiments may provide some local training for the anomaly detector. In those embodiments, message section 500 includes hyperparameter field 509. In these embodiments, the model defined by model definition field 508 is received by a product facility (e.g., any one or more of 104a-104d). The product facility then further trains the model (e.g., as stored in 508) using hyperparameters (e.g., as stored in 509) and algorithm specifications (e.g., as stored in 506). The product facility then detects relevant events based on the further trained model.

[0043] Following all exception detectors included in the integrated exception detectors specified by message section 500, combiner configuration field 512 is included. (As mentioned above...) Figure 2C As discussed in combiner configuration 246, combiner configuration field 512 specifies how multiple sorts are combined to produce a single output sort (e.g., 246), where multiple sorts are generated by an anomaly detector included in an integrated anomaly detector. If message section 500 specifies a non-aggregate anomaly detector (e.g., an anomaly detector that includes only one anomaly detection algorithm and one anomaly detection model), then combiner configuration field 512 is not included in message section 500.

[0044] Figure 6 An example message portion 600 is shown that may be implemented in one or more of the disclosed embodiments. In some embodiments, message portion 600 is transmitted from a product facility to an anomaly detector creation hub 102. For example, in some embodiments, one or more fields discussed below with respect to example message portion 600 are transmitted from any one or more product facilities 104a-104d to creation hub 102 via data streams 106a-106d, respectively.

[0045] Example message section 600 includes an event identifier field 616 and a number of parameters field 618. The event identifier field 616 identifies an event identified by the product facility. In some embodiments, the event is identified via an anomaly detector previously provided to the product facility by the creation hub. The number of parameters field 618 identifies the number of parameters specified by message section 600. Following the number of parameters field 618 are pairs of fields, each pair for each parameter indicated by field 618. Each pair includes a parameter field 622 and a parameter value field 624. The parameter field 622 identifies the parameter. For example, some embodiments in the disclosed examples maintain a mapping of system parameters to predetermined identifiers. Each system parameter can then be identified via a predetermined number and one of the mappings. The parameter value field 624 stores the value used for the identified parameter.

[0046] The system parameters identified by Example Message Section 600 can convey various information about the operation of a particular system. System parameters include one or more parameters that are either hardware operating parameters or software operating parameters. Example system parameters may include CPU utilization, memory utilization, available memory, I / O utilization, free disk space, and temperature. System parameters may also include parameters specific to a particular software application, such as parameters indicating the number of network connections, error counts (including error counts for different types of errors), and other parameters that convey information about the operation of the computing system.

[0047] In some embodiments, as described above Figure 6 The described information can be repeated within a single message. For example, message portion 600 may include multiple event ID fields 616 describing multiple events, along with a number field 618 for multiple parameters and corresponding and supported fields 622 and 624.

[0048] Figure 7 This is a flowchart used to define the process of anomaly detectors. In some aspects, the following... Figure 7 One or more of the functions discussed can be executed by a hardware processing circuitry system. For example, in some embodiments, instructions stored in memory configure the hardware processing circuitry system to perform the following... Figure 7 One or more of the functions and / or operations discussed.

[0049] In operation 710, a trained model analyzes multiple computing system events. For example, as mentioned above... Figure 2AThe trained classifier (e.g., model) 205 discussed reads event data from a computer event data store 202. In some embodiments, operation 710 includes processing the event data (e.g., from the event data store 202) via a clustering-based feature transformation. The resulting events are then provided to the trained classifier. For example, in some embodiments, multiple computing system events are generated by projecting the most frequently occurring set of features that vary within the event data.

[0050] In operation 720, a first ranking is determined based on the analysis of operation 710. A trained classifier 205 ranks the computer events identified in the computer event data store 202. Each event is assigned a unique ranking that indicates the degree of anomalousness of that event relative to other events included in the computer event data store 202. The first ranking determined in operation 720 is considered a reference ranking, where the ranking of events within the reference ranking is considered an event reference ranking, as discussed further below.

[0051] In Operation 730, multiple anomaly detectors analyze multiple computer system events. For example, as mentioned above... Figure 2A As illustrated, anomaly detectors 204a-204d read computer system events from computer event data storage 202. Two of the multiple anomaly detectors use equivalent anomaly detection algorithms (e.g., Figure 4 (430), but using different hyperparameter values ​​(e.g.) Figure 4 (420). In some embodiments, at least two of the plurality of anomaly detectors use different anomaly detection algorithms. In some embodiments, one or more of the anomaly detectors may utilize a single-class support vector machine (SVM) algorithm, a k-means algorithm, or a local outlier factor algorithm. However, the embodiments are not limited to these algorithmic examples.

[0052] In some embodiments, at least two of the multiple anomaly detectors utilize a local outlier factor algorithm. Each of the at least two anomaly detectors uses a different locality (K) value. In some embodiments, at least two of the anomaly detectors utilize a one-class support vector machine. The at least two anomaly detectors utilize different kernels. The kernels are selected from radial basis functions (RBF), linear, polynomial, or sigmoid. In some aspects, at least two of the multiple anomaly detectors utilize a one-class support vector machine algorithm. The at least two anomaly detectors are configured with different nu values ​​(e.g., hyperparameter value 420). In some embodiments, at least two of the multiple anomaly detectors utilize a K-means clustering algorithm. These at least two anomaly detectors are configured with different locality (K) values ​​(e.g., hyperparameter value 420).

[0053] In operation 740, multiple second rankings of computer system events are generated based on the second analysis in operation 730. Each anomaly detector in the anomaly detector ranks the computer system events. For example... Figure 2A As illustrated in the figure, the individual ordering of each anomaly detector is shown as data streams 206a-206d.

[0054] In operation 750, multiple correlations are determined. In some embodiments, each of the multiple correlations is represented by a corresponding correlation score. Each correlation is the correlation between a first ranking (reference ranking) and a different second ranking in the second ranking. For example, as mentioned above... Figure 2AAs illustrated, each correlator in correlators 210a-210d receives the output of a trained classifier 205, which indicates a first ranking. Each correlator in correlators 210a-210d also receives input from different anomaly detectors in anomaly detectors 204-204d. In some embodiments, the correlation is determined by operation 750 using one or more of Spearman's p, Kendall's t, a combined Spearman and Kendall method, rank biarray, or average rank biarray. In some embodiments, the correlation is determined only based on the highest ranking portion of each ranking in the (first and second) rankings. In other words, in some embodiments, to improve the quality of the selected anomaly detector and / or reduce processing overhead, a certain percentage or absolute number of the highest ranking events from each ranking in the first and second rankings are correlated. For example, in some embodiments, the top one percent, top two percent, top five percent, or top ten percent of each ranking are correlated. In some embodiments, the top five (5), top ten (10), top twenty (20), or top fifty (50) of the highest ranking events from each ranking are correlated. In some embodiments, each of the second rankings in the second ranking is adjusted via outlier preservation normalization before the correlation is determined.

[0055] Some embodiments employ a rank-bicol correlation method, which is a modification of the mature version of this method. In some embodiments, a top K threshold (e.g., the top 100) is defined, and events ranked above this threshold are considered in the correlation calculation. A first set of event pairs is generated. Each event pair includes a first event ranked above the threshold and a second event of the pair ranked below the threshold.

[0056] Then, a second set of event pairs is generated based on the first set of event pairs. The second set of event pairs is generated based on a first selection criterion applied to the first set of event pairs. The first selection criterion identifies event pairs where the first event has a higher reference ranking than the second event in the pair (which has a ranking below a ranking threshold). The reference ranking is the ranking assigned to the events by the trained classifier 205.

[0057] A fraction is then determined, representing pairs of events that are above a threshold (first event) and have a higher reference score than events that are below the threshold (second event). Intuitively, this fraction represents the frequency with which the top few anomalies of the detector are more relevant than other events, as measured by the trained classifier. This fraction is then used as a relevance score when comparing anomaly detectors to determine which one is selected for use in the product facility.

[0058] Then, the relevance score for the anomaly detector is determined according to the following Equation 1:

[0059] ascore i =p i -(1-p i ) = 2p i -1, (1)

[0060] in:

[0061] ascore i This represents the relevance score configured for a specific (i-th) anomaly detector.

[0062] p i It is the score that matches the above reference scoring criteria.

[0063] In operation 760, an anomaly detector is selected from multiple relevance detectors. In some embodiments, the anomaly detector with the highest relevance score (e.g., the best relevance to the reference ranking) is selected in operation 760. Some aspects use a multiple relevance method to generate multiple relevance scores. For example, multiple relevance methods including Spearman's and the computer Kendall's methods can be computed based on the results of each anomaly detector. In some embodiments of these examples, these multiple relevance scores are averaged or otherwise aggregated. The aggregated relevances are then compared among the anomaly detectors, and the anomaly detector with the best aggregated relevance score is selected in operation 760.

[0064] In operation 770, data defining the selected anomaly detector is transmitted to the product facility. For example, as mentioned above... Figure 1 The discussion focuses on creating hub 102 to transmit data to product facilities (e.g., any of 104a-104d), where this data defines anomaly detectors. As mentioned above... Figure 5 As discussed, in at least some embodiments, data defining the anomaly detector is included in message portion 500. In various embodiments, the message portion includes one or more of the following: data defining the anomaly detection algorithm (e.g., field 506 defining anomaly detection algorithm 430), hyperparameters (e.g., field 509), and data defining the model (e.g., field 508 defining model 425).

[0065] Decision operation 780 determines whether the additional event has been received. If no event has been received, process 700 moves from decision operation 780 to end operation 790. If the event has been received, process 700 moves from decision operation 780 to operation 710, where operation 710 analyzes the newly received event as "multiple computing system events," and the above... Figure 7The described processing is repeated for new events. In some aspects, before returning control to operation 710, decision operation 780 may wait for a period of time to allow for the accumulation of new events (e.g., when the elapsed time and / or the number of new events exceeds a predetermined threshold).

[0066] Although process 700 as described above includes transmitting data defining the anomaly detector to the product facility. At least refer to... Figure 1 as well as Figure 7 and Figure 9 This functional division is described using a combination of methods. However, some embodiments disclosed do not necessarily include this functional division. For example, some embodiments may perform functions associated with, for example, operations 705-760 and 910-935. In other words, some embodiments compare various anomaly detector configurations, select one anomaly detector, and then also detect events based on the selected anomaly detector.

[0067] Figure 8 This is a flowchart used to define the process of anomaly detectors. In some respects, the following... Figure 8 The process 800 discussed defines a single anomaly detector, which consists of a single anomaly detection algorithm, learned features / models (e.g., 425), and a single feature / selection transformation configuration (e.g., 440). In some aspects, process 800 defines an integrated anomaly detector, for example, having information about... Figure 2C Anomaly detector for the described structure.

[0068] In some respects, the following are about Figure 8 One or more of the functions discussed can be executed by a hardware processing circuitry system. For example, in some embodiments, instructions stored in memory configure the hardware processing circuitry system to perform the following... Figure 8 One or more of the functions or operations discussed. In some respects, the following refers to... Figure 8 One or more of the functions discussed operate in parallel. For example, multiple anomaly detector configurations can process event data in parallel (e.g., multiple parallel executions of operations 810 and 815 discussed below) and generate results, which are then compared in operation 825.

[0069] In operation 805, the result set is initialized. The result set is designed to store results that reflect the accuracy of the anomaly detection configuration. For example, the result set may include data indicating the correlation between the results of the anomaly detection configuration and a reference set of results. The reference set of results may indicate the expected result set or an acceptable result set. Initializing the result set involves setting the result set to values ​​indicating very low correlation. In process 800, discussed further below, the initialization values ​​are used to replace the initial result set with a first valid set of results.

[0070] In operation 810, an anomaly detection parameter set is generated. This parameter set defines the configuration of the anomaly detector. For example, as mentioned above... Figure 4 and Figure 5 The anomaly detector configuration discussed may include the definition of an algorithm for anomaly detection (e.g., 430), learned feature data (e.g., 425), and specifications for a feature selection / transformation scheme (e.g., to control component 440). In generating the anomaly detector... Figure 8 In this embodiment, multiple versions of the above parameters are generated, with each version corresponding to one of the anomaly detectors included in the integrated detector. When the integrated detector is defined by operation 810, operation 810 also defines combiner configuration data (e.g., 246) to control how the outputs of each anomaly detector in the defined anomaly detectors are combined to generate a single sort.

[0071] In some respects, the generation of parameter sets can include random components. For example, Figure 8 Operation 810 can be executed multiple times. Each execution of operation 810 changes the generated parameters to alter how anomaly detection is performed in the resulting anomaly detector (e.g., in operation 815 discussed below). In some aspects, parameter generation is performed according to a strategy. This strategy defines the methods for changing the parameters. For example, some embodiments may utilize Bayesian hyperparameter optimization and / or recommender system techniques to generate the parameters.

[0072] In operation 815, results are generated by analyzing the dataset using the anomaly detector configuration defined by the parameters generated in operation 810. For example, as mentioned above... Figure 2A Each of the anomaly detector configurations discussed in anomaly detector configurations 204a-204d reads computer event data from data storage 202 and generates results (e.g., 206a-206d respectively).

[0073] In operation 820, the results are evaluated. An indication of the correctness of the generated results is evaluated. In some aspects, the indication of correctness indicates the degree of correlation between the results and a reference set of results.

[0074] The decision operation 825 determines whether the result is superior to the stored result set. Therefore, in some respects, the decision operation 825 compares the correlation between the result obtained in operation 815 and the reference set of results with a second correlation of the results stored in the result set. If the current result reflects a greater correlation with the reference results than the stored results, then process 800 moves from decision operation 825 to operation 830.

[0075] In operation 830, the results are saved to a result set. Thus, for example, in some embodiments, the correlation between the results generated in operation 815 and a reference set of results is stored in the result set.

[0076] In operation 835, the anomaly detection parameters of the generated results are saved.

[0077] The decision operation 840 determines whether the iteration of a series of variations in the anomaly detection parameters has ended or completed (e.g., Figure 8 The iteration represents the execution of operations between each execution of operation 840. The determination operation 840 can vary depending on the embodiment. In some embodiments, the determination operation 840 evaluates whether the saved results represent a threshold level of accuracy or quality. For example, if the correlation value reflected by the saved results is higher than a predetermined threshold, the determination operation 840 determines that the iteration is complete.

[0078] In other embodiments, the determination operation 840 measures the improvement in the saved results over time or for each iteration of process 800. If the improvement over a predetermined time period is below a threshold, or the improvement over each iteration (after multiple iterations) is below a second predetermined improvement threshold, then operation 840 determines that sufficient iterations have been performed. In some other cases, the determination operation determines that process 800 has consumed a threshold amount of computational power and / or elapsed time, and that the saved results obtained so far are sufficient to complete process 800.

[0079] In these various cases, process 800 moves from operation 840 to operation 845, whereby operation 845 sends or transmits the saved parameters to the product facility (e.g., any one or more product facilities 104a-104d). The saved parameters define the anomaly detector configuration because they define one or more of the following: the anomaly detection algorithm, the model data used for the defined algorithm, and the control parameters (e.g., 442 for 440) used for the feature selection / transformation components of the anomaly detector.

[0080] Figure 9 This is a flowchart of the process for configuring the anomaly detector. In some aspects, the following... Figure 9One or more of the functions discussed can be performed by a hardware processing circuitry system. For example, in some embodiments, instructions stored in memory configure the hardware processing circuitry system to perform the functions described below. Figure 9 One or more of the functions and / or operations discussed. In some embodiments, process 900 is performed by equipment within the product facility or the local environment of the product facility (e.g., such as those mentioned above). Figure 1 (Any of the devices discussed in 104a-104d) to perform.

[0081] In operation 910, data defining the anomaly detector configuration is received. For example, as mentioned above... Figure 1 As discussed, any one or more product facilities 104a-104d receive data defining anomaly detectors from creation hub 102 via data streams 106a-106d. The data defining the anomaly detector configuration defines one or more of the following: the algorithm used by the anomaly detector (e.g., 430); data defining learned features or learned models (e.g., 425); the definition of the anomaly detection algorithm (e.g., 430); or data defining feature selection and / or transformation control parameters (e.g., configuring parameters 442 of feature transformation component 440). In some embodiments, the received data includes instructions for implementing the anomaly detection algorithm. In some other embodiments, the received data indicates a predetermined number of algorithms identified via a mapping between a predetermined number and the algorithms. In these embodiments, the device performing operation 910 includes instructions necessary for implementing the identified algorithms.

[0082] In operation 920, the anomaly detector is instantiated based on the received configuration. For example, operation 920 executes the algorithm specified by the anomaly detector configuration received in operation 910. Operation 920 provides any model data specified by the received configuration to the execution algorithm, and before features are provided to the instantiated anomaly detector (e.g., via component 440, as mentioned above). Figure 4 (Discussed), providing any feature selection / transformation control parameters specified by the configuration for use in feature transformation. Note that if the received anomaly detector configuration specifies an integrated anomaly detector, operation 920 instantiates one or more anomaly detector algorithms and provides each of the instantiated anomaly detectors with learned feature / model data (e.g., 425) and feature selection information (e.g., 440). When an integrated anomaly detector is specified by the configuration, operation 920 also instantiates a combiner component (e.g., 244) and combines the outputs of multiple anomaly detectors based on the combiner configuration (e.g., 246) specified by the anomaly detector configuration for the integrated detector.

[0083] In operation 925, the anomaly detector is trained based on local events. For example, in some aspects, the anomaly detector configuration received in operation 910 includes hyperparameter values ​​(e.g., via fields 509 and 506) for training the anomaly detector. Process 900 then receives event information (e.g., 600) generated locally (e.g., within the local environment of any product facility such as 104a-104d). These events are then used to train the instantiated anomaly detector. Note that not all embodiments train the instantiated anomaly detector locally. In some embodiments, hyperparameters are not specified in the anomaly detector configuration and / or are not specified in message section 500, as described above.

[0084] In operation 930, the instantiated anomaly detector (e.g., an integrated anomaly detector or a single anomaly detector) sorts the computer events. In some embodiments, sorting the computer events includes classifying the computer events. For example, determining the probability that each event belongs to one of a plurality of event types. For example, in some embodiments, the anomaly detector determines whether the computer file associated with the event is malware or a known good program. In some embodiments, the anomaly detector assigns a probability to each of these event types (malware / known good program).

[0085] In embodiments where an anomaly detector is trained locally via operation 925, the ordering of events is performed by the locally trained anomaly detector. In some embodiments, computer events may be events generated by the product facility in a local environment. For example, as described above regarding... Figure 2B As discussed, computer systems 224 and / or 226 can generate events that are analyzed by an instantiated anomaly detector running on the product facility equipment (e.g., 104). Some embodiments of operation 930 can generate alerts based at least on the highest-ranked events. In some embodiments, one or more of the highest-ranked events result in mitigation actions for the events. For example, as described above regarding... Figure 3 As discussed, mitigation table 320 provides identifiers for one or more mitigation actions (e.g., via field 324) associated with an event. Therefore, in at least some embodiments, if an event is ranked above a certain threshold, a mitigation action associated with that event is executed.

[0086] In operation 935, the highest-ranked event in the sorted events is mitigated. (As mentioned above...) Figure 3As discussed, some embodiments define a mapping between events and mitigation actions. Operation 935 identifies the mitigation action to be performed based on the event ranked highest by operation 930. Mitigation actions may include a number of possible actions, which vary depending on the embodiment. Various embodiments perform mitigation actions such as restarting the computer that generated the event, generating an alert to a specific messaging address (e.g., an email address or text messaging address) to warn the IT provisioning of the event, deleting one or more files associated with the event, or other actions. In some embodiments, mitigation actions are performed via the above-mentioned... Figure 3 The mitigation measures discussed in Table 320 are identified. For example, the highest-ranking event identified in operation 935 has an associated event identifier. In some embodiments, the event identifier is shared for events of the same type.

[0087] In operation 940, computer events are transmitted to the creation hub. Computer events are events that occur within the local environment of the product facility. For example, as mentioned above... Figure 2B As discussed, computer systems 224 and / or 226 generate events detected or monitored by product facility 104. The product facility then sends data indicative of these events to creation hub 102. This is in... Figure 1 This is also shown as any data stream in data streams 108a-108d. In at least some embodiments, computer events can be transmitted via the above references. Figure 6 The message portion 600 discussed is substantially identical to the message being sent. Note that in embodiments where the creation hub 102 is not separated from the product facility, the computer events identified in operation 940 are provided to the computer event data store (e.g., 202) without necessarily transmitting the computer events. For example, if the device performing process 900 has physical access to the computer event data store (e.g., 202), in some embodiments, the device can write the computer events directly to the computer event data store.

[0088] Figure 10A block diagram of an example machine 1000 is shown, on which any one or more of the techniques (e.g., methodologies) discussed herein can be implemented. In alternative embodiments, machine 1000 may operate as a standalone device or be connected (e.g., networked) to other machines. In a networked deployment, machine 1000 may operate as a server machine, a client machine, or both in a server-client network environment. In one example, machine 1000 may act as a peer-to-peer (P2P) (or other distributed) network environment. Machine 1000 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, smartphone, web device, network router, switch or bridge, server computer, database, conference room equipment, or any machine capable of executing (sequentially or otherwise) instructions specifying the actions to be taken by the machine. In various embodiments, machine 1000 may execute the above-mentioned... Figures 1-9 The process described. Furthermore, although only a single machine is shown, the term "machine" should also be understood to include any collection of machines that individually or jointly execute (or multiple instruction sets) of instructions to perform any one or more methodologies discussed herein, such as cloud computing, Software as a Service (SaaS), and other computer cluster configurations.

[0089] As described herein, examples may include logic or multiple components, modules, or mechanisms (all referred to below as "modules") or operations that can be performed on logic or multiple components, modules, or mechanisms. A module is a tangible entity (e.g., hardware) capable of performing a specified operation and can be configured or arranged in a certain way. In one example, circuitry may be arranged in a specified manner (e.g., internally or relative to an external entity such as other circuitry) as a module. In one example, all or part of one or more computer systems (e.g., standalone, client, or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application portions, or applications) as modules for performing specific operations. In one example, the software may reside on a machine-readable medium. In one example, when executed by the underlying hardware of the module, the software causes the hardware to perform a specific operation.

[0090] Therefore, the term "module" is understood to encompass tangible entities, namely entities that are physically constructed, concretely configured (e.g., hardwired), or temporarily (e.g., transitionally) configured (e.g., programmed) to operate or perform part or all of any of the operations described herein in a specified manner. Consider examples where modules are temporarily configured; it is not necessary to instantiate each module at any given time. For example, in cases where modules include general-purpose hardware processors configured using software, the general-purpose hardware processors can be configured as distinct modules at different times. The software can accordingly configure the hardware processors, for example, to constitute a particular module at one time and different modules at different times.

[0091] Machine (e.g., computer system) 1000 may include a hardware processor 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 1004, and static memory 1006, some or all of which may communicate with each other via an interconnect (e.g., bus) 1008. Machine 1000 may also include a display unit 1010, an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In one example, display unit 1010, input device 1012, and UI navigation device 1014 may be a touchscreen display. Machine 1000 may also include a storage device (e.g., a drive unit) 1016, a signal generation device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1021 (such as a global positioning system (GPS) sensor, compass, accelerometer, or other sensor). Machine 1000 may include output controller 1028, such as serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC) etc.) connections, to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0092] Storage device 1016 may include machine-readable medium 1022 on which a set of data structures embodying or utilizing any one or more of the techniques or functions described herein, or any one or more of the techniques or functions described herein, or one or more of the instructions 1024 (e.g., software) are stored. During execution of instructions 1024 by machine 1000, instructions 1024 may also reside wholly or at least partially within main memory 1004, static memory 1006, or hardware processor 1002. In one example, one or any combination of hardware processor 1002, main memory 1004, static memory 1006, or storage device 1016 may constitute a machine-readable medium.

[0093] Although machine-readable medium 1022 is illustrated as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 1024.

[0094] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions for execution by machine 1000 and causing machine 1000 to perform any one or more of the technologies disclosed herein, or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media can include solid-state memory as well as optical and magnetic media. Specific examples of machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSDs); and CD-ROMs and DVD-ROMs. In some examples, machine-readable media can include non-transitory machine-readable media. In some examples, machine-readable media can include machine-readable media that are not transient propagating signals.

[0095] The machine 1000 can also send or receive instructions 1024 on the communication network 1026 via a transmission medium using the network interface device 1020. The machine 1000 can communicate with one or more other machines using any of a variety of transmission protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), conventional telephone (POTS) networks, and wireless data networks (e.g., referred to as…). The Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards, known as The IEEE 802.16 standard family, the IEEE 802.15.4 standard family, the Long Term Evolution (LTE) standard family, and the general standard family. (UMTS) standard family, peer-to-peer (P2P) networks, etc. In one example, network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connection to communication network 1026. In one example, network interface device 1020 may include multiple antennas for wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. In some examples, network interface device 1020 may use multi-user MIMO technology for wireless communication.

[0096] Examples as described herein may include logic or multiple components, multiple modules, or multiple mechanisms, or operations that can be performed on logic or multiple components, multiple modules, or multiple mechanisms. A module is a tangible entity (e.g., hardware) capable of performing a specified operation and may be configured or arranged in a certain way. In one example, circuitry may be arranged in a specified manner (e.g., internally or relative to an external entity such as other circuitry) as a module. In one example, all or part of one or more computer systems (e.g., standalone, client, or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application portions, or applications) as modules for performing specified operations. In one example, the software may reside on a machine-readable medium. In one example, when executed by the underlying hardware of the module, the software causes the hardware to perform a specific operation.

[0097] Example 1 is a system comprising: a hardware processing circuitry system; one or more hardware memories storing instructions, which, when executed, configure the hardware processing circuitry system to perform the following operations: analyze multiple computing system events via a trained model; determine a first ordering of the multiple computing system events based on the analysis; analyze the multiple computing system events by each of a plurality of anomaly detectors; determine a plurality of corresponding second orders of the multiple computing system events based on the analysis by each of the plurality of anomaly detectors; determine a plurality of correlations, each of the correlations being a correlation between the first ordering and a corresponding second ordering in the second ordering; select one anomaly detector from the plurality of anomaly detectors based on the plurality of correlations; and transmit data defining the selected anomaly detector from the plurality of anomaly detectors to a product facility.

[0098] In Example 2, the subject of Example 1 may optionally include at least two of a plurality of anomaly detectors utilizing equivalent anomaly detection algorithms with different values ​​for hyperparameters.

[0099] In Example 3, any one or more of the topics in Examples 1-2 may optionally include at least two of the multiple anomaly detectors that utilize different anomaly detection algorithms.

[0100] In Example 4, any one or more of the topics in Examples 1-3 may optionally be included, wherein multiple correlations are determined using Spearman's ρ, Kendall's τ, combined Spearman and Kendall, rank bicolumn, and average rank bicolumn.

[0101] In Example 5, any one or more of the topics in Examples 1-4 may optionally include one or more of the anomaly detectors, such as a single-class support vector machine (SVM) algorithm, a K-means clustering algorithm, or a local outlier factor algorithm.

[0102] In Example 6, the subject of Example 5 may optionally include at least two of the multiple anomaly detectors utilizing a local outlier factor algorithm with different locality (K) values.

[0103] In Example 7, any one or more of the topics in Examples 5-6 may optionally include at least two of the anomaly detectors utilizing a single-class SVM with different kernels selected from (radial basis function (rbf), linear, polynomial, or sigmoid).

[0104] In Example 8, any one or more of the topics in Examples 5-7 may optionally include at least two of the anomaly detectors utilizing a single-class SVM with different nu values.

[0105] In Example 9, any one or more of the topics in Examples 5-8 may optionally include at least two of the anomaly detectors utilizing K-means clustering with different locality (K) values.

[0106] In Example 10, the topics of any one or more of Examples 1-9 may optionally be included, wherein relevance is determined based on the highest-ranked portion of each sort in the sorting.

[0107] In Example 11, any one or more of the topics in Examples 1-10 may optionally include an operation that further includes adjusting the second sort via outlier preservation normalization.

[0108] In Example 12, the subject of any one or more of Examples 1-11 may also optionally include an operation that further includes generating multiple computing system events via cluster-based feature transformations of multiple intermediate computing system events.

[0109] In Example 13, the subject of Example 12 optionally includes an operation that further includes generating multiple computing system events by projecting onto the set of features that change most frequently within a plurality of intermediate computing system events.

[0110] In Example 14, the subject matter of any one or more of Examples 1-13 may optionally include an operation that further includes: receiving a second plurality of computer system events from a product facility; performing a second analysis on the second plurality of computer system events by each of the second plurality of anomaly detectors; determining a plurality of third sortings corresponding to the second plurality of computer system events based on the second analysis performed by each of the second plurality of anomaly detectors; selecting one of the second plurality of anomaly detectors based on the third sorting; and transmitting data defining the selected anomaly detector among the second plurality of anomaly detectors to the product facility.

[0111] In Example 15, any one or more of the subjects in Examples 1-14 may optionally include a second hardware processing circuitry system configured to perform a second operation, the second operation further comprising: ranking a second plurality of computer system events based on a selected anomaly detector; identifying a mitigation action associated with the highest-ranked event among the second plurality of computer system events based on the ranking; and performing the mitigation action.

[0112] Example 16 is a machine-implemented method that includes: analyzing multiple computing system events via a trained model; determining a first ranking of the multiple computing system events based on the analysis; analyzing the multiple computing system events using multiple anomaly detectors; determining multiple corresponding second rankings of the multiple computing system events based on the analysis by the multiple anomaly detectors; determining multiple correlations, each of which is a correlation between the first ranking and a corresponding second ranking in the second ranking; selecting one anomaly detector from the multiple anomaly detectors based on the multiple correlations; ranking a second plurality of computing system events based on the selected anomaly detector; identifying a mitigation action associated with the highest-ranked event among the second plurality of computing system events based on the ranking; and performing the mitigation action.

[0113] In Example 17, the subject of Example 16 may optionally include at least two of a plurality of anomaly detectors utilizing equivalent anomaly detection algorithms with different hyperparameter values.

[0114] In Example 18, the subject matter of any one or more of Examples 16-17 may optionally include at least two of the multiple anomaly detectors utilizing different anomaly detection algorithms.

[0115] In Example 19, the subject matter of any one or more of Examples 16-18 may optionally include the determination of multiple correlations using Spearman's ρ, Kendall's τ, combined Spearman and Kendall, rank bicolumn, and average rank bicolumn.

[0116] Example 20 is a non-transitory computer-readable storage medium including instructions that, when executed, configure a hardware processing circuitry system to perform the following operations: analyze multiple computing system events via a trained model; determine a first ordering of the multiple computing system events based on the analysis; analyze the multiple computing system events by multiple anomaly detectors; determine multiple corresponding second orders of the multiple computing system events based on the analysis by the multiple anomaly detectors; determine multiple correlations, each of which is a correlation between the first ordering and a corresponding second ordering in the second ordering; select one anomaly detector from the multiple anomaly detectors based on the multiple correlations; and transmit data defining the selected anomaly detector from the multiple anomaly detectors to a product facility.

[0117] In Example 21, the subject of Example 20 may optionally include at least two of a plurality of anomaly detectors utilizing equivalent anomaly detection algorithms with different values ​​for hyperparameters.

[0118] In Example 22, the subject matter of any one or more of Examples 20-21 may optionally include at least two of the multiple anomaly detectors utilizing different anomaly detection algorithms.

[0119] In Example 23, the subject matter of any one or more of Examples 20-22 may optionally include the determination of multiple correlations using Spearman's ρ, Kendall's τ, combined Spearman and Kendall, rank bicolumn, and average rank bicolumn.

[0120] In Example 24, the subject matter of any one or more of Examples 20-23 may optionally include, wherein the multiple anomaly detectors include one or more of a class of Support Vector Machine (SVM) algorithms, K-means clustering algorithms, or local outlier factor algorithms.

[0121] In Example 25, the subject of Example 24 may optionally include at least two of the multiple anomaly detectors utilizing a local outlier factor algorithm with different locality (K) values.

[0122] In Example 26, the subject of any one or more of Examples 24-25 may optionally include at least two of the multiple anomaly detectors utilizing a single-class SVM with different kernels selected from (radial basis function (rbf), linear, polynomial, or sigmoid).

[0123] In Example 27, any one or more of the topics in Examples 24-26 may optionally include at least two of the anomaly detectors utilizing a single-class SVM with different nu values.

[0124] In Example 28, the subject of any one or more of Examples 24-27 may optionally include at least two of the multiple anomaly detectors utilizing K-means clustering with different locality (K) values.

[0125] In Example 29, the topics of any one or more of Examples 20-28 may optionally be included, wherein relevance is determined based on the highest-ranked portion of each sort in the sorting.

[0126] In Example 30, any one or more of the topics in Examples 20-29 may optionally include an operation that further includes adjusting the second sort via outlier preservation normalization.

[0127] In Example 31, any one or more of the subjects in Examples 20-30 may optionally include an operation that further includes generating the plurality of computing system events via a cluster-based feature transformation of the plurality of intermediate computing system events.

[0128] In Example 32, the subject of Example 31 optionally includes an operation that further includes generating multiple computing system events by projecting onto the set of features that change most frequently within a plurality of intermediate computing system events.

[0129] In Example 33, the subject matter of any one or more of Examples 20-32 may optionally include an operation that further includes: receiving a second plurality of computer system events from a product facility; performing a second analysis on the second plurality of computer system events by each of the second plurality of anomaly detectors; determining a plurality of third sortings corresponding to the second plurality of computer system events based on the second analysis by each of the second plurality of anomaly detectors; selecting one of the second plurality of anomaly detectors based on the third sorting; and transmitting data defining the selected anomaly detector among the second plurality of anomaly detectors to the product facility.

[0130] In Example 34, the subject matter of any one or more of Examples 20-33 may optionally include an operation that further includes: ranking a second plurality of computer system events based on a selected anomaly detector; identifying a mitigation action associated with the highest-ranked event among the second plurality of computer system events based on the ranking; and performing the mitigation action.

[0131] Therefore, the term "module" is understood to include tangible entities, that is, entities that are physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., transitionally) configured (e.g., programmed) to operate or perform part or all of any of the operations described herein in a specified manner. Consider an example where modules are temporarily configured; it is not necessary to instantiate each module in the module at any given time. For example, in the case where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. Thus, software can configure the hardware processor, for example, to constitute a particular module at one time and different modules at different times.

[0132] Various embodiments can be implemented entirely or partially as software and / or firmware. This software and / or firmware can take the form of instructions contained in or on a non-transitory computer-readable storage medium. These instructions can then be read and executed by one or more processors to perform the operations described herein. The instructions can be in any suitable form, such as, but not limited to, source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Such computer-readable media can include any tangible non-transitory medium for storing information in one or more computer-readable forms, such as, but not limited to, read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory, etc.

Claims

1. A system comprising: Hardware processing circuit system; as well as One or more hardware memories, the one or more hardware memories storing instructions, which, when executed, configure the hardware processing circuitry to perform operations, the operations including: The analysis of multiple computing system events is performed using a trained model, the analysis including processing event data via cluster-based feature transformation, wherein the multiple computing system events are generated by projecting onto the set of most frequently occurring features that change within the event data; Based on the analysis, a first order of the plurality of computing system events is determined; The events of the multiple computing systems are analyzed by each of the multiple anomaly detectors. Based on the analysis performed by each of the plurality of anomaly detectors, a plurality of second sorts corresponding to the plurality of computing system events are determined; Determine multiple correlations, each of which is the correlation between the first ranking and a corresponding second ranking in the second ranking; Based on the multiple correlations, one of the multiple anomaly detectors is selected; Based on the selected anomaly detector, the second plurality of computer system events are sorted; Based on the ranking, identify mitigation actions associated with the highest-ranking event among the second plurality of computer system events; and Perform the mitigation action.

2. The system of claim 1, wherein at least two of the plurality of anomaly detectors utilize equivalent anomaly detection algorithms having different values ​​for hyperparameters.

3. The system according to claim 1, wherein at least two of the plurality of anomaly detectors utilize different anomaly detection algorithms.

4. The system of claim 1, wherein the determination of the plurality of correlations is made using Spearman's p, Kendall's t, combined Spearman and Kendall, rank biarray, and average rank biarray.

5. The system according to claim 1, wherein the plurality of anomaly detectors comprises one or more of the following algorithms: single-class support vector machine (SVM) algorithm, K-means clustering algorithm, or local outlier factor algorithm.

6. The system of claim 5, wherein at least two of the plurality of anomaly detectors utilize a local outlier factor algorithm with different locality K values.

7. The system of claim 5, wherein at least two of the plurality of anomaly detectors utilize a single-class SVM with different kernels, the different kernels being selected from radial basis functions (RBF), linear, polynomial, or sigmoid.

8. The system of claim 5, wherein at least two of the plurality of anomaly detectors utilize a single-class SVM with different nu values.

9. The system of claim 5, wherein at least two of the plurality of anomaly detectors utilize K-means clustering with different locality K values.

10. The system of claim 1, wherein the correlation is determined based on the highest-ranked portion of each of the first and second rankings.

11. The system of claim 1, wherein the operation further includes adjusting the second sorting via outlier retention normalization.

12. The system according to claim 1, wherein the operation further comprises: Receive a second or more computer system events from the product facility; A second analysis is performed on the events of the second plurality of computer systems by each of the second plurality of anomaly detectors; Based on the second analysis performed by each of the second plurality of anomaly detectors, a plurality of third sorts corresponding to the events of the second plurality of computing systems are determined; Based on the third sorting, select one of the second plurality of anomaly detectors; as well as The data of the selected anomaly detector from the second plurality of anomaly detectors is transmitted to the product facility.

13. The system according to claim 1, further comprising: A second hardware processing circuit system is configured to perform a second operation, the second operation including: Based on the selected anomaly detector, the second plurality of computer system events are sorted; Based on the ranking, identify mitigation actions associated with the highest-ranking event among the second plurality of computer system events; and Perform the mitigation action.

14. A machine-implemented method, the method comprising: The analysis of multiple computing system events is performed using a trained model, the analysis including processing event data via cluster-based feature transformation, wherein the multiple computing system events are generated by projecting onto the set of most frequently occurring features that change within the event data; Based on the analysis, a first order of the plurality of computing system events is determined; Multiple anomaly detectors are used to analyze the events of the multiple computing systems; Based on the analysis performed by the plurality of anomaly detectors, a plurality of second sorts corresponding to the plurality of computing system events are determined; Determine multiple correlations, each of which is the correlation between the first ranking and a corresponding second ranking in the second ranking; Based on the multiple correlations, one of the multiple anomaly detectors is selected; Based on the selected anomaly detector, the second plurality of computer system events are sorted; Based on the ranking, identify the anomaly mitigation action associated with the highest-ranking event among the second plurality of computer system events; and Perform the mitigation action.

15. The method of claim 14, wherein at least two of the plurality of anomaly detectors utilize equivalent anomaly detection algorithms having different values ​​for hyperparameters.

16. The method of claim 14, wherein at least two of the plurality of anomaly detectors utilize different anomaly detection algorithms.

17. The method of claim 14, wherein the determination of the plurality of correlations is made using Spearman's p, Kendall's t, combined Spearman and Kendall, rank biarray, and average rank biarray.

18. A non-transient computer-readable storage medium, the computer-readable storage medium comprising instructions that, when executed, configure a hardware processing circuitry system to perform an operation, the operation comprising: The analysis of multiple computing system events is performed using a trained model, the analysis including processing event data via cluster-based feature transformation, wherein the multiple computing system events are generated by projecting onto the set of most frequently occurring features that change within the event data; Based on the analysis, a first order of the plurality of computing system events is determined; Multiple anomaly detectors are used to analyze the events of the multiple computing systems; Based on the analysis performed by the plurality of anomaly detectors, a plurality of second sorts corresponding to the plurality of computing system events are determined; Determine multiple correlations, each of which is the correlation between the first ranking and a corresponding second ranking in the second ranking; Based on the multiple correlations, one of the multiple anomaly detectors is selected; Based on the selected anomaly detector, the second plurality of computer system events are sorted; Based on the ranking, identify mitigation actions associated with the highest-ranking event among the second plurality of computer system events; and Perform the mitigation action.

Images