Alert management in data processing systems

Abstract

Some embodiments provide for dynamically updating an alert management system that matches alerts in a data processing system with an automaton that handles the alerts using a master rule set. The method includes training a machine learning model to correlate alerts with the automaton using a training dataset including alerts successfully handled by the automaton. The machine learning model is then applied to correlate unmatched alerts that were not matched to the automaton by the master rule set with the automaton. The method further includes analyzing the behavior of the machine learning model related to the correlation of unmatched alerts to define a new rule set for matching the unmatched alerts with the automaton, and outputting the new rule set for auditing each rule in the new rule set. In response to approval of the audited rule, the audited rule is added to the master rule set.

Description

[Technical field]

[0001] The present invention relates generally to alert management in data processing systems, and more particularly to a method and system for dynamically updating an alert management system that matches alerts with automated processes that handle the alerts. [Background technology]

[0002] Alerts are generated in data processing systems when some detectable condition or other event occurs that is important for the overall management of the system operation. Processing systems such as computer networks, data centers, cloud computing systems, and other complex IT infrastructures may generate a large number of such alerts. These alerts, which contain a description of the event that occurred, must be processed and handled appropriately to mitigate the alert condition and maintain reliable system operation. Alert management systems receive and process the alerts to determine which can be handled by an automated process ("automaton") and which should be escalated to a systems engineer for response. The use of automatons allows alert management systems to manage millions of alerts per day. Alerts may be matched to automatons using predefined rules to determine which automated tool (if any) is appropriate to respond to the alert, based on the alert description.

[0003] Automata often operate with elevated privileges on potentially sensitive systems. Rule sets for deploying these tools must be carefully constructed and are subject to constant manual review and maintenance. As processing systems evolve and infrastructure changes, alerts may change accordingly. Adapting to such changes while at the same time ensuring efficient and appropriate use of automata presents a significant challenge. Summary of the Invention

[0004] One aspect of the invention provides a computer-implemented method for dynamically updating an alert management system that matches alerts in a data processing system with an automated process ("automaton") that handles the alerts using a master rule set. The method includes training a machine learning model to correlate alerts with the automaton using a training data set that includes matching alerts that were successfully handled by the automaton. The machine learning model is then applied to correlate unmatched alerts with the automaton that were not matched by the master rule set. The method further includes analyzing operation of the machine learning model related to the correlation of unmatched alerts to define a new rule set for matching the unmatched alerts with the automaton. The method further includes outputting a new rule set for auditing each rule in the new rule set. In response to approval of the audited rule, the audited rule is added to the master rule set.

[0005] Methods embodying the invention allow alert management systems to dynamically evolve and improve during system operation. The power of machine learning (ML) can be leveraged to improve system operation while overcoming the inherent conflict between the inscrutability of ML models and the need for transparency and auditing of the rules that control the deployment of automata. ML models trained on alerts that were successfully handled by matched automata are not used to simply assign automata to alerts that were mismatched by the master rule set. Instead, these alerts are applied to the model and the correlation behavior of these particular alerts is analyzed to extract new rule sets. The new rule sets derived for the mismatched alerts can then be audited before the rules are added to the master rule set. This provides an elegant and powerful technique for systematic improvement of alert management systems. Systems can continually adapt to expand their automated response capabilities and improve their handling of alerts, adapting to new alerts as the processing system evolves.

[0006] Methods embodying the invention may use a variety of ML models, even "black box" type models that are not easily understandable, such as neural networks. In these embodiments, the step of analyzing the operation of the model may use a model-independent explanation technique to identify one or more elements of each mismatched alert that are important to obtaining a correlation result for that alert. For example, ML models typically provide some kind of confidence value for the correlation result that estimates the correlation accuracy. The model may, for example, provide a set of probabilities that indicate the likelihood that each possible correlation result, i.e., classification, is correct for the input sample. With these models, the step of analyzing the operation of the model may include applying the model to correlate each mismatched alert and calculating an initial confidence value for the initial correlation result that estimates the correlation accuracy. For a different element of the alert (e.g., a particular field or string), the element is then removed and the model is reapplied to the resulting alert to calculate a new confidence value for the initial correlation result. Based on the difference between the initial confidence value and the new confidence value for the different element, at least one element of the alert may be identified as a critical element for the initial correlation result. The generation of new rule sets can then be based on key elements of the discrepancy alerts. This gives simple rule extraction techniques flexibility in the choice of ML models, allowing even black-box models to be used if desired.

[0007] In preferred embodiments, the ML model includes an ensemble tree model. In these embodiments, the step of analyzing the model's behavior may include generating new trees that define new rule sets based on features of paths through the trees of the ensemble tree where the mismatch alerts are correlated to the automaton. The analysis of the inference process for the mismatch alerts may exploit the structure of the ensemble tree model for efficient generation of new rule sets. These embodiments may exploit the understandability of the model to obtain more advanced rule sets than can be achieved with black-box models. The rule sets may be generated in a variety of ways and are advantageously adapted based on reliability and / or coverage aspects as further detailed below.

[0008] Another aspect of the present invention provides a computer program product including a computer readable storage medium embodying program instructions executable by a processing device to cause the processing device to perform a method for dynamically updating an alert management system as described above.

[0009] An additional aspect of the present invention provides an alert management system including a communications interface for receiving alerts from a data processing system, a memory storing an automaton for handling the alerts and a master rule set for matching the alerts with the automaton, and control logic configured to implement the above-described method.

[0010] Embodiments of the invention are described in more detail below, by way of illustrative and non-limiting example, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram of an example computing device for implementing methods of an alert management system in accordance with an embodiment of the present invention. [Diagram 2]FIG. 2 illustrates component modules of an alert management system in accordance with an embodiment of the present invention. [Diagram 3] 3 is a flowchart of steps of the alert management system of FIG. 2 in accordance with an embodiment of the present invention. [Figure 4] FIG. 2 illustrates a decision tree structure of an ensemble tree model used in the alert management system according to an embodiment of the present invention. [Diagram 5] FIG. 2 illustrates the operation of an analysis process, according to an embodiment of the present invention. [Figure 6] FIG. 4 illustrates the operation of an alternative analysis process, according to an embodiment of the present invention. [Figure 7] FIG. 1 illustrates steps in an analysis process using a black-box model according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present invention may be a system, method, or computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0013] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that allow instructions to be recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through an electrical wire.

[0014] The computer readable program instructions described herein may be downloaded from the computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in the computer readable storage medium in the respective computing / processing device.

[0015] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may execute completely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by individualizing the electronic circuitry with state information of the computer readable program instructions to perform aspects of the invention.

[0016] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0017] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored in a computer readable storage medium that may instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium in which the instructions are stored includes an article of manufacture including instructions for performing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0018] Computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0019] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order described in the drawings. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0020] The described embodiment may be implemented as a computer-implemented method of dynamically updating an alert management system. The method may be implemented by a computing system including one or more general-purpose or special-purpose computers, each of which may include one or more (real or virtual) machines providing functionality for performing the operations described herein. Steps of the method embodying the invention may be implemented by program instructions, e.g., program modules, executed by a processor of the system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computing system may be implemented in a distributed computing environment, such as a cloud computing environment, where tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0021] 1 is a block diagram of an example computing device for implementing an alert management system method in accordance with an embodiment of the present invention. The computing device is shown in the form of a general purpose computer 1. Components of computer 1 may include a processing unit, such as one or more processors represented by processing unit 2, a system memory 3, and a bus 4 coupling various system components including system memory 3 to processing unit 2.

[0022] Bus 4 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a high speed graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example and not limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0023] Computer 1 typically includes a variety of computer-readable media. Such media may be any available media accessible by computer 1, including volatile and non-volatile media, as well as removable and non-removable media. For example, system memory 3 may include computer-readable media in the form of volatile memory, such as random access memory (RAM) 5 or cache memory 6, or both. Computer 1 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 7 may be provided for reading from and writing to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although not shown, a magnetic disk drive for reading from and writing to removable non-volatile magnetic disks (e.g., "floppy disks"), as well as an optical disk drive for reading from or writing to removable non-volatile optical disks, such as CD-ROMs, DVD-ROMs, or other optical media, may also be provided. In such instances, each may be connected to bus 4 by one or more data media interfaces.

[0024] The memory 3 may include at least one program product having one or more program modules configured to perform the functions of embodiments of the present invention. As an example, a program / utility 8 having a set (at least one) of program modules 9 may be stored in the memory 3, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 9 generally perform the functions and / or methods of embodiments of the present invention described herein.

[0025] The computer 1 may also communicate with one or more external devices 10, such as a keyboard, pointing device, display 11, one or more devices that allow a user to interact with the computer 1, or any device that allows the computer 1 to communicate with one or more other computing devices (e.g., network cards, modems, etc.), or combinations thereof. Such communication may occur through an input / output (I / O) interface 12. Additionally, the computer 1 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or combinations thereof, through a network adapter 13. As shown, the network adapter 13 communicates with other components of the computer 1 through a bus 4. The computer 1 may also communicate with an additional processing unit 14, such as a GPU (graphics processing unit) or FPGA, that implements an embodiment of the invention. Although not shown, it is understood that other hardware and / or software components may be used in conjunction with the computer 1. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data archive storage systems, and the like.

[0026] 2 illustrates component modules of an alert management system 20 according to an embodiment of the present invention. The alert management system 20 includes a communication interface 21, a memory 22, and control logic 23, which controls system operation. The control logic 23 includes an alert manager module 25, a data collection module 26, a training module 27, and an inference & analysis module 28.

[0027] In operation, the alert management system 20 receives, via the communications interface 21, alerts generated in a data processing system, represented here by a network 29. An alert manager 25 receives and processes these alerts to determine which can be handled by an automated response process and which should be escalated to a systems engineer. In particular, the memory 22 stores a set of automated processes ("automatons") 30 that implement the automated response process. The automatons 30 are implemented by software tools A, B, C, D, E ... i where i = 1 to M, and software tool A i may be deployed by the alert manager 25 to implement processes for handling various alert events in the network 29. Such automata may handle numerous system tasks, such as archiving logs in response to facility utilization alerts, identifying and removing so-called "zombie" processes that do not produce useful output, and the like. The memory 22 also stores a master ruleset 31, which defines rules for matching alerts with appropriate automata. The master ruleset 31 provides a set of closed-form rules for deterministic matching of alerts with automata. Such rules may use regular expressions to map alerts to automata based on elements of the alert description, e.g., the content or strings of specific fields in the alert format. The alert manager 25 uses the master ruleset 31 to determine whether a received alert corresponds to an automaton A. iIf there is no match, the alert is communicated to a systems engineer, for example via an operator's graphical user interface (GUI) provided at one or more user computers. The one or more user computers may be local or remote from the system 20. If a matching automaton is identified, the alert manager deploys the automaton to handle the alert and monitors for task completion. If the resolution of the alert condition is successful, the alert may be terminated. If further action is required, the systems engineer may be notified accordingly.

[0028] The data collection module 26 monitors the overall performance of the alert handling system by storing details of all alerts, responses and outcomes in the system memory 22. This alert history is represented in Figure 2 as a database 32. This database thus accumulates details of all alerts in the system that have been successfully handled by the automaton, alerts that have been handled by the automaton but not successfully resolved, and details of any "unmatched alerts", i.e. alerts that have not been matched by the master ruleset to the automaton and therefore must be escalated for a human response.

[0029] The training module 27 and inference / analysis module 28 of the control logic 23 provide the functionality for implementing a dynamic update process for the alert management system 20. This process involves training and analysis of ML models 33 whose parameters are stored in the system memory 22. The dynamic update process is described in detail below.

[0030] The functions of the control logic modules 25-28 may generally be implemented by software (e.g., program modules) or hardware or a combination thereof. In other embodiments, functions may be allocated differently among the modules of the system, or functions of one or more modules may be combined. In general, the component modules of the alert management system 20 may be provided in one or more computers of a computing system. For example, all of the modules may be provided in the computer 1 of FIG. 1, which may be integrated into the alert management system 20, or the modules may be distributed across multiple computers / servers that may be local or remote from the alert management system 20. The system memory 22 may be implemented by one or more memory / storage components associated with one or more computers of the alert management system 20.

[0031] FIG. 3 shows a flow chart of steps of the alert management system of FIG. 2 according to an embodiment of the present invention. Step 40 represents the alert management and data collection operations performed by the alert manager module 25 and the data collection module 26 during system operation, as described above. Details of all alerts (including, in particular, non-matching alerts and alerts that have been successfully handled by the automaton) are stored in the database 32 as described above. The successfully handled alerts form a training data set that can be used to train the ML model 33. In particular, in step 41, the training module 27 uses these "successful" alerts to train the ML model 33 to correlate the alerts with the automaton via a supervised learning process. Such processes are well known and include known classifications for the training samples, here automaton A matching successful alerts, as labels for a learning process in which model parameters are gradually adapted until the model is able to correlate the alerts with the automaton to the desired accuracy. i Use.

[0032] After training of the model 33, in step 42, the inference & analysis module 28 applies the model to correlate non-matching alerts in the database 32 with the automaton. The correlation results for these inference operations are stored in the database 32. In step 43, the inference & analysis module 28 analyzes the operation of the model related to the correlation of non-matching alerts. This process examines the inference processes in the model specifically for non-matching alerts, as described in more detail below. Based on this analysis, in step 44, the inference & analysis module 28 defines a new rule set for matching non-matching alerts with the automaton. The new rule set (which may include one or more rules) is output in step 45 for auditing by a systems engineer, for example by displaying the new rules in a GUI. In response to approval of the audited rules as shown in step 46, the inference & analysis module 28 adds the audited rules to the master rule set 31 in the system memory 22 in step 47.

[0033] The above process allows the alert management system 20 to dynamically update its automatic response capabilities to reduce escalation of non-matching alerts and to incrementally improve its operation by handling alerts with new / modified alert descriptions that inevitably arise during the evolution of the monitored system. The alert management system utilizes the power of machine learning while at the same time accommodating the need for transparency and auditing of rules for deploying automata. New rules generated by analysis of reasoning behavior for non-matching alerts can be reviewed and approved to ensure that the deployed automata are appropriate for the alert events and meet the terms and conditions of any relevant customer agreements. The dynamic update process thus allows for systematic improvement of the alert management system by ensuring that all updates can be authorized with due diligence while improving the quality and reducing the cost of system maintenance.

[0034] Training of the model in step 41 of FIG. 3 may be performed when sufficient successful alerts are available or may be performed periodically as successful alerts accumulate. Training may thus continue to improve model performance during system operation. Inference on mismatch alerts may also be performed periodically, for example on batches of mismatch alerts, or as individual mismatch alerts are received to record classification results. The analysis process in step 43 may be performed when deemed appropriate for a given system, for example when a predefined number of mismatch alerts, or mismatch alerts of a given type, or mismatch alerts with a given classification result in inference step 42, have been accumulated. Analysis may also be initiated under operator control, based on monitoring of the alert history in database 32. Approval of the audited rules in step 46 may be indicated by operator input to either approve the rules as is, or to approve a modified version of the rules defined via the audit process, for example by modifying the rules in the GUI. The rules may be modified to broaden their application, for example to cover all forms / endings of a particular keyword used in the rules at the discretion of the operator.

[0035] As a simple illustration of the above process, training data may contain a number of alerts with summaries resolved using the automaton “disk compactor”, such as “disk utilization exceeded 95%” or “storage space utilization exceeded 90%”. A new system sensor may generate similar messages that do not match any existing rules at all, for example with summaries such as “disk utilization exceeded 92%”. Inference identifies “disk compactor” as the likely classification for those alerts, and analysis suggests that any alert with a summary containing the words “disk”, “utilization”, and “exceeds” should match “disk compactor”. The new rule may be manually modified during audit to allow either “exceeds” or “exceeded” before being added to the master rule set.

[0036] An example of the inference analysis process in the inference & analysis module 28 is described in more detail below. In a preferred embodiment, the ML model 33 includes an ensemble tree model, such as a boosted random forest (RF) model. The ensemble tree model includes multiple decision trees that are generated during training to map training samples to appropriate correlation results, here alerts to an automaton. The trees are generated by iteratively branching samples in the training dataset into subsets based on whether randomly selected elements of the samples satisfy a selected condition. Each resulting tree includes a hierarchy of nodes and branches, as shown in the simplified schematic diagram of FIG. 4. Each node represents a decision as to whether a specified element of the input sample satisfies a specified condition, and the branches from that node indicate the decision outcome, here "yes" or "no" (Y or N) according to whether the condition is satisfied or not. In this embodiment, the input sample is an alert, and the element to which the condition applies represents a particular field of the alert format. For example, an alert may contain a set of name:value pairs, e.g., in Java Script Object Notation (JSON) format, with name pairs such as "Header", "Summary", "Description", "Priority", etc., each with a text field that defines the corresponding "Value". A condition may test for the occurrence of a specified string in the text field of a specified element. Model training may use all or a selected subset of the alert format (e.g., Header, Summary, and Description fields) as model input for training samples, and the corresponding elements of non-matching alerts may then be used in the inference process. The inference process for a sample starts at the first (root) node of the tree, applying the relevant condition to the test sample and following the resulting branches to the next node where the next condition applies, and so on. Thus, inference for a sample defines a path through the tree from the root node to one of the terminal branches (leaves). Each leaf represents a particular correlation result, here an automaton A1-A2. MThe classification is associated with one of the following:

[0037] While Figure 4 shows a simple example of a decision tree, an ensemble tree model may contain many decision trees, with various complex algorithms used to determine branching factors, conditions, and other model parameters. The inference output of such a model typically includes some average (e.g., mode) of the classification results from all the trees with associated confidence values. The model may output, for example, a set of probabilities, one for each possible classification, with the most likely classification representing the overall correlation result.

[0038] In a preferred embodiment using an ensemble tree model, the inference analysis operations performed by the inference & analysis module 28 include generating new trees, i.e., defining a new rule set for mismatch alerts, based on features of paths through the trees of the ensemble tree where the mismatch alerts are correlated to the automaton. The features here may include specific branches with their associated conditions, or sections of paths that include more than one branch / condition in the inference path where the mismatch alerts are classified in the ensemble tree, or both. Because different trees use different sample elements / conditions in different orders in the tree structure, the preferred embodiment uses pruning to reduce the number of tree features considered in the rule extraction process. Pruning may be based on confidence values ​​associated with the paths / branches in the trees, or on the coverage, i.e., the number of mismatch alerts correlated to the automaton via those paths / branches, or both. A new tree defining a new rule set for mismatch alerts may then be generated using some or all of the remaining unpruned paths / branches.

[0039] A preferred embodiment performs aggressive pruning based on both confidence and coverage to reduce the number of tree features to a practical level for rule extraction. FIG. 5 illustrates the operation of the inference & analysis module 28 in an embodiment that uses a composition process to generate a tree for a new rule set. In step 50, inference paths in the model's decision tree are first pruned based on confidence values, which indicate the correlation accuracy, calculated for each path. The confidence values ​​here may correspond to the probability output by the model for a particular classification result as described above. These values ​​may similarly be calculated by combining (e.g., summing) the confidence values ​​associated with the individual branches of inference paths in the trees. Various techniques may be employed for pruning based on confidence values. As examples, paths below a defined threshold confidence value may be pruned, or all but a defined number of highest confidence value paths may be pruned, either globally or per class. In general, one or a combination of confidence evaluation criteria may be employed, with the overall goal of keeping only high confidence value paths.

[0040] In step 51, the resulting paths are further pruned depending on the number of mismatch alerts correlated to the automaton via the respective paths, with the aim here to keep paths with greater coverage (i.e., applying to a greater number of mismatch alerts). Again, one or a combination of various criteria may be applied here, for example, keeping only paths above a threshold coverage, keeping a defined number of highest coverage paths, etc. The overall effect of this pruning process is to discard paths that are least likely to be of significant value for the classification of mismatch alerts.

[0041] In step 52, the remaining unpruned paths are merged to generate a new tree from these paths. This process analyzes the elements / conditions associated with the nodes / branches of the paths and generates a new tree that accommodates those elements / conditions. This may involve proper ordering of the nodes and / or modification (e.g., merging) of the conditions to ensure that the new tree encompasses the broad range of decision processes defined by all the unpruned paths. In step 53, the reasoning & analysis module 28 outputs the rule set defined by the new tree. A decision tree is logically equivalent to a list of rules, and either can be built from the other. The review / audit phase (step 54) may now be distributed by assigning each path of the tree to a subject matter expert of the assigned remediation class. Once each audited rule is approved, in step 55 the new rule is added to the master rule set.

[0042] FIG. 6 illustrates the analysis operation in an embodiment where a decomposition process is used to generate a tree for a new rule set. In step 60, the inference & analysis module 28 first selects features from the model's decision tree based on the significance value associated with each feature. The significance value here is associated with the trained model feature and indicates the importance of the associated feature for correlating the alerts in the original training data set with one of the automata. When assessing the quality of any model, it is common practice to perform an analysis process to extract the model features that are most significant for each possible classification result. The results of such an analysis can be used here to obtain the most significant model features associated with the classification result obtained for the mismatch alert. In the case of the tree model of the present embodiment, the significance value can be a confidence value typically associated with each individual branch of the tree for each classification result.

[0043] After selecting the most significant features, in step 61, these features are pruned according to their importance in the inference process of the mismatch alert. In particular, the features may be pruned based on one, preferably both, considerations of confidence value and coverage. This process is similar to steps 51 and 52 of FIG. 5, but is applied to significant features, here branches, as opposed to paths. The selected significant features may first be pruned based on the number of mismatch alerts correlated to the automaton through paths that include each feature, keeping those features with higher coverage. This process may prune all features that did not appear in any inference path, features that appeared in a number of paths below a threshold, or least used features as a defined percentage, etc. The remaining features may be further pruned based on confidence values ​​calculated for the inference paths for the mismatch alert, for example, keeping only features that appeared in high-confidence inference paths, which may be identified in the various methods described above.

[0044] In step 62, a new tree is generated from the remaining unpruned features. The process again analyzes the factors / conditions associated with the remaining branches and generates a new tree that accommodates those factors / conditions as generally described above in the process of Figure 5. Subsequent steps 63, 64, and 65 correspond to steps 53, 54, and 55, respectively, of Figure 5.

[0045] The analysis process of Figures 5 and 6 may leverage the structure of an ensemble tree model to extract features of particular importance for classification of discrepant alerts and build new rule sets for these alerts accordingly. This is a powerful rule extraction technique that uses insights into model behavior for the generation of fine-tuned rule sets. However, methods embodying the invention may use other types of ML models, such as neural networks (including deep neural networks), support vector machines (SVMs), etc.

[0046] Figure 7 shows the steps of the analysis process executed by the inference & analysis module 28 in a system that employs a general black box ML model 33. The analysis process is here built on top of a Local Interpretable Model Agnostic Explanations (LIME) type of technique for explaining model predictions. At step 70, a disagreement alert containing an element e where j = 1 to n (e.g., name: value pairs) is selected for inference. At step 71, the ML model is applied with elements e1 to e as model inputs to correlate the alert to an automaton. An initial correlation result A related to a confidence value C is thus calculated and stored for the alert. The loop counter j is initialized at step 72. At step 73, the element e is removed from the alert (e.g., replaced with a null or random string), and the model is reapplied at step 74 to the resulting alert to obtain a new confidence value C for the initial correlation result A. At decision step 75, if j < n, the loop counter is incremented at step 76, and steps 73 and 74 are repeated for the next element of the alert. The inference process thus repeats removing different elements e of the alert and obtaining new confidence values C for the correlation result A. j for inference. n as model inputs to i correlate the alert to an i automaton. An initial j correlation result A related i to a confidence value C ij is thus calculated and j stored for the alert. The i loop counter j is ij initialized at step 72. At

[0047] step 73, the element e i is removed from the alert i1 (e.g., replaced with a null in or random string), and the iVarious criteria may be employed here to identify the important elements, such as selecting elements that when removed reduced the confidence value by more than a threshold amount, or selecting the element or elements associated with the largest drop in confidence value. The resulting set of important elements is then stored for the alert in question.

[0048] This above process is repeated for all non-matching alerts in the current batch, as shown in step 78. This process may be performed batch-wise, where each batch is a set of alerts for which the model is the same class / automaton A i The batch may be further limited to the group of alerts for which the ML model classified above a threshold confidence value. In step 79, the inference & analysis module 28 then builds a new set of rules based on the set of key elements for the non-matching alerts. The rules are now i For example, for a given class A i For,a,class,, the rule is that alerts that contain any of the,identified sets of important elements for non-matching alerts are assigned to,that class, or the common elements of these sets are,identified by the automaton A. i should match

[0049] Figure 7 shows an example of an analysis process for a black-box model, but generally any model-independent explanation technique may be used to identify elements of the discrepant alert that are significant to the correlation results obtained for the alert. A rule set may then be conventionally constructed based on the significant elements of the discrepant alert.

[0050] It can be seen that embodiments of the present invention provide an improved alert management system that can dynamically evolve to enhance their automatic response capabilities while preserving the security of audited deterministic rule sets for automatic remediation procedures. It should of course be understood that many changes and modifications can be made to the exemplary embodiments described above. For example, the steps of the flow diagrams may be performed in a different order than that illustrated, and some steps may be performed in parallel, if desired. Generally, when features are described herein with reference to a method embodying the invention, the corresponding features may be provided in a system / computer program product embodying the invention, and vice versa.

[0051] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for dynamically updating an alert management system that uses a master rule set to match alerts in a data processing system with automated processes ("automatons") that handle the alerts, the method comprising: training, by one or more processors, a machine learning model including an ensemble tree model to correlate the alerts with the automaton using a training dataset including matching alerts that have been successfully handled by the automaton; applying, by the one or more processors, the machine learning model to correlate with the automaton non-matching alerts that were not matched by the master rule set; analyzing, by the one or more processors, behavior of the machine learning model related to correlation of the mismatch alerts to define a new rule set for matching the mismatch alerts with the automaton, further comprising generating, by the one or more processors, a new tree based on features of a path through a tree in the ensemble tree model where the mismatch alert is correlated with the automaton, the new tree defining the new rule set; outputting, by the one or more processors, the new rule set for auditing of each rule in the new rule set; adding, by the one or more processors, in response to approval of the audited rules of the new rule set to the master rule set; 4. A computer-implemented method comprising:

2. pruning the paths based on a confidence value calculated for each path by the one or more processors, the confidence value being an estimate of correlation accuracy; generating, by the one or more processors, the new tree based on features of the unpruned paths; The computer-implemented method of claim 1 , further comprising:

3. pruning the paths based on a number of the inconsistency alerts correlated with the automaton via each path by the one or more processors; generating, by the one or more processors, the new tree based on features of the unpruned paths; The computer-implemented method of claim 1 , further comprising:

4. pruning the paths based on a confidence value calculated for each path by the one or more processors and a number of the non-matching alerts correlated with the automaton via each path, the confidence value being an estimate of correlation accuracy; generating, by the one or more processors, the new tree based on features of the unpruned paths; The computer-implemented method of any one of claims 1 to 3, further comprising:

5. Analyzing the operation of the machine learning model selecting, by the one or more processors, features from the trees of the ensemble tree model based on significance values, each significance value indicating the importance of an associated feature for correlating alerts in the training dataset with one of the automatons to which a respective mismatch alert is correlated; generating, by the one or more processors, the new tree based on the selected features; The computer-implemented method of any one of claims 1 to 4, further comprising:

6. pruning the selected features based on a number of the non-matching alerts correlated with the automaton via the paths that include each feature by the one or more processors; generating, by the one or more processors, the new tree based on unpruned features; The computer-implemented method of claim 5 , further comprising:

7. correlating, by the one or more processors, the mismatch alerts with the automaton and pruning the selected features based on a confidence value calculated for each path that included each selected feature among the selected features, the confidence value being an estimate of correlation accuracy; generating, by the one or more processors, the new tree based on unpruned features; The computer-implemented method of claim 5 or 6, further comprising:

8. pruning the selected features based on a number of the non-match alerts correlated with the automaton via the paths including each feature and based on a confidence value calculated for each path, the confidence value being an estimate of correlation accuracy; generating, by the one or more processors, the new tree based on unpruned features; The computer-implemented method of any one of claims 5 to 7, further comprising:

9. Analyzing the operation of the machine learning model in response to applying the machine learning model to obtain correlation results for discrepant alerts, identifying, by the one or more processors, at least one element of the alert as a significant element for the correlation results using model independent explanation techniques; generating, by the one or more processors, the new rule set based on the at least one element of the mismatch alert; The computer-implemented method of any one of claims 1 to 8, further comprising:

10. Analyzing the operation of the machine learning model calculating, by the one or more processors, an initial confidence value for the initial correlation result when applying the machine learning model to correlate each mismatch alert, the initial confidence value being an estimate of correlation accuracy; for a different element of the alert, removing the element and reapplying the machine learning model to the resulting alert to calculate a new confidence value for the initial correlation result, by the one or more processors; identifying, by the one or more processors, at least one element of the alert as a critical element for the initial correlation result based on a difference between the initial confidence value and the new confidence value for the different element; generating, by the one or more processors, the new rule set based on the at least one element of the mismatch alert; The computer-implemented method of any one of claims 1 to 9, further comprising:

11. 11. The computer-implemented method of claim 1, wherein the machine learning model comprises a boosted random forest model.

12. A computer program product causing a processor to carry out the method according to any one of claims 1 to 11.

13. A computer-readable recording medium having the computer program according to claim 12 recorded thereon.

14. An alert management system comprising: a communications interface for receiving alerts from a data processing system; a memory for storing an automated process ("automaton") for handling said alerts and a master rule set for matching said alerts against said automaton; training a machine learning model, including an ensemble tree model, to correlate the alerts with the automaton using a training dataset including matching alerts that were successfully handled by the automaton; applying the machine learning model to correlate non-matching alerts with the automaton that were not matched by the master rule set; analyzing behavior of the machine learning model related to correlation of the mismatched alerts to define a new rule set for matching the mismatched alerts with the automaton; generating a new tree based on features of paths through trees in the ensemble tree model where the mismatched alerts are correlated with the automaton, the new tree defining the new rule set; outputting the new rule set for auditing each rule in the new rule set; responsive to approval of the audited rule, adding the audited rule to the master rule set; and control logic configured to:

Images