Computer-implemented methods, information processing systems, and computer programs (performance monitoring in the anomaly detection domain for IT environments)

Abstract

To provide performance monitoring in an anomaly detection domain for an IT environment.SOLUTION: An approach is disclosed that retrieves a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model that is monitoring the computer system. The current system data and the anomaly detection model outputs are inputs to a trained anomaly detection supervisor model. The trained anomaly detection supervisor model processes the inputs and provides a set of performance data corresponding to the anomaly detection model. The anomaly detection model is then adjusted when the set of performance data indicates that the anomaly detection model is performing below a threshold.SELECTED DRAWING: Figure 3

Description

[Background technology]

[0001] With the rapid emergence of digital transformation around the world, managing information technology (IT) operations is a complex task for enterprises. Every year, companies face outages that negatively impact revenue, labor costs, corporate reputation, and customer satisfaction. When it comes to anomaly detection in an enterprise's current workflow, it is important for enterprises to have a way to efficiently monitor performance and take appropriate actions. Machine learning models are often implemented to perform anomaly detection in IT environments. However, there is currently no accurate method for monitoring the performance of these models and monitoring data drift. This information is needed to enable site reliability engineers (SREs) to assess the IT environment and make appropriate decisions regarding actions to take regarding anomaly detection. System data, such as system logs, metric data, and topology information, change from time to time. It is difficult to understand when an anomaly detection machine learning model needs retraining, especially when there is a lack of understanding about the performance of the current anomaly detection model. Training an anomaly detection model requires time and resources. As a result, SREs try to avoid unnecessary training of anomaly detection models. However, a poorly performing anomaly detection model may fail to accurately detect anomalies in the computer system it is monitoring, which may lead to security breaches and other system problems. Summary of the Invention [Problem to be solved by the invention]

[0002] A poorly performing anomaly detection model may fail to accurately detect anomalies in the computer system it is monitoring, which may lead to security breaches and other system problems. [Means for solving the problem]

[0003] A technique is disclosed for obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring the computer system. The current system data and the output of the anomaly detection model are inputs to a trained anomaly detection supervisor model. The trained anomaly detection supervisor model processes the inputs and provides a set of performance data corresponding to the anomaly detection model. The anomaly detection model is then adjusted when the set of performance data indicates that the performance of the anomaly detection model is below a threshold.

[0004] The foregoing is a summary and thus necessarily contains simplifications, generalizations, and omissions of detail. As a result, those skilled in the art will appreciate that this summary is merely illustrative and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the present invention will become apparent in the non-limiting detailed description set forth below. [Brief explanation of the drawings]

[0005] The present invention may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings.

[0006] [Figure 1] FIG. 1 illustrates a network environment including a knowledge manager that utilizes a knowledge base.

[0007] [Figure 2] FIG. 2 is a block diagram illustrating a processor and components of an information processing system such as that shown in FIG.

[0008] [Figure 3] FIG. 1 is a component diagram illustrating various components and interactions for implementing performance monitoring in the anomaly detection domain for an information technology (IT) environment.

[0009] [Figure 4] FIG. 10 is a flow chart illustrating the logic used by the anomaly detection supervisor.

[0010] [Figure 5] FIG. 1 is a flowchart illustrating the logic used to train a supervisory model.

[0011] [Figure 6] FIG. 10 is a flowchart illustrating the logic used by a trained supervisory model to monitor the performance of an anomaly detection model. DETAILED DESCRIPTION OF THE INVENTION

[0012] 1-6 illustrate an approach for performing performance monitoring in the anomaly detection domain for an information technology (IT) environment. The technique described herein relates to automatically monitoring performance in the anomaly detection domain by using machine learning and rule-based methods to determine model performance. The technique monitors distribution changes and microservice interactions from a topology perspective and provides continuous model enhancements from user feedback. System data, such as system logs, metric data, and topology information, as well as output (data) from anomaly detection models currently running in the environment, are used as inputs to the technique. These inputs are utilized by a supervised machine learning model to monitor the performance of the anomaly detection model and monitor data drift. Data drift refers to changes over time in the distribution of data used by the anomaly detection model. If the baseline dataset on which the anomaly detection model was trained differs from the current real-time production data being processed by the anomaly detection model, model performance can sometimes deteriorate significantly. The technique generates performance monitoring results and recommended actions for existing trained models for anomaly detection by utilizing a supervised machine learning model and rule-based methods to determine whether retraining is necessary. Furthermore, the technique can identify thresholds that may require tuning. The method monitors the performance of an anomaly detection model and trains a supervisory machine learning model to detect data drift in the model data. The supervised machine learning model and rule-based methods determine the accuracy rate of the anomaly detection model and identify recommended actions. In this way, the method improves existing anomaly detection models in IT environments and helps more accurately identify anomalies in IT operations environments.

[0013] This approach addresses the problem of poorly performing anomaly detection models in three main areas. First, the approach analyzes the performance of anomaly detection models by examining evaluation metrics such as precision, accuracy, and F1 score when "ground truth" exists. Second, a supervisory machine learning model predicts distribution changes based on the current anomaly detection model and makes suggestions to improve the performance and accuracy of the anomaly detection model. The supervisory model is trained based on system data and data (e.g., logs, metrics, topology information, etc.) and output from the current anomaly detection model (e.g., inference dataset, count vector, etc.). To monitor distribution changes, log templates and count vectors are generated from input logs and continuously monitored for data drift. The log templates, count vectors, and metric data are also continuously collated to determine whether the proportion is below a predefined threshold for a given time window. In one embodiment, the log templates are generated from all input logs, and the count vector is the distribution of templates extracted from log messages. If the ratio is persistently below the threshold, it suggests persistent data drift, and then a retraining of the model is suggested to the SRE. Furthermore, a rule-based comparison is performed between the old and new inference datasets generated by the current anomaly detection model to assist in data drift analysis and providing recommendations to the SRE. Furthermore, the interactions of different microservices are monitored with the provided topology information. Monitoring of microservice interactions provides suggestions for reconfiguring the connections of various microservices. Finally, user feedback is used to adjust the thresholds of the supervisor model as well as tune future recommended actions. In this way, this approach provides a dynamic monitoring system and context-specific recommended actions regarding threshold tuning of the anomaly detection model for each microservice along with retraining of the anomaly detection model.

[0014] FIG. 1 shows a schematic diagram of one illustrative embodiment of an artificial intelligence (AI) system 100 on a computer network 102. The AI ​​system 100 includes an artificial intelligence computing device 104 (comprising one or more processors and one or more memories, and possibly any other computing device including buses, storage devices, communication interfaces, etc., as generally known in the art), which connects the AI ​​system 100 to the computer network 102. The network 102 may include multiple computing devices 104 that communicate with each other and with other devices or components via one or more wired and / or wireless data communication links, each of which may comprise one or more wires, routers, switches, transmitters, receivers, etc. The AI ​​system 100 and network 102 may enable functionality such as question / answer (QA) generation functionality for one or more content users. Other embodiments of the AI ​​system 100 may be used with components, systems, subsystems, or devices, or combinations thereof, other than those shown herein.

[0015] The AI ​​system 100 maintains a knowledge base 106, also known as a "corpus," which is a store of information or data that the AI ​​system uses to solve problems. The knowledge base contains the base set of facts, assumptions, models, and rules that the AI ​​system has available to solve problems.

[0016] AI system 100 may be configured to receive input from a variety of sources. For example, AI system 100 may receive input from network 102, electronic documents 107 or other corpora of data, content creators, content users, and other possible input sources. In one embodiment, some or all of the inputs to AI system 100 may be routed through network 102. Various computing devices on network 102 may include access points for content creators and content users. Some of the computing devices may include database devices storing corpora of data. Network 102 may include local network connections and remote connections in various embodiments, allowing artificial intelligence 100 to operate locally and globally, in environments of any size, including the Internet. Furthermore, artificial intelligence 100 functions as a front-end system that can make available various knowledge extracted from or represented in documents, network-accessible sources, or structured data sources, or combinations thereof. In this manner, some processes feed artificial intelligence, including input interfaces for receiving knowledge requests and responding accordingly.

[0017] In one embodiment, content creators produce content for electronic documents 107 for use by AI system 100 as part of a corpus of data. Electronic documents 107 may include any file, text, article, or data source for use by AI system 100. Content users may access AI system 100 via a network or internet connection to network 102 and, in one embodiment, may input questions to AI system 100, which may be answered by content from the corpus of data. As described further below, when a process evaluates a given section of a document for semantic content, the process may use various conventions to query the artificial intelligence.

[0018] The types of information handling systems that can utilize AI system 100 range from small handheld devices, such as handheld computer / cell phone 110, to large mainframe systems, such as mainframe computer 170. Examples of handheld computers 110 include personal digital assistants (PDAs), personal entertainment devices, such as MP3 players, portable televisions, and compact disc players. Other examples of information handling systems include pen or tablet computers 120, laptop or notebook computers 130, personal computer systems 150, and servers 160. As shown, various information handling systems can be networked together using computer network 102. Types of computer networks 102 that can be used to interconnect various information handling systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, public switched telephone networks (PSTNs), other wireless networks, and any other network topologies that can be used to interconnect information handling systems. Many information handling systems include non-volatile data stores, such as hard drives, and / or non-volatile memory. 1 show separate non-volatile data stores (server 160 utilizes non-volatile data store 165, and mainframe computer 170 utilizes non-volatile data store 175). The non-volatile data stores may be components external to the various information handling systems or may be internal to one of the information handling systems. An illustrative example of an information handling system showing an exemplary processor and various components commonly accessed by the processor is shown in FIG.

[0019] 2 illustrates an information handling system 200, and more particularly, processors and common components, which are a simplified example of a computer system capable of performing the computing operations described herein. The information handling system 200 includes one or more processors 210 coupled to a processor interface bus 212. The processor interface bus 212 connects the processors 210 to a northbridge 215, also known as a memory controller hub (MCH). The northbridge 215 is connected to system memory 220 and provides a means for the processors 210 to access the system memory. A graphics controller 225 is also connected to the northbridge 215. In one embodiment, a PCI Express bus 218 connects the northbridge 215 to the graphics controller 225. The graphics controller 225 is connected to a display device 230, such as a computer monitor.

[0020] The Northbridge 215 and Southbridge 235 are connected to each other using bus 219. In one embodiment, this bus is a Direct Media Interface (DMI) bus, which transfers data in each direction between the Northbridge 215 and Southbridge 235 at high speeds. In another embodiment, a Peripheral Component Interconnect (PCI) bus connects the Northbridge and Southbridge. The Southbridge 235, also known as an I / O Controller Hub (ICH), is a chip that generally implements performance that runs slower than that offered by the Northbridge. The Southbridge 235 typically provides various buses used to connect various components. These buses include, for example, PCI and PCI Express buses, ISA buses, System Management Buses (SM Buses or SMBs), or Low Pin Count (LPC) buses, or combinations thereof. The LPC bus often connects low-bandwidth devices such as Boot ROM 296 and "legacy" I / O devices (which use "super I / O" chips). The "legacy" I / O devices (298) may include, for example, serial and parallel ports, a keyboard, a mouse, or a floppy disk controller, or a combination thereof. The LPC bus also connects the southbridge 235 to a trusted platform module (TPM) 295. Other components often included in the southbridge 235 include a direct memory access (DMA) controller, a programmable interrupt controller (PIC), and a storage device controller, which use bus 284 to connect the southbridge 235 to a non-volatile storage device 285, such as a hard disk drive.

[0021] The ExpressCard 255 is a slot that connects hot-pluggable devices to the information handling system. The ExpressCard 255 connects to the Southbridge 235 using both a Universal Serial Bus (USB) and a PCI Express bus, thereby supporting PCI Express and USB connections. The Southbridge 235 includes a USB controller 240 that provides USB connectivity for devices connected to the USB. These devices include a webcam (camera) 250, an infrared (IR) receiver 248, a keyboard and trackpad 244, and a Bluetooth® device 246 that provides a wireless personal area network (PAN). The USB controller 240 also provides USB connectivity for a wide variety of other USB-connected devices 242, such as a mouse, a removable non-volatile storage device 245, a modem, a network card, an ISDN connector, a fax machine, a printer, a USB hub, and many other types of USB-connected devices. Although the removable non-volatile storage device 245 is shown as a USB-connected device, the removable non-volatile storage device 245 may be connected using a different interface, such as a Firewire® interface.

[0022] A wireless local area network (LAN) device 275 is connected to the Southbridge 235 via a PCI or PCI Express bus 272. The LAN device 275 typically implements one of the IEEE 802.11 standards for over-the-air modulation technology, all of which use the same protocol for wireless communication between the information handling system 200 and another computer system or device. An optical storage device 290 is connected to the Southbridge 235 using a Serial ATA (SATA) bus 288. Serial ATA adapters and devices communicate over a high-speed serial link. The Serial ATA bus also connects the Southbridge 235 to other forms of storage devices, such as hard disk drives. An audio circuit 260, such as a sound card, is connected to the Southbridge 235 via bus 258. The audio circuit 260 also provides features such as an audio line-in and optical digital audio input port 262, an optical digital output and headphone jack 264, an internal speaker 266, and an internal microphone 268. Ethernet controller 270 is connected to southbridge 235 using a bus such as a PCI or PCI Express bus. Ethernet controller 270 connects information handling system 200 to computer networks such as local area networks (LANs), the Internet, and other public and private computer networks.

[0023] While Figure 2 illustrates one information handling system, an information handling system may take many forms, some of which are illustrated in Figure 1. For example, an information handling system may take the form of a desktop, server, portable, laptop, notebook, or other form factor computer or data processing system. Additionally, an information handling system may take other form factors, such as a personal digital assistant (PDA), gaming device, ATM machine, portable telephone device, communications device, or other device that includes a processor and memory.

[0024] 3 is a component diagram illustrating various components and interactions for implementing performance monitoring in the anomaly detection domain for information technology (IT) environments. A computer system 300 is monitored for anomalies by an anomaly detection artificial intelligence (AI) system 320, which uses a trained anomaly detection model 330 to detect anomalies in the system 300. The anomaly detection system 320 receives data from the monitored computer system as various system data 310 collected from the system. This system data includes system logs, performance metric data (e.g., average load data, available memory data, CPU usage data, available disk space data, running process data, etc.). The anomalies detected by the anomaly detection AI system 320 are provided to a system administrator 340. The system administrator can then take action to address the anomalies and improve the performance of the system 300.

[0025] Anomaly detection supervisor 360 is an AI system that utilizes a trained supervisory model 370 to determine whether anomaly detection model 330 is operating within parameters. The anomaly detection supervisor receives inputs 350 that include both system data and the output of the anomaly detection model. System data inputs include system logs, metrics, topology information, etc. Outputs from the anomaly detection model that are used as inputs to supervisory model 370 include an inference dataset, a count vector, and other outputs of the anomaly detection model.

[0026] After inputting the data 350, the trained anomaly detection supervisor provides results 380. These results may include recommended adjustments to the anomaly detection model. These adjustments may include recommendations to retrain the anomaly detection model, recommendations to tune thresholds used by the anomaly detection model, and other adjustments to the anomaly detection model. The recommended changes are provided to a site reliability engineer (SRE) or other specialist responsible for maintaining the anomaly detection system. The SRE can then implement the recommended model adjustments to the anomaly detection model, such as retraining the model.

[0027] Figure 4 is a flowchart diagram illustrating the logic used by the anomaly detection supervisor. The process of Figure 4 begins at 400 and shows the steps taken by a process performed by the anomaly detection supervisor artificial intelligence (AI) system. At step 410, the process checks the status of the supervisory machine learning model used by the anomaly detection supervisor. The process determines whether training of the supervisory model is required, such as whether this is the first time the anomaly detection supervisor has been run on this system (decision 420).

[0028] If supervisory model training is required, decision 420 branches to the "Yes" path, where a supervisory model training process is performed in predefined process 430 (see FIG. 5 and corresponding text for details). As shown, this results in training data being provided to supervisory model 370, thereby training this model for the anomaly detection model being used in the computer system. On the other hand, if the supervisory model has already been trained, decision 420 branches to the "No" path, and steps 440-470 are performed.

[0029] In predefined process 440, the process implements anomaly supervision using a trained supervisory model (see FIG. 6 and corresponding text for details). As shown, current anomaly detection model data is input to anomaly detection supervisor 360, which has a trained supervisory model 370. Based on training of the supervisory model, the supervisor returns evaluation data to predefined process 440. The evaluation data is processed to form recommendations for tuning and managing the anomaly detection model stored in data store 380. The process determines whether a set of ground truth data has been defined for the anomaly detection system (decision 450). If ground truth has been defined, decision 450 branches to the “yes” branch, where in step 460 the process analyzes the use of evaluation metrics of the anomaly model with the predefined ground truth data (e.g., ground truth for precision, accuracy, F1 score, etc.). The decisions made based on the ground truth data are formed into recommendations for anomaly detection models and stored in data store 380. On the other hand, if ground truth has not been defined, decision 450 branches to the "No" branch, bypassing step 460. In step 470, the process provides the recommendations stored in data store 380 to a site reliability engineer (SRE) 390.

[0030] The process determines whether to continue monitoring the anomaly detection model using the supervisor (decision 480). If monitoring of the anomaly detection model continues, decision 480 branches to the "yes" branch, which cycles back to step 410 and the process described above is repeated. This cycle continues until monitoring of the anomaly detection model ceases, such as when the system is shut down, at which point decision 480 branches to the "no" branch and exits the cycle. The processing of Figure 4 then ends at 495.

[0031] Figure 5 is a flowchart diagram illustrating the logic used to train a supervisory model. The process of Figure 5 begins at 500 and shows the steps taken by a process to perform supervisory model training. In step 520, the process selects a first set of historical data from a data store 540. The historical data used to train the supervisory model includes data such as log templates, predefined thresholds, inference data sets, count vectors, system metrics, system topology information, and other system data. In step 560, the process trains an anomaly detection supervisory model 370 using the selected set of historical data.

[0032] The process determines whether there is more historical data to use to train the supervisory model (decision 580). If more historical data is available to train the supervisory model, decision 580 branches to the "yes" branch, which cycles back to step 520 to select the next set of historical data for training model 370. This cycle continues until all of the available historical data has been used to train the model, at which point decision 580 branches to the "no" branch and exits the cycle. The processing of Figure 5 then returns to the calling routine at 595 (see Figure 4).

[0033] Figure 6 is a flowchart diagram illustrating the logic used by a trained supervisory model to monitor the performance of an anomaly detection model. The process of Figure 6 begins at 600 and shows the steps taken by the trained supervisory model to process current anomaly data to determine whether the performance of the anomaly detection model is sufficient. In step 610, the process provides system data (e.g., current system logs, metric data, topology information) and current output from the anomaly detection model (e.g., inference data set, count vector, etc.) to an anomaly detection supervisor 360 with a trained supervisory model. The anomaly detection supervisor 360 performs various analyses (620-660) using the provided data to form results regarding the performance of the anomaly detection model.

[0034] In analysis 620, the anomaly detection supervisor matches the log templates and compares them to a predefined log template threshold. In analysis 630, the anomaly detection supervisor compares the current inference dataset obtained from the anomaly detection model with the training inference dataset used to train the supervisor model. In analysis 640, the anomaly detection supervisor compares the reference count vector obtained from the anomaly detection model within a given time frame with the current count vector for the same time frame. In analysis 650, the anomaly detection supervisor checks the metric data for data drift. In analysis 660, the anomaly detection supervisor compares the current inference dataset with the previous inference dataset.

[0035] The analysis performed by the anomaly detection supervisor forms a set of results, which are received in step 670 and stored in data store 680. In step 690, the process generates a set of recommendations based on the anomaly detection supervisor results stored in data store 680. For example, the recommendation may be to retrain the anomaly detection model because of data drift detected by the anomaly detection supervisor. The processing of Figure 6 then returns to the calling routine in 695 (see Figure 4).

[0036] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0037] A computer-readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved ridge structures having instructions recorded thereon, and any suitable combination thereof. As used herein, computer-readable storage media should not be construed as being transitory signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0038] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0039] The computer-readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer-readable program instructions may run entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter situation, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection to an external computer may be made (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to implement aspects of the present invention.

[0040] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0041] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to create a machine, whereby the instructions, executed by the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions / acts specified in the blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the computer-readable storage medium having instructions stored therein comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in the blocks of the flowcharts and / or block diagrams.

[0042] Also, the computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to create a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in the flowchart and / or block diagram blocks.

[0043] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or several blocks may possibly be executed in the reverse order, depending on the functionality required. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may execute a combination of dedicated hardware and computer instructions.

[0044] While particular embodiments have been shown and described, it will be apparent to those skilled in the art that changes and modifications may be made based on the teachings herein without departing from the present invention and its broader aspects. Accordingly, the appended claims are intended to encompass within their scope all such changes and modifications as are within the true spirit and scope of the present invention. It should be further understood that the present invention is defined solely by the appended claims. Where a specific number of introduced claim elements is intended, such intention will be expressly stated in the claims, and those skilled in the art will understand that no such limitation exists in the absence of such statement. As a non-limiting example and as an aid to understanding, the following appended claims include the use of the introductory phrases "at least one" and "one or more" to introduce claim elements. However, the use of such phrases should not be construed to suggest that the introduction of a claim element by the indefinite article "a" or "an" limits any particular claim containing the claim element so introduced to an invention containing only one such element, even if the same claim contains the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an," and the same applies to the use of definite articles in a claim.

Claims

1. 1. A computer-implemented method implemented by an information handling system including a processor and a memory, comprising: obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model; adjusting the anomaly detection model in response to the set of performance data indicating performance of the anomaly detection model is below a threshold; training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model; and wherein the training comprises: inputting a training inference data set into the anomaly detection supervisor model, the training inference data set being an output of the anomaly detection model monitoring the computer system; A method in which the adjustment of the anomaly detection model is selected from the group consisting of retraining the anomaly detection model, changing one or more metrics used by the anomaly detection model, and changing one or more thresholds used by the anomaly detection model.

2. receiving a current inference data set that is output from the anomaly detection model monitoring the computer system; using the trained anomaly detection supervisor model to compare the current inference data set with one or more previous inference data sets, one of the one or more previous inference data sets being the training inference data set, the comparison resulting in one of the performance data indicating whether the performance of the anomaly detection model is below the threshold; The method of claim 1 further comprising:

3. A computer-implemented method implemented by an information processing system including a processor and a memory, comprising: obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model; adjusting the anomaly detection model in response to the set of performance data indicating performance of the anomaly detection model is below a threshold; inputting the current set of system metric data into the trained anomaly detection supervisor model, the current set of system metric data being compared with one or more sets of previous system metric data, the comparison resulting in an analysis of data drift in the anomaly detection model, the analysis of data drift being one of the performance data indicating whether the performance of the anomaly detection model is below the threshold; A method comprising:

4. A computer-implemented method implemented by an information processing system including a processor and a memory, comprising: obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model; adjusting the anomaly detection model in response to the set of performance data indicating performance of the anomaly detection model is below a threshold; inputting the current set of system log templates into the trained anomaly detection supervisor model, which compares the current set of system log templates with one or more sets of previous system log templates, the comparison providing one of the performance data indicative of whether the performance of the anomaly detection model is below the threshold; A method comprising:

5. training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model, said training comprising:

5. The method of claim 1, further comprising inputting a training reference count vector corresponding to a predetermined time period into the anomaly detection supervisor model, the training reference count vector being an output of the anomaly detection model monitoring the computer system.

6. A computer-implemented method implemented by an information processing system including a processor and a memory, comprising: obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model; adjusting the anomaly detection model in response to the set of performance data indicating performance of the anomaly detection model is below a threshold; training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model, said training comprising: inputting a training reference count vector corresponding to a predetermined time period into the anomaly detection supervisor model, the training reference count vector being an output of the anomaly detection model monitoring the computer system; receiving a reference count vector output from the anomaly detection model monitoring the computer system; using the trained anomaly detection supervisor model to compare a current reference count vector with one or more previous reference count vectors, one of the one or more previous reference count vectors being the training reference count vector, the comparison providing one of the performance data indicative of whether the performance of the anomaly detection model is below the threshold; The method further comprises:

7. An information processing system, one or more processors; a memory coupled to at least one of the one or more processors; stored in the memory, obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model; adjusting the anomaly detection model in response to the set of performance data indicating performance of the anomaly detection model is below a threshold; training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model; a set of computer program instructions, executed by at least one of the one or more processors, to perform actions including: Equipped with The training inputting a training inference data set into the anomaly detection supervisor model, the training inference data set being an output of the anomaly detection model monitoring the computer system; An information processing system wherein the adjustment of the anomaly detection model is selected from the group consisting of retraining the anomaly detection model, changing one or more metrics used by the anomaly detection model, and changing one or more thresholds used by the anomaly detection model.

8. The action is receiving a current inference data set output from the anomaly detection model monitoring the computer system; using the trained anomaly detection supervisor model to compare the current inference data set with one or more previous inference data sets, one of the one or more previous inference data sets being the training inference data set, the comparison resulting in one of the performance data indicating whether the performance of the anomaly detection model is below the threshold; The information processing system of claim 7 further comprising:

9. An information processing system, one or more processors; a memory coupled to at least one of the one or more processors; stored in the memory, obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model; adjusting the anomaly detection model in response to the set of performance data indicating performance of the anomaly detection model is below a threshold; inputting the current set of system metric data into the trained anomaly detection supervisor model, the current set of system metric data being compared with one or more sets of previous system metric data, the comparison resulting in an analysis of data drift in the anomaly detection model, the analysis of data drift being one of the performance data indicating whether the performance of the anomaly detection model is below the threshold; a set of computer program instructions, executed by at least one of the one or more processors, to perform actions including: An information processing system comprising:

10. An information processing system, one or more processors; a memory coupled to at least one of the one or more processors; stored in the memory, obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model; adjusting the anomaly detection model in response to the set of performance data indicating performance of the anomaly detection model is below a threshold; inputting the current set of system log templates into the trained anomaly detection supervisor model, which compares the current set of system log templates with one or more sets of previous system log templates, the comparison providing one of the performance data indicative of whether the performance of the anomaly detection model is below the threshold; a set of computer program instructions, executed by at least one of the one or more processors, to perform actions including: An information processing system comprising:

11. The action is training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model, said training comprising:

11. The information processing system of claim 7, further comprising: inputting a training reference count vector corresponding to a predetermined time period into the anomaly detection supervisor model, the training reference count vector being an output of the anomaly detection model monitoring the computer system.

12. An information processing system, one or more processors; a memory coupled to at least one of the one or more processors; stored in the memory, obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving, from the trained anomaly detection supervisor model, a set of performance data corresponding to the anomaly detection model; adjusting the anomaly detection model in response to the set of performance data indicating performance of the anomaly detection model is below a threshold; training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model, said training comprising: inputting a training reference count vector corresponding to a predetermined time period into the anomaly detection supervisor model, the training reference count vector being an output of the anomaly detection model monitoring the computer system; receiving a reference count vector output from the anomaly detection model monitoring the computer system; using the trained anomaly detection supervisor model to compare a current reference count vector with one or more previous reference count vectors, one of the one or more previous reference count vectors being the training reference count vector, the comparison providing one of the performance data indicative of whether the performance of the anomaly detection model is below the threshold; a set of computer program instructions, executed by at least one of the one or more processors, to perform actions including: An information processing system comprising:

13. A computer program that, when executed by an information processing system, obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving a set of performance data corresponding to the anomaly detection model from the trained anomaly detection supervisor model; adjusting the anomaly detection model in response to the set of performance data indicating that performance of the anomaly detection model is below a threshold; training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model; computer program code for performing actions including: The training inputting a training inference data set into the anomaly detection supervisor model, the training inference data set being an output of the anomaly detection model monitoring the computer system; a computer program product, wherein the adjustment of the anomaly detection model is selected from the group consisting of retraining the anomaly detection model, changing one or more metrics used by the anomaly detection model, and changing one or more thresholds used by the anomaly detection model.

14. The action is receiving a current inference data set output from the anomaly detection model monitoring the computer system; using the trained anomaly detection supervisor model to compare the current inference data set with one or more previous inference data sets, one of the one or more previous inference data sets being the training inference data set, the comparison resulting in one of the performance data indicating whether the performance of the anomaly detection model is below the threshold; 14. The computer program of claim 13, further comprising:

15. A computer program that, when executed by an information processing system, obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving a set of performance data corresponding to the anomaly detection model from the trained anomaly detection supervisor model; adjusting the anomaly detection model in response to the set of performance data indicating that performance of the anomaly detection model is below a threshold; inputting the current set of system metric data into the trained anomaly detection supervisor model, the current set of system metric data being compared with one or more sets of previous system metric data, the comparison resulting in an analysis of data drift in the anomaly detection model, the analysis of data drift being one of the performance data indicating whether the performance of the anomaly detection model is below the threshold; 1. A computer program comprising computer program code for performing actions including:

16. A computer program that, when executed by an information processing system, obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving a set of performance data corresponding to the anomaly detection model from the trained anomaly detection supervisor model; adjusting the anomaly detection model in response to the set of performance data indicating that performance of the anomaly detection model is below a threshold; inputting the current set of system log templates into the trained anomaly detection supervisor model, which compares the current set of system log templates with one or more sets of previous system log templates, the comparison providing one of the performance data indicative of whether the performance of the anomaly detection model is below the threshold; 1. A computer program comprising computer program code for performing actions including:

17. The action is training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model, said training comprising:

17. The computer program of claim 13, further comprising inputting a training reference count vector corresponding to a predetermined time period into the anomaly detection supervisor model, the training reference count vector being an output of the anomaly detection model monitoring the computer system.

18. A computer program that, when executed by an information processing system, obtaining a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring said computer system; inputting the set of current system data and the set of current outputs from the anomaly detection model into a trained anomaly detection supervisor model; receiving a set of performance data corresponding to the anomaly detection model from the trained anomaly detection supervisor model; adjusting the anomaly detection model in response to the set of performance data indicating that performance of the anomaly detection model is below a threshold; training an anomaly detection supervisor model, said training resulting in said trained anomaly detection supervisor model, said training comprising: inputting a training reference count vector corresponding to a predetermined time period into the anomaly detection supervisor model, the training reference count vector being an output of the anomaly detection model monitoring the computer system; receiving a reference count vector output from the anomaly detection model monitoring the computer system; using the trained anomaly detection supervisor model to compare a current reference count vector with one or more previous reference count vectors, one of the one or more previous reference count vectors being the training reference count vector, the comparison providing one of the performance data indicative of whether the performance of the anomaly detection model is below the threshold; 1. A computer program comprising computer program code for performing actions including:

Images