Performance monitoring in anomaly detection domain for it environment

Abstract

This disclosure relates to performance monitoring in the anomaly detection domain for IT environments. A method is disclosed for retrieving a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring the computer system. The current system data and the outputs of the anomaly detection model are input to a trained anomaly detection supervisor model. The trained anomaly detection supervisor model processes the input and provides a set of performance data corresponding to the anomaly detection model. The set of performance data then indicates that the anomaly detection model is performing below a threshold, and adjusts the anomaly detection model accordingly.

Description

Technical Field

[0001] This disclosure relates to performance monitoring in the anomaly detection domain for IT environments. Background Technology

[0002] Information technology (IT) operations management is a complex task for companies because digital transformation is happening rapidly worldwide. Every year, companies face outages that negatively impact revenue, labor costs, corporate reputation, and customer satisfaction. Having methods that allow companies to efficiently monitor performance and take appropriate action when anomalies are detected in their current workflows is crucial. Machine learning models are often implemented to perform anomaly detection in IT environments. However, currently, there is no accurate way to monitor the performance of these models and monitor data drift. This information is needed to allow Site Reliability Engineers (SREs) to assess the IT environment and make appropriate decisions about actions to take regarding anomaly detection. System data, such as system logs, metrics, and topology information, changes from time to time. Understanding when an anomaly detection machine learning model needs retraining is difficult, especially with a poor understanding of the current model's performance. Training anomaly detection models is time-consuming and resource-intensive. Therefore, SREs attempt to avoid unnecessary training of anomaly detection models. However, an underperforming anomaly detection model may fail to correctly detect anomalies in the computer systems it is monitoring, potentially leading to security vulnerabilities and other system problems. Summary of the Invention

[0003] A method is disclosed for retrieving a set of current system data corresponding to a computer system and a set of current outputs from an anomaly detection model monitoring the computer system. The current system data and the anomaly detection model outputs are input into a trained anomaly detection supervisor model. The trained anomaly detection supervisor model processes the input and provides a set of performance data corresponding to the anomaly detection model. Then, when the set of performance data indicates that the performance of the anomaly detection model is below a threshold, the anomaly detection model is adjusted.

[0004] The foregoing is an overview and therefore necessarily includes simplifications, generalizations, and omissions of details; thus, those skilled in the art will recognize that this overview is merely illustrative and not intended to be limiting in any way. Other aspects, inventive features, and advantages of the invention will become clear in the following non-limiting detailed description. Attached Figure Description

[0005] The invention can be better understood by referring to the accompanying drawings, and its various objects, features, and advantages will become clear to those skilled in the art, in which:

[0006] Figure 1It describes a network environment that includes a knowledge manager that uses a knowledge base;

[0007] Figure 2 Information processing systems (such as...) Figure 1 A block diagram of the processors and components of the information processing systems shown;

[0008] Figure 3 This is a component diagram illustrating the different components and interactions used to perform performance monitoring of an information technology (IT) environment in the anomaly detection domain;

[0009] Figure 4 A flowchart depicts the logic used by the anomaly detection supervisor;

[0010] Figure 5 A flowchart depicting the logic used to train the supervised model; and

[0011] Figure 6 A flowchart illustrating the logic used by a trained supervised model to monitor the performance of an anomaly detection model is depicted. Detailed Implementation

[0012] Figure 1-6 This paper describes a method for performing performance monitoring of an information technology (IT) environment in the anomaly detection domain. The method described involves automatically monitoring performance in the anomaly detection domain using machine learning and rule-based methods to determine model performance. The method monitors distribution changes and microservice interactions from topology monitoring and provides continuous model enhancement from user feedback. System data (such as system logs, metrics, and topology information) and the output (data) from the anomaly detection model currently running in the environment are used as inputs to the method. These inputs are used to monitor the performance of the anomaly detection model and the supervised machine learning model to monitor data drift. Data drift is the change in the distribution of data used by the anomaly detection model over time. Model performance can sometimes degrade significantly when the baseline dataset on which the anomaly detection model is trained differs from the current real-time production data being processed by the anomaly detection model. This method generates performance monitoring results and recommended actions for the existing trained model for anomaly detection by leveraging the supervised machine learning model and a rule-based method to determine whether retraining is needed. Furthermore, the method can identify thresholds that may require tuning. The method trains a supervised machine learning model to monitor the performance of the anomaly detection model and detect data drift in the model data. By employing supervised machine learning models and rule-based methods, the accuracy of anomaly detection models is determined, and recommended actions are identified. In this way, the method improves existing anomaly detection models in IT environments and helps to more accurately identify anomalies in IT operational environments.

[0013] This method addresses the problem of poor anomaly detection model performance in three key areas. First, when a "ground truth" exists, the method analyzes anomaly detection model performance by examining evaluation metrics such as precision, accuracy, and F1 score. Second, a supervisory machine learning model predicts distribution changes based on the current anomaly detection model and makes recommendations to improve the model's performance and accuracy. The supervisory model is trained on system data and data (e.g., logs, metrics, topology information, etc.) and outputs from the current anomaly detection model (e.g., inference datasets, count vectors, etc.). To monitor distribution changes, log templates and count vectors are generated from the input logs, and data drift is continuously monitored. Log templates, count vectors, and metric data are continuously matched to check if the ratio falls below a defined threshold for a given time window. In one embodiment, log templates are generated from all logs used as input, and the count vectors are the distribution of templates extracted from log messages. If the ratio consistently falls below the threshold, indicating consistent data drift, retraining the model is recommended to the SRE. Furthermore, a rule-based approach is used to compare old and new inference datasets generated by the current anomaly detection model to aid data drift analysis and provide recommendations to the SRE. Additionally, the interactions between different microservices are monitored using the provided topology information. Monitoring microservice interactions provides suggestions for reconfiguring connections between different microservices. Finally, user feedback is used to adjust the supervisor model thresholds and tune future recommendation actions. In this way, the method provides a dynamic monitoring system for retraining the anomaly detection model and for threshold tuning of the anomaly detection model for each microservice, along with customized recommendation actions.

[0014] Figure 1 A schematic diagram illustrating an illustrative embodiment of an artificial intelligence (AI) system 100 in a computer network 102 is provided. The AI ​​system 100 includes an AI computing device 104 (including one or more processors and one or more memories, and potentially any other computing device elements known in the art, including buses, storage devices, communication interfaces, etc.) connecting the AI ​​system 100 to the computer network 102. The network 102 may include multiple computing devices 104 communicating with each other and with other devices or components via one or more wired and / or wireless data communication links, wherein each communication link may include one or more of a cable, router, switch, transmitter, receiver, etc. The AI ​​system 100 and the network 102 may enable functionality for one or more content users, such as question / answer (QA) generation functionality. Other embodiments of the AI ​​system 100 may be used with components, systems, subsystems, and / or devices other than those depicted herein.

[0015] AI system 100 maintains a knowledge base 106, also known as a "corpus," which is a storage of information or data extracted by the AI ​​system to solve problems. This knowledge base includes a lower-level set of facts, assumptions, models, and rules that the AI ​​system can use to solve problems.

[0016] AI system 100 can be configured to receive input from various sources. For example, AI system 100 may receive input from network 102, electronic document set 107 or other data, content creators, content users, and other possible input sources. In one embodiment, some or all of the inputs to AI system 100 can be routed through network 102. Different computing devices on network 102 may include access points for content creators and content users. Some computing devices may include devices for storing databases of data corpora. In various embodiments, network 102 may include local network connectivity and remote connectivity, enabling AI 100 to operate in environments of any size, including local and global, such as the Internet. Additionally, AI 100 acts as a front-end system that makes various kinds of knowledge extracted from or represented therein from documents, network-accessible sources, and / or structured data sources available. In this way, some processes are populated with AI that also includes an input interface that receives knowledge requests and responds accordingly. In one embodiment, a content creator creates content in electronic document 107 to use with AI system 100 as part of a data corpus. Electronic document 107 may include any file, text, article, or data source for AI system 100. Content users can access AI system 100 via a network connection to network 102 or an internet connection, and in one embodiment, can input questions into AI system 100 that can be answered based on content in a data corpus. As further described below, when the process evaluates a given portion of a document for semantic content, it can use various conventions to query the artificial intelligence.

[0017] The range of information processing systems that can utilize AI system 100 extends from small handheld devices such as handheld computers / mobile phones 110 to mainframe systems such as mainframe computers 170. Examples of handheld computers 110 include personal digital assistants (PDAs), personal entertainment devices such as MP3 players, portable televisions, and CD players. Other examples of information processing systems include pen or tablet computers 120, laptop or notebook computers 130, personal computer systems 150, and servers 160. As shown, different information processing systems can be networked together using computer network 102. Types of computer network 102 that can be used to interconnect different information processing systems include local area networks (LANs), wireless local area networks (WLANs), the Internet, the public switched telephone network (PSTN), other wireless networks, and any other network topologies that can be used to interconnect information processing systems. Many information processing systems include non-volatile data storage, such as hard disk drives and / or non-volatile memory. Figure 1 Some of the information processing systems shown depict separate non-volatile data storage (server 160 utilizes non-volatile data storage 165, mainframe 170 utilizes non-volatile data storage 175). The non-volatile data storage can be a component external to different information processing systems, or it can be internal to one of the information processing systems. Figure 2 The illustration shows an example of an information processing system, which demonstrates an exemplary processor and different components accessed by the processor.

[0018] Figure 2 An information processing system 200, and more specifically, a processor and general components, is illustrated as a simplified example of a computer system capable of performing the computational operations described herein. The information processing system 200 includes one or more processors 210 coupled to a processor interface bus 212. The processor interface bus 212 connects the processors 210 to a northbridge 215, also referred to as a memory controller hub (MCH). The northbridge 215 is connected to system memory 220 and provides the processors 210 with a means of accessing system memory. A graphics controller 225 is also connected to the northbridge 215. In one embodiment, a PCI Express bus 218 connects the northbridge 215 to the graphics controller 225. The graphics controller 225 is connected to a display device 230, such as a computer monitor.

[0019] Northbridge 215 and Southbridge 235 are connected to each other using bus 219. In one embodiment, the bus is a Direct Media Interface (DMI) bus, which transmits data at high speed in each direction between Northbridge 215 and Southbridge 235. In another embodiment, a Peripheral Component Interconnect (PCI) bus connects the Northbridge and Southbridge. Southbridge 235 (also known as the I / O Controller Hub (ICH)) is a chip that typically implements the ability to operate at a slower speed than that provided by the Northbridge. Southbridge 235 typically provides different buses for connecting different components. These buses include, for example, PCI and PCI Express buses, ISA buses, system management buses (SMBus or SMB), and / or low pin count (LPC) buses. LPC buses often connect low-bandwidth devices such as boot ROM 296 and “legacy” I / O devices (using “super I / O” chips). “Legacy” I / O devices (298) may include, for example, serial and parallel ports, keyboards, mice, and / or floppy disk controllers. The LPC bus also connects the Southbridge 235 to the Trusted Platform Module (TPM) 295. Other components typically included in the Southbridge 235 include a Direct Memory Access (DMA) controller, a Programmable Interrupt Controller (PIC), and a storage device controller that uses bus 284 to connect the Southbridge 235 to a non-volatile storage device 285 (such as a hard disk drive).

[0020] The ExpressCard 255 is a slot for connecting hot-swappable devices to an information processing system. The ExpressCard 255 supports both PCI Express and USB connectivity because it uses both Universal Serial Bus (USB) and PCI Express buses to connect to the Southbridge 235. The Southbridge 235 includes a USB controller 240, which provides USB connectivity to devices connected to USB. These devices include a webcam (camera) 250, an infrared (IR) receiver 248, a keyboard and touchpad 244, and a Bluetooth device 246 providing a wireless personal area network (PAN). The USB controller 240 also provides USB connectivity to a variety of other USB-connected devices 242, such as a mouse, a removable non-volatile storage device 245, a modem, a network card, an ISDN connector, a fax machine, a printer, a USB hub, and many other types of USB-connected devices. Although the removable non-volatile storage device 245 is shown as a USB-connected device, it can be connected using different interfaces, such as FireWire.

[0021] Wireless Local Area Network (LAN) device 275 is connected to Southbridge 235 via PCI or PCI Express bus 272. LAN device 275 typically implements one of the IEEE 802.11 standards for over-the-air modulation techniques, all of which use the same protocol for wireless communication between information processing system 200 and another computer system or device. Optical storage device 290 is connected to Southbridge 235 via Serial ATA (SATA) bus 288. SATA adapters and devices communicate via a high-speed serial link. The SATA bus also connects Southbridge 235 to other forms of storage devices, such as hard drives. Audio circuitry 260 (e.g., a sound card) is connected to Southbridge 235 via bus 258. Audio circuitry 260 also provides functions such as audio line input and optical digital audio input ports 262, optical digital output and headphone jack 264, internal speakers 266, and internal microphone 268. Ethernet controller 270 is connected to Southbridge 235 via a bus such as PCI or PCI Express bus. Ethernet controller 270 connects information processing system 200 to computer networks, such as local area networks (LANs), the Internet, and other public and private computer networks. Although Figure 2 An information processing system is shown, but information processing systems can take many forms, some of which are... Figure 1 As shown in the diagram. For example, an information processing system can take the form of a desktop computer, server, portable, laptop, notebook, or other form factor computer or data processing system. Furthermore, an information processing system can take other form factors, such as a personal digital assistant (PDA), gaming device, ATM machine, portable telephone device, communication device, or other device including a processor and memory. Figure 3 This is a component diagram illustrating the different components and interactions used to perform performance monitoring of an information technology (IT) environment in the anomaly detection domain. Computer system 300 is being monitored for anomalies by an anomaly detection artificial intelligence (AI) system 320, which uses a trained anomaly detection model 330 to detect anomalies within system 300. The anomaly detection system 320 receives data from the monitored computer system via various system data 310 collected from the system. This system data includes system logs, performance metrics data (e.g., average load data, available memory data, CPU usage data, available disk space data, running process data, etc.). Anomalies detected by the anomaly detection AI system 320 are provided to the system administrator 340. The system administrator can then take action to resolve the anomalies to improve the performance of system 300.

[0022] The anomaly detection monitor 360 is an AI system that uses a trained supervised model 370 to determine whether the anomaly detection model 330 is operating within parameters. The anomaly detection supervisor receives inputs 350, including system data and the outputs of the anomaly detection model. System data inputs include system logs, metrics, topology information, etc. Outputs from the anomaly detection model, used as inputs to the supervised model 370, include the inference dataset, count vectors, and other outputs from the anomaly detection model.

[0023] After inputting 350 data points, the trained anomaly detection supervisor produces 380 results. These results may include recommendations for adjusting the anomaly detection model. These adjustments may include recommendations for retraining the anomaly detection model, recommendations for thresholds used to tune the anomaly detection model, and other adjustments to the anomaly detection model. The recommended changes are provided to the field reliability engineer (SRE) or other professionals responsible for maintaining the anomaly detection system. The SRE can then make recommended model adjustments to the anomaly detection model, such as retraining the model.

[0024] Figure 4 This is a flowchart illustrating the logic used by the anomaly detection supervisor. Figure 4 The process begins at 400 and illustrates the steps taken by the anomaly detection supervisor artificial intelligence (AI) system. In step 410, the process checks the state of the supervised machine learning model used by the anomaly detection supervisor. The process determines whether the supervised model needs to be trained, such as if this is the first time the anomaly detection supervisor is running on the system (decision 420).

[0025] If training of the supervised model is required, then decision 420 moves to the "yes" branch, and in the pre-defined process 430, supervised model training is executed (see details). Figure 5 (and the corresponding text). As shown in the figure, this results in training data being provided to the supervised model 370, so that the model is trained for use as an anomaly detection model in the computer system. On the other hand, if the supervised model has already been trained, decision 420 is made to the "No" branch to execute steps 440 to 470.

[0026] At the predetermined processing point 440, this processing uses a trained supervised model to execute an anomaly supervisor (see [link to processing details]). Figure 6(and corresponding text). As shown in the figure, the current anomaly detection model data is input to an anomaly detection supervisor 360 with a trained supervised model 370. Based on the training of the supervised model, the supervisor returns evaluation data to a predetermined process 440. The evaluation data is processed to form recommendations for adjusting and managing the anomaly detection model stored in the data storage 380. The process determines whether a set of ground truth data has been defined for the anomaly detection system (decision 450). If ground truth data has been defined, decision 450 branches to the 'yes' branch, and in step 460, the process analyzes the anomaly model using evaluation metrics with predefined ground truth data (e.g., ground truth data regarding precision, accuracy, F1 score, etc.). The determination made based on the ground truth data forms recommendations about the anomaly detection model and is stored in the data storage 380. On the other hand, if no ground truth data has been defined, decision 450 branches to the 'no' branch, bypassing step 460. In step 470, the process provides the recommendations stored in the data storage 380 to the site reliability engineer (SRE) 390.

[0027] The process determines whether to continue monitoring the anomaly detection model using the supervisor (Decision 480). If monitoring of the anomaly detection model continues, Decision 480 branches to the 'Yes' branch, and the loop returns to step 410 to repeat the above process. This loop continues until monitoring of the anomaly detection model stops, such as when the system is shut down, at which point Decision 480 branches to the 'No' branch, and the loop exits. Figure 4 The processing then ended at point 495.

[0028] Figure 5 This is a flowchart illustrating the logic used to train the supervised model. Figure 5 The process begins at 500 and illustrates the steps taken by the process of training the supervised model. In step 520, the process selects a first set of historical data from data store 540. The historical data used to train the supervised model includes data such as log templates, defined thresholds, inference datasets, count vectors, system metrics, system topology information, and other system data. In step 560, the process trains the anomaly detection supervisor model 370 using the selected set of historical data.

[0029] The process determines whether more historical data is available for training the supervised model (Decision 580). If more historical data is available for training the supervised model, then Decision 580 branches to the "Yes" branch, and the loop returns to step 520 to select the next set of historical data to train the model 370. This loop continues until all available historical data has been used to train the model, at which point Decision 580 branches to the "No" branch, and the loop exits. After this, Figure 5 The processing returns to the calling routine at 595 (see...). Figure 4).

[0030] Figure 6 This is a flowchart illustrating the logic used by a trained supervised model to monitor the performance of an anomaly detection model. Figure 6 The processing begins at 600 and illustrates the steps taken by the trained supervised model to process the current anomalous data to determine whether the anomaly detection model is performing appropriately. In step 610, the process provides system data (e.g., current system logs, metric data, topology information) and the current output from the anomaly detection model (inference dataset, count vectors, etc.) to an anomaly detection supervisor 360 with the trained supervised model. The anomaly detection supervisor 360 uses the provided data to perform various analyses (620-660) to produce results related to the performance of the anomaly detection model.

[0031] In analysis 620, the anomaly detection supervisor matches log templates and compares them to defined log template thresholds. In analysis 630, the anomaly detection supervisor compares the current inference dataset from the anomaly detection model with the training inference dataset used to train the supervisor model. At analysis 640, the anomaly detection supervisor compares the reference count vector within a given time frame from the anomaly detection model with the current count vector in the same time frame. In analysis 650, the anomaly detection supervisor examines the metric data for data drift. In analysis 660, the anomaly detection supervisor compares the current inference dataset with an older inference dataset.

[0032] The analysis performed by the anomaly detection supervisor forms a set of results, which are received and stored in data storage 680 at step 670. In step 690, the process generates a set of recommendations based on the anomaly detection supervisor results stored in data storage 680. For example, the recommendations could be for retraining the anomaly detection model due to data drift detected by the anomaly detection supervisor. Figure 6 The processing then returns to the calling routine at 695 (see...). Figure 4 ).

[0033] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0034] Computer-readable storage media can be a tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0035] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0036] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0037] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0038] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions comprises an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram. The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / actions specified in one or more blocks of a flowchart and / or block diagram. The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this context, each box in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the boxes may occur in a different order than those marked in the diagram. For example, depending on the functions involved, two consecutively shown blocks may actually execute substantially simultaneously, or these blocks may sometimes execute in reverse order. It should also be noted that each box in a block diagram and / or flowchart, and combinations of boxes in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0039] While specific embodiments have been shown and described, it will be apparent to those skilled in the art that changes and modifications can be made based on the teachings herein without departing from the invention and its broader aspects. Therefore, the appended claims are intended to cover within their scope all such changes and modifications that fall within the true spirit and scope of the invention. Furthermore, it should be understood that the invention is limited only by the appended claims. Those skilled in the art will understand that if a specific number of the introduced claim elements are intended, such an intention will be explicitly stated in the claims, and without such a statement, there is no such limitation. As a non-limiting example, to aid understanding, the following appended claims contain the use of the introductory phrases “at least one” and “one or more” to introduce claim elements. However, the use of such phrases should not be construed as implying that introducing a claim element by the indefinite article “a” or “an” limits any particular claim containing such an introduced claim element to an invention containing only one such element, even when the same claim includes the introductory phrases “one or more” or “at least one” and the indefinite article “a” or “an”; the same applies to the use of the definite article in the claims.

Claims

1. A computer-implemented method, implemented by an information processing system including a processor and a memory, the method comprising: Retrieve a set of current system data corresponding to the computer system and a set of current outputs from the anomaly detection model monitoring the computer system; The current system data and the current output from the anomaly detection model are input into the trained anomaly detection supervisor model; Receive a set of performance data corresponding to the anomaly detection model from the trained anomaly detection supervisor model; as well as In response to the set of performance data indicating that the performance of the anomaly detection model is below a threshold, the anomaly detection model is adjusted. The adjustment of the anomaly detection model is selected from the group consisting of: retraining the anomaly detection model, changing one or more metrics used by the anomaly detection model, and changing one or more thresholds used by the anomaly detection model.

2. The method according to claim 1, further comprising: Training an anomaly detection supervisor model, wherein the training produces the trained anomaly detection supervisor model, and wherein the training further includes: The training inference dataset is input into the anomaly detection supervisor model, wherein the training inference dataset is the output of the anomaly detection model that is monitoring the computer system.

3. The method according to claim 2, further comprising: Receive the current inferred dataset output from the anomaly detection model that is monitoring the computer system; as well as The trained anomaly detection supervisor model is used to compare the current inference dataset with one or more older inference datasets, one of which is the training inference dataset, and the comparison yields one of the performance data indicating whether the performance of the anomaly detection model is below the threshold.

4. The method according to claim 1, further comprising: Training an anomaly detection supervisor model, wherein the training produces the trained anomaly detection supervisor model, and wherein the training further includes: The training reference count vector corresponding to a predetermined time frame is input into the anomaly detection supervisor model, wherein the training reference count vector is the output of the anomaly detection model that is monitoring the computer system.

5. The method of claim 4, further comprising: Receive a reference count vector output from the anomaly detection model that is monitoring the computer system; as well as The trained anomaly detection supervisor model is used to compare the reference count vector with one or more older reference count vectors, one of which is the training reference count vector, and the comparison yields one of the performance data indicating whether the performance of the anomaly detection model is below the threshold.

6. The method of claim 1, further comprising: A set of current system metrics is input into the trained anomaly detection supervisor model, which compares the set of current system metrics with one or more sets of older system metrics, wherein the comparison leads to an analysis of data drift in the anomaly detection model, and wherein the analysis of data drift is one of the performance data indicating whether the performance of the anomaly detection model is below the threshold.

7. The method of claim 1, further comprising: A set of current system log templates is input into the trained anomaly detection supervisor model, which compares the set of current system log templates with one or more sets of older system log templates, wherein the comparison yields one of the performance data indicating whether the performance of the anomaly detection model is below the threshold.

8. An information processing system, comprising: One or more processors; A memory coupled to at least one of the processors; A set of computer program instructions, the set of computer program instructions being stored in the memory and executed by at least one of the processors to perform actions including: Retrieve a set of current system data corresponding to the computer system and a set of current outputs from the anomaly detection model monitoring the computer system; The current system data and the current output from the anomaly detection model are input into the trained anomaly detection supervisor model; Receive a set of performance data corresponding to the anomaly detection model from the trained anomaly detection supervisor model; as well as In response to the set of performance data indicating that the performance of the anomaly detection model is below a threshold, the anomaly detection model is adjusted. The adjustment of the anomaly detection model is selected from the group consisting of: retraining the anomaly detection model, changing one or more metrics used by the anomaly detection model, and changing one or more thresholds used by the anomaly detection model.

9. The information processing system according to claim 8, wherein the action further includes: Training an anomaly detection supervisor model, wherein the training produces the trained anomaly detection supervisor model, and wherein the training further includes: The training inference dataset is input into the anomaly detection supervisor model, wherein the training inference dataset is the output of the anomaly detection model that is monitoring the computer system.

10. The information processing system according to claim 9, wherein the action further includes: Receive the current inferred dataset output from the anomaly detection model that is monitoring the computer system; as well as The trained anomaly detection supervisor model is used to compare the current inference dataset with one or more older inference datasets, one of which is the training inference dataset, and the comparison yields one of the performance data indicating whether the performance of the anomaly detection model is below the threshold.

11. The information processing system according to claim 8, wherein the action further includes: Training an anomaly detection supervisor model, wherein the training produces the trained anomaly detection supervisor model, and wherein the training further includes: The training reference count vector corresponding to a predetermined time frame is input into the anomaly detection supervisor model, wherein the training reference count vector is the output of the anomaly detection model that is monitoring the computer system.

12. The information processing system according to claim 11, wherein the action further includes: Receive a reference count vector output from the anomaly detection model that is monitoring the computer system; as well as The trained anomaly detection supervisor model is used to compare the reference count vector with one or more older reference count vectors, one of which is the training reference count vector, and the comparison yields one of the performance data indicating whether the performance of the anomaly detection model is below the threshold.

13. The information processing system according to claim 8, wherein the action further includes: A set of current system metrics is input into the trained anomaly detection supervisor model, which compares the set of current system metrics with one or more sets of older system metrics, wherein the comparison leads to an analysis of data drift in the anomaly detection model, and wherein the analysis of data drift is one of the performance data indicating whether the performance of the anomaly detection model is below the threshold.

14. The information processing system according to claim 8, wherein the action further includes: A set of current system log templates is input into the trained anomaly detection supervisor module, which compares the set of current system log templates with one or more sets of older system log templates, wherein the comparison yields one of the performance data indicating whether the performance of the anomaly detection model is below the threshold.

15. A computer program product comprising computer program code that, when executed by an information processing system, performs the steps of any one of claims 1-7.

16. A computer-readable storage medium comprising computer program code, which, when executed by an information processing system, performs the steps of any one of claims 1-7.

Images