Method, system, and program for training a monitoring system for anomaly detection

Abstract

A method is presented for facilitating the training of a large number of machine learning performance models used to detect anomalies in computing operations. The models are grouped according to model type and assigned to different pods of a computing environment used to perform the monitored operations. An initial training of the models in a group is performed while monitoring resource utilization, and a particular pod is selected for further training based on the resource utilization. The pod selected for training preferably has the smallest change in resource utilization before and after the initial training. A different pod can be selected for scoring the trained models. The pod selected for scoring preferably has the largest resource utilization among all pods during the initial scoring.

Description

[Technical Field]

[0001] The present invention relates generally to computer systems, and more particularly to a method for training a performance model to detect behavioral anomalies. [Background technology]

[0002] As computing operations become more complex and the underlying infrastructure becomes less centralized, such as in cloud computing, being able to monitor such operations becomes increasingly important in order to optimize system performance. Many techniques have been devised to automatically detect potential anomalies in the functioning of large-scale computing systems that may indicate serious operational problems. Some of these techniques use various models for the system based on time-critical performance indicators.

[0003] This area is part of a broader technical field known as information technology (IT) operations analytics, which attempts to discover complex patterns in large volumes of, often noisy, performance data. These analytics can include artificial intelligence for IT operations, known as AIOP, which relies on cognitive systems. Cognitive systems (sometimes referred to as deep learning) are a form of artificial intelligence that uses machine learning and problem solving. Cognitive systems often utilize neural networks, but alternative designs such as support vector machines (SVMs) or Bayesian networks can be used. A modern implementation of artificial intelligence is the Watson™ cognitive technology sold by International Business Machines Corporation.

[0004] Models used for anomaly detection can employ such cognitive systems. The models attempt to capture the normal functioning of computing operations. If the current operating state deviates significantly from the model, an anomaly may be detected, and an alert can be generated for a supervisor or other automated solution. Various model types can be used for anomaly detection, including simple statistical methods or exercises, or machine learning-based techniques such as density-based, clustering-based, SVM-based, Bayesian networks, and custom detection models. Each model must be trained appropriately for its model type; that is, it must be given a training data set that represents the system's normal behavior. Training can be unsupervised, supervised, or semi-supervised. Summary of the Invention

[0005] In at least one embodiment, the present invention is generally directed to a computer-implemented method for training a monitoring system for detecting anomalies in computing operations by receiving details about performance models to be used in detecting anomalies, forming groups of performance models, selecting a particular one of the performance models in the group, training the particular performance model, and applying the training to the remaining performance models in the group. In an example implementation, the performance models are trained using machine learning, and each of the performance models in the group has the same model type. The performance models may be embodied in individual computing containers of a computing pod that provides shared storage, shared network resources, and a shared context for all containers in a given computing pod, and the particular computing pod is selected for training, and the particular computing pod includes a training service that performs the training. Selecting the computing pod may include determining that the pod has the smallest change in resource utilization among all computing pods in the group that include the performance model, comparing a first period before the initial training with a second period after the initial training. The present invention may be further implemented with additional scoring by initiating initial scoring of the trained performance model in some computing pods once the performance model has been trained, monitoring the resource utilization of these computing pods during the initial scoring, selecting a particular computing pod other than the computing pod used for training based on its resource utilization, and completing the scoring of the performance model using a scoring service contained in this particular computing pod.The selection of the computing pod may include determining that the pod has the highest resource utilization during the initial scoring among all computing pods performing the initial scoring.

[0006] These and additional objects, features, and advantages of various embodiments of the present invention will become apparent from the following detailed written description.

[0007] The present invention may be better understood, and its numerous objects, features and advantages of various embodiments made apparent to those skilled in the art by referencing the accompanying drawings. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a computer system programmed to train a performance model used to detect behavioral anomalies, according to one implementation of the present invention. [Figure 2] 1 is a pictorial representation of a cloud computing environment, according to one implementation of the present invention. [Figure 3] 1 is a block diagram of a computing system having an application, in this example a database, deployed via cloud computing whose performance is to be modeled, according to one implementation of the present invention. [Figure 4] FIG. 4 is a block diagram of a computing pod of the computing system of FIG. 3 showing various models and training and scoring services according to one implementation of the present invention. [Figure 5] 1 is a set of equations governing the selection of a particular computing pod for the purpose of training a model, according to one implementation of the present invention. [Figure 6] 1 is a chart illustrating the logic flow for a model training process according to one implementation of the present invention. [Figure 7]1 is a chart illustrating the logical flow for a model scoring process according to one implementation of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] The use of the same reference numbers in different drawings indicates similar or identical items.

[0010] When monitoring computing behavior in large-scale applications, such as cloud-deployed databases, it is important to be able to detect all kinds of behavioral anomalies across many metrics. Typical monitoring systems build performance models for each metric to improve anomaly detection accuracy. However, as a result, large computing operations can require hundreds of thousands, or even millions, of different models. For example, a database bank with 2,000 related databases and 100 metrics per database would require 200,000 models to be able to find anomalies per metric in real time. This creates a significant challenge when creating models, as they must be trained individually based on various model types and associated metric data. Training a single anomaly detection model can be extensive, making training such a large number of models prohibitive. Once trained, models require further scoring, which can be computationally intensive at this scale.

[0011] Therefore, it is desirable to devise an improved method for managing the creation and evaluation of large numbers of performance models. It would be further beneficial if the method enabled the training and scoring of large numbers of models in a system with relatively limited resources. These and other advantages are achieved in various implementations of the present invention by adjusting the computational infrastructure (pods) and available resources while training models based on the number and type of models and resource utilization over time. Training can be balanced by distribution to different pods. Models can be grouped by type, and particular pods may be selected to train certain groups based on resource utilization. Model scoring can also be based on the resource consumption of scoring the models after packing them into different pods.

[0012] Referring now to the drawings, and particularly to FIG. 1 , there is depicted one embodiment of a computer system 10 in which the present invention can be implemented to train performance models for anomaly detection in large-scale computing operations. The computer system 10 is a symmetric multiprocessor (SMP) system having multiple processors 12a, 12b connected to a system bus 14. The system bus 14 is further connected to and communicates with a combined memory controller / host bridge (MC / HB) 16, which interfaces with a system memory 18. The system memory 18 may be a local memory device or, alternatively, may include multiple distributed memory devices, preferably dynamic random access memory (DRAM). Additional memory structures, not depicted, exist in the memory hierarchy, such as an on-board (L1) cache and a second-level (L2) or third-level (L3) cache. The system memory 18 has one or more applications or program modules loaded therein in accordance with the present invention. In an example implementation, the application includes a database application with resource management tools, and the program modules include performance models and training and scoring services.

[0013] The MC / HB 16 also interfaces to peripheral component interconnect (PCI) Express links 20a, 20b, and 20c. Each PCI Express (PCIe) link 20a, 20b is connected to a respective PCIe adapter 22a, 22b, and each PCIe adapter 22a, 22b is connected to a respective input / output (I / O) device 24a, 24b. The MC / HB 16 additionally interfaces to an I / O bus 26, which is connected to a switch (I / O fabric) 28. The switch 28 provides a fanout for the I / O bus to multiple PCI links 20d, 20e, and 20f. These PCI links are connected to additional PCIe adapters 22c, 22d, and 22e, which support additional I / O devices 24c, 24d, and 24e. I / O devices may include, but are not limited to, a keyboard, a graphical pointing device (mouse), a microphone, a display device, speakers, a persistent storage device (hard disk drive) or an array of such storage devices, an optical disk drive that accepts an optical disk 25 (an example of a computer-readable storage medium) such as a CD or DVD, and a network card. Each PCIe adapter provides an interface between the PCI link and an individual I / O device. The MC / HB 16 provides a low-latency path through which processors 12a, 12b can access PCI devices mapped elsewhere within the bus memory or I / O address space. The MC / HB 16 also provides a high-bandwidth path, allowing PCI devices to access memory 18. The switch 28 facilitates peer-to-peer communication between different endpoints; this data traffic does not need to be forwarded to the MC / HB 16 if the MC / HB 16 is not involved in cache-coherent memory transfers. Although switch 28 is shown as a separate logic component, it may be integrated into MC / HB 16.

[0014] In this embodiment, PCI link 20c connects MC / HB 16 to a service processor interface 30, enabling communication between I / O device 24a and service processor 32. Service processor 32 is connected to processors 12a, 12b via a JTAG interface 34 and uses an attention line 36 to interrupt the operation of processors 12a, 12b. Service processor 32 may have its own local memory 38 and is connected to read-only memory (ROM) 40, which stores various program instructions for system startup. Service processor 32 may also have access to a hardware operator panel 42 to provide system status and diagnostic information.

[0015] In alternative embodiments, computer system 10 may include variations on or interconnections of these hardware components or additional components, and thus the depicted example should not be construed as implying any architectural limitations with respect to the present invention. The present invention may also be implemented on an equivalent cloud computing network.

[0016] When computer system 10 is first powered on, service processor 32 uses JTAG interface 34 to interrogate system (host) processors 12a, 12b and MC / HB 16. After the interrogation is complete, service processor 32 obtains an inventory and topology for computer system 10. Service processor 32 then runs various tests on the components of computer system 10, such as built-in self-tests (BISTs), basic assurance tests (BATs), and memory tests. Any error information for failures detected during the tests is reported by service processor 32 to operator panel 42. After removing any components found to be faulty during the tests, computer system 10 is allowed to continue if a valid configuration of system resources is still possible. The executable code is loaded into memory 18, and service processor 32 frees host processors 12a, 12b to execute program code, such as applications and an operating system (OS) used to launch the model training and scoring programs of the present invention, the results of which may be stored on the system's hard disk drive (I / O device 24). While host processors 12a, 12b execute the program code, service processor 32 may enter a mode in which it monitors and reports any operating parameters or errors, such as cooling fan speed and operation, thermal sensors, power regulators, and recoverable and unrecoverable errors reported by any of processors 12a, 12b, memory 18, and MC / HB 16. Service processor 32 may take further action based on the type of error or a defined threshold.

[0017] The present invention may be a system, a method, and / or a computer program product, which may include one or more computer-readable storage media having, collectively, computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0018] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or grooved structures having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as ephemeral signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cable), or electrical signals transmitted over electrical wires.

[0019] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a respective computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0020] The computer-readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages ​​such as Java®, Smalltalk®, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer as a standalone software package, partly on the user's computer and partly on a remote computer, or all on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions and individualize the electronic circuitry by utilizing state information of the computer readable program instructions to implement aspects of the present invention.

[0021] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0022] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium, capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for implementing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0023] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to produce a computer-implemented process and cause the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0024] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be performed as a single step, or may be executed simultaneously, substantially simultaneously, partially, or fully in a time-overlapping manner, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, are implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0025] The computer system 10 executes program instructions for a performance monitoring process that uses novel computational techniques to manage the creation and evaluation of numerous performance models. Accordingly, programs embodying the present invention may additionally include conventional aspects of various performance modeling tools, details of which will be apparent to those skilled in the art upon review of this disclosure. Training is critical to the proper operation of performance models, particularly cognitive systems, and constitutes a technical field in itself. Thus, the present invention represents a significant improvement in the technical field of cognitive system training.

[0026] In some embodiments, one or more aspects of the present invention may be performed using cloud computing. While this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0027] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model can include a variety of characteristics, service models, and deployment models.

[0028] Characteristics include, but are not limited to, on-demand service, broad network access, resource pooling, rapid scalability, and service metering. On-demand self-service refers to the cloud consumer's ability to unilaterally provision computing capabilities, such as server time and network storage, automatically as needed without requiring human interaction with the service provider. Broad network access refers to capabilities that are available on the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and personal digital assistants). Resource pooling occurs when a provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Location independence is implied in that consumers generally have no control or information about the exact location of the resources provided, although location (e.g., country, state, or data center) may be identified at a higher level of abstraction. Rapid scalability means that functionality can be quickly and elastically provisioned, quickly scaled out, and quickly released and quickly scaled in, sometimes automatically. To the consumer, the functionality available for provisioning often appears unlimited, and any amount can be purchased at any time. Service metering is the ability of a cloud system to automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource utilization can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0029] Service models include, but are not limited to, software as a service, platform as a service, and infrastructure as a service. Software as a Service (SaaS) refers to the ability offered to consumers to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface, such as a web browser. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, except for limited user-specific application configuration settings. Platform as a Service (PaaS) refers to the ability offered to consumers to deploy consumer-created or off-the-shelf applications, created using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but do have control over the deployed applications and, in some cases, the application hosting environment configuration. Infrastructure as a Service (IaaS) refers to the capability offered to consumers to provision processing, storage, network, and other fundamental computing resources on which they can deploy and run any software, which may include operating systems and applications.Consumers do not manage or control the underlying cloud infrastructure, but do have control over the operating system, storage, deployed applications, and in some cases limited control over select networking components (e.g., host firewalls).

[0030] Deployment models include, but are not limited to, private cloud, community cloud, public cloud, and hybrid cloud. A private cloud refers to cloud infrastructure operated exclusively for an organization. It is managed by that organization or a third party and can exist on- or off-premises. A community cloud has cloud infrastructure shared by several organizations to support a specific community with a shared mission (e.g., mission, security requirements, policies, and compliance considerations). It is managed by the organization or a third party and can exist on- or off-premises. In a public cloud, cloud infrastructure is made available to the general public or large industry groups and is owned by the organization that sells cloud services. The cloud infrastructure for a hybrid cloud combines two or more clouds (private, community, or public) that remain unique entities but are joined by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0031] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure comprising a network of interconnected nodes. An exemplary cloud computing environment 50 is depicted in FIG. 2. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 52 that can communicate with local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54a, a desktop computer 54b, a laptop computer 54c, or an automotive computer system 54d, or a combination thereof. The nodes 52 can communicate with each other. These can be grouped physically or virtually in one or more networks (not shown), such as private, community, public, or hybrid clouds, or a combination thereof, as described herein. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service, eliminating the need for cloud consumers to maintain resources on their local computing devices. It will be understood that the types of computing devices 54a-54d shown in FIG. 2 are intended to be exemplary only, and that computing node 52 and cloud computing environment 50 can communicate with any type of computerized device over any type of network and / or network-addressable connection (e.g., using a web browser).

[0032] In an exemplary implementation, some aspects of the present invention may be performed by a cloud server or cloud computing system. A cloud computing system may include, for example, node 52 of FIG. 2 having an architecture similar to computer system 10 of FIG. 1 or other architectures that communicate with clients via the Internet. A cloud computing system may host any number and type of applications. FIG. 3 illustrates a cloud computing system 60 deployed on a cloud platform 62, such as the IBM Cloud™ platform, according to one implementation of the present invention. The IBM Cloud™ platform is a suite of cloud computing services from International Business Machines Corporation (IBM®) that offers both platform-as-a-service (PaaS) and infrastructure-as-a-service (IaaS). Furthermore, in this example, cloud platform 62 hosts a database application, such as a Db2 database. Db2 is a family of data management products, including a database server, developed by IBM®. Db2 was originally designed as a relational database management system but has been extended to support object-relational features as well as non-relational structures such as JSON and XML file formats.

[0033] In this implementation, the database application 64 is embodied in a Kubernetes-type computing infrastructure, such as the IBM Cloud™ Kubernetes Service. This service is a managed offering built to create Kubernetes clusters of compute hosts on IBM Cloud™ to deploy and manage containerized apps. Kubernetes defines a set of building blocks (primitives) that together provide mechanisms for deploying, maintaining, and scaling applications based on CPU, memory, or custom metrics. A service includes a master or controller 66 and multiple pods. A pod is the smallest deployable unit of computing or scheduling that can be created and managed within Kubernetes. A pod is a group of one or more containers with shared storage and network resources and a specification for how the containers should run. The contents of a pod are always co-located and scheduled together, running in a shared context. In a Db2 application, a pod could include a storage pod 67, a Db2 pod 68, and a model pod 70. Storage pods 67 house the actual operand data that is the subject of a particular database. Db2 pods 68 handle database operations. Model pods 70 contain performance models used to detect anomalies in Db2 database operations. Other pods not shown may be present. Controller 66 performs resource management for the cluster, such as increasing the number of pods as needed or removing pods if no longer in use, and selecting pods for model training and scoring, as discussed further below. Controller 66 may also provide a metrics collection service that measures resource utilization, such as CPU, memory, and I / O utilization, for different pods or containers.

[0034] Model training can be understood with further reference to FIG. 4, which shows a model pod 70' according to an example implementation. The model pod 70' contains multiple models 72 (0-N). This particular group of models are all of the same model type. A given model pod can be dedicated to only one group or can handle multiple model groups. Some models in a group are assigned to different pods to balance resource utilization; FIG. 4 shows a representation of each such pod.

[0035] A training service 74 is used to train the various models 72. While the training services 74 can be located in different pods, it is advantageous to locate them within the same pod in which the models are trained. Multiple training services can exist in different pods or groups. The training service 74 first performs a training process using conventional training techniques to perform an initial, limited training of all of the models 72 for the different pods 70'. The initial training is limited in that it involves a significantly smaller training data set than required for reliable training. After this initial training, a single pod 70' is selected to complete the training, as described further below with respect to FIG. 5. Once the optimal pod for training has been selected, a given model within that pod undergoes full training. This completed training is then applied to all models of that type, greatly simplifying the task of training a large number of models. The completed training can be applied in a variety of ways, depending on the nature of the particular models involved. For example, in models that use neural network infrastructure, the completed training is embodied in a set of weights and biases for the neural nodes, and these parameters can be easily replicated from the trained model and programmed into other models.

[0036] In a preferred embodiment, the pods used for training are selected based on resource utilization over time. As shown in Figure 4, for a given model i, at time t, the model's CPU utilization is denoted as C(i,t), the model's memory utilization is denoted as M(i,t), and the model's I / O utilization is denoted as I(i,t). Pod metrics 80 can then be calculated, as can be seen in Figure 5. The CPU utilization S of a given pod is C (t) is

number

number

number

[0037] Training of the present invention can be further understood with reference to the chart in FIG. 6 , which illustrates a computer-implemented training process 90 according to one implementation. Process 90 begins by receiving 92 details about the models to be used in detecting anomalies arising from the behavior of the particular application involved. Such details include the number and type of models and the metrics used in each model. The models are then grouped according to type 94, and the groups are allocated among different pods 96 to balance resource utilization. Limited training of all models within the pod is performed 98. Pod resource utilization is calculated 100, and a pod is selected for further training 102 according to equation (1) above. Full training for the model is then completed 104 in the selected pod, and this training is applied to other models 106.

[0038] Once training is complete, the model needs to be scored to assess its accuracy. Therefore, the training process 90 can continue with the selection 108 of a single pod for scoring purposes, again optimizing computational efficiency in scoring what would otherwise be a large number of performance models. This selection process is further described below in connection with FIG. 7. After scoring, the model can be evaluated 110 to determine its accuracy, terminating the process 90. If the model does not score well, further training can be instituted.

[0039] In an example implementation, a particular one of the pods is again selected to optimize the process, but this time for scoring rather than training. In other words, the optimal pod for scoring may be different from the optimal pod for training. As can be seen from FIG. 4, pod 70′ may, in some implementations, have a scoring service 76. The scoring service 76 may alternatively be in a different pod, so the number of pods can be reduced after training. FIG. 7 illustrates the scoring pod selection process 108 performed by the scoring service 76. The scoring process 108 begins by assigning 120 scoring requirements to pods to balance resource utilization. Initial scoring then begins 122 for all pods. As scoring progresses, the resource utilization of the scoring service is monitored 124. The pod with the greatest resource utilization is selected for continued scoring 126 because it is considered the most expansive scoring of all of the scoring services. Scoring of the trained models can then be completed on the selected pod. 128 Thus, the present invention provides a superior approach for training and scoring a large number of performance models in a manner that optimally balances system resources.

[0040] While the present invention has been described with reference to particular implementations, this description is not intended to be construed in a limiting sense. Various modifications of the disclosed embodiments, as well as alternative embodiments of the invention, will become apparent to those skilled in the art upon reference to the description of the invention. It is therefore contemplated that such modifications may be made without departing from the scope of the invention as defined in the appended claims.

Claims

1. 1. A computer-implemented method for training a monitoring system for detection of anomalies in computing operations, comprising: receiving details regarding a plurality of performance models used in detecting the anomaly, the details including the number of the performance models, the types of the performance models, and resource utilization used by each of the performance models; forming a group of the performance models, the group being a subset of the performance models that is fewer in number than the total number of the performance models; selecting a particular one of the performance models in the group; at least some of the performance models in the group are embodied in individual computing containers in a particular one of a plurality of computing pods that provides shared storage, shared network resources, and a shared context for all containers in a given computing pod; the selecting includes selecting the particular computing pod for the training; the special computing pod includes a training service that performs the training; training the special performance model; Applying the trained model parameters obtained as a result of the training to the remaining performance models in the group.

11. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the performance models in the group are trained using machine learning.

3. The computer-implemented method of claim 1 , wherein each of the performance models in the group has the same model type.

4. 2. The computer-implemented method of claim 1, wherein the selection of the particular computing pod includes determining that the particular computing pod has the smallest change in resource utilization among all computing pods in the group that include a performance model when comparing a first period before an initial training and a second period after an initial training.

5. Initiating initial scoring of the trained performance model on some of the computing pods; monitoring the resource utilization of the number of computing pods during the initial scoring; selecting a particular computing pod other than the special computing pod for ongoing scoring based on the resource utilization; completing the scoring of at least one performance model using a scoring service contained in said particular computing pod; The computer-implemented method of claim 1 , further comprising:

6. 6. The computer-implemented method of claim 5, wherein the selection of the particular computing pod includes determining that the particular computing pod has the highest resource utilization during the initial scoring among all computing pods performing the initial scoring.

7. 1. A computer system comprising: one or more processors for processing program instructions; a memory device coupled to the one or more processors; receiving details regarding a plurality of performance models used in detecting anomalies, the details including a number of the performance models, a type of the performance models, and a resource utilization rate used by each of the performance models; forming a group of the performance models, the group being a subset of the performance models that is fewer than the total number of the performance models; and selecting a particular one of the performance models in the group; at least some of the performance models in the group are embodied in individual computing containers in a particular one of a plurality of computing pods that provides shared storage, shared network resources, and a shared context for all containers in a given computing pod; selecting includes selecting the particular computing pod for training; the special computing pod includes a training service that performs the training; program instructions resident in the memory device for training a monitoring system for detecting the anomalies in computing operations by training the particular performance model and applying parameters of the trained model resulting from the training to the remaining performance models in the group; A computer system comprising:

8. The computer system of claim 7 , wherein the performance models in the group are trained using machine learning.

9. 8. The computer system of claim 7, wherein each of the performance models in the group has the same model type.

10. 8. The computer system of claim 7, wherein the selection of the particular computing pod includes determining that the particular computing pod has the smallest change in resource utilization among all computing pods in the group that include a performance model when comparing a first period before an initial training and a second period after an initial training.

11. 8. The computer system of claim 7, wherein the program instructions further include: initiating initial scoring of trained performance models in a number of computing pods; monitoring the resource utilization of the number of computing pods during the initial scoring; selecting a particular computing pod other than the special computing pod based on the resource utilization for continued scoring; and completing scoring of at least one performance model using a scoring service included in the particular computing pod.

12. 12. The computer system of claim 11, wherein the selection of the particular computing pod includes determining that the particular computing pod has the highest resource utilization during the initial scoring among all computing pods performing the initial scoring.

13. A computer program causing a computer to execute the computer-implemented method according to any one of claims 1 to 6.

14. 14. A storage medium having the computer program of claim 13 stored on one or more computer-readable storage media.

Images