Massive execution of model training and scoring

Abstract

A method is presented that facilitates training a very large number of machine learning execution models for detecting anomalies in computing operations. The models are grouped together according to model type and assigned to different pods of a computing environment used to perform the monitored operations. Initial training of the models in a group is performed while monitoring resource usage, and based on the resource usage, a particular pod is selected for further training. The pod selected for training preferably has minimal resource usage variation before and after the initial training. A different pod can be selected to score the trained models. The pod selected for scoring preferably has the greatest resource usage among all containers during the initial scoring.

Description

Technical Field

[0001] This invention relates generally to computer systems, and more specifically to a method for training execution models to detect operational anomalies. Background Technology

[0002] As computational operations become more complex and the underlying infrastructure becomes more decentralized (such as in cloud computing), the ability to monitor such operations to optimize system performance becomes increasingly important. Numerous methods have been devised to automatically detect potential anomalies in the operation of large computing systems that may indicate serious operational problems. Some of these methods utilize different models of the system based on time-critical performance indicators.

[0003] This field is part of a larger technical area called Information Technology (IT) Operations Analytics, which attempts to discover complex patterns in large amounts of often noisy performance data. These analyses can include artificial intelligence for IT operations that relies on cognitive systems, known as AIOP. Cognitive systems (sometimes referred to as deep learning) are forms of artificial intelligence that use machine learning and problem-solving. While alternative designs such as Support Vector Machines (SVMs) or Bayesian networks can be used, cognitive systems typically utilize neural networks. A modern implementation of this artificial intelligence is Watson, sold by IBM. TM Cognitive technology.

[0004] Models used in anomaly detection can employ this cognitive system. The model attempts to capture the normal functioning of computational operations. If the current operational state deviates significantly from the model, a potential anomaly has been detected, and an alert can be generated for a supervisor or other automated solution. Different model types can be used in anomaly detection (such as simple statistical methods or challenges) or machine learning-based methods (such as density-based, clustering-based, SVM-based, Bayesian network-based), as well as custom detection models. Each model must be appropriately trained according to its model type (i.e., given a training dataset indicating the normal behavior of the system). Training can be unsupervised, supervised, or semi-supervised. Summary of the Invention

[0005] This invention, in at least one embodiment, generally targets a computer-implemented method comprising receiving details about a plurality of execution models to be used in anomaly detection, forming a group of the execution models, selecting a specific execution model from the group, training the specific execution model, and applying the training to the remaining execution models in the group to train a monitoring system for detecting anomalies in computational operations. In an illustrative embodiment, machine learning is used to train the execution models, and each execution model in the group has the same model type. The execution models may be implemented in corresponding containers within a compute pod, which provides shared storage, shared network resources, and a shared context for all containers in a given compute pod, and a specific compute pod is selected from the training, the specific compute container containing the training service for performing the training. The selection of the compute pod may include determining that it has the least change in resource usage among all compute pods containing the execution models in the group, compared to a first time period before initial training and a second time period after initial training. Once the execution model has been trained, the invention can be further implemented through additional scoring. This involves initiating initial scoring of the trained execution model in certain compute pods, monitoring the resource usage of those compute pods during the initial scoring period, selecting a specific compute pod (other than the initial compute pod) for further scoring based on the resource usage, and using a scoring service contained in that specific compute pod to complete the scoring of at least one execution model. The selection of this compute pod may include determining that it has the highest resource usage among all pods performing the initial scoring during the initial scoring period.

[0006] In the following detailed description, the above and other objectives, features and advantages of various embodiments of the present invention will become apparent. Attached Figure Description

[0007] The invention can be better understood by referring to the accompanying drawings, and many objects, features and advantages of its various embodiments will become apparent to those skilled in the art.

[0008] Figure 1 This is a block diagram of a computer system programmed to perform training of an execution model for detecting operational anomalies according to an embodiment of the present invention;

[0009] Figure 2 This is a graphical representation of a cloud computing environment according to an embodiment of the present invention;

[0010] Figure 3 This is a block diagram of a computing system with an application according to one embodiment of the present invention. In this example, the application is a database deployed via cloud computing, and the performance of the application will be modeled.

[0011] Figure 4 This is according to one embodiment of the present invention. Figure 3 A block diagram of the computing pod of the computing system, which shows the different models and training and scoring services;

[0012] Figure 5 It is a set of equations governing the selection of a particular computational pod for the purpose of training a model, according to one embodiment of the present invention;

[0013] Figure 6 This is a logic flowchart illustrating a model training process according to an embodiment of the present invention; and

[0014] Figure 7 This is a logic flowchart illustrating a model scoring process according to an embodiment of the present invention.

[0015] The same reference symbols are used in different figures to indicate similar or identical items. Detailed Implementation

[0016] When monitoring computational operations in large-scale applications (such as cloud-deployed databases), the ability to detect any kind of operational anomalies across many different metrics is crucial. A typical monitoring system would build an execution model for each metric to improve the accuracy of anomaly detection. However, in large computational operations, this could result in the need for hundreds of thousands, or even over a million, different models. For example, a database library with 2,000 related databases and 100 metrics per database would require 200,000 models to be able to detect anomalies for each metric in real time. This presents a significant challenge in model creation, as models must be trained individually based on different model types and associated metric data. Training a single anomaly detection model can be extensive, making training such a large number of models cumbersome. Once trained, they also need to be scored, which can be additionally computationally intensive at this scale.

[0017] Therefore, it is desirable to design an improved method for managing the creation and evaluation of a very large number of execution models. It would be even more advantageous if this method could allow the training and scoring of a very large number of models in a system with relatively limited resources. In different embodiments of the invention, these and other advantages are achieved by training models based on the number and type of models and resource usage over time, while simultaneously adjusting the compute infrastructure (pods) and available resources. Training can be balanced by allocating models to different pods. Models can be grouped according to type, and specific pods can be selected to train groups based on resource usage. Model scoring can also be based on the resource consumption of model scoring after the models have been packaged into different pods.

[0018] Now refer to the attached diagram, and specifically see... Figure 1 The illustration depicts an embodiment 10 of a computer system in which the present invention can be implemented to train an execution model for anomaly detection during large-scale computational operations. The computer system 10 is a symmetric multiprocessor (SMP) system having multiple processors 12a, 12b connected to a system bus 14. The system bus 14 is further connected to and communicates with a combined memory controller / host bridge (MC / HB) 16, which provides an interface to a system memory 18. The system memory 18 may be a local memory device, or alternatively may include multiple distributed memory devices, preferably dynamic random access memory (DRAM). Additional structures not depicted may be present in the memory hierarchy, such as onboard (L1) and second-level (L2) or third-level (L3) caches. One or more application or program modules according to the present invention are loaded into the system memory 18. In an exemplary embodiment, the application includes a database application with resource management tools, and the program module includes an execution model and training and scoring services.

[0019] The MC / HB16 also features interfaces to Peripheral Component Interconnect (PCIe) Fast Links 20a, 20b, and 20c. Each PCIe Fast Link 20a, 20b connects to a corresponding PCIe adapter 22a, 22b, and each PCIe adapter 22a, 22b connects to a corresponding Input / Output (I / O) device 24a, 24b. The MC / HB16 may additionally have an interface to an I / O bus 26, which connects to a switch (I / O architecture) 28. The switch 28 provides fan-out for the I / O bus to multiple PCIe Links 20d, 20e, 20f. These PCIe Links connect to more PCIe adapters 22c, 22d, 22e, which in turn support more I / O devices 24c, 24d, 24e. I / O devices may include, but are not limited to, keyboards, graphical pointing devices (mice), microphones, display devices, speakers, persistent storage devices (hard disk drives) or arrays of such storage devices, optical disc drives that receive optical discs 25 (an example of computer-readable storage media) such as CDs or DVDs, and network cards. Each PCIe adapter provides an interface between a PCI link and the corresponding I / O device. The MC / HB16 provides a low-latency path through which processors 12a, 12b can access PCI devices mapped anywhere within the bus memory or I / O address space. The MC / HB16 further provides a high-bandwidth path to allow PCI devices to access memory 18. Switch 28 provides peer-to-peer communication between different endpoints, and if the data traffic does not involve cache-coherent memory transfers, the data traffic does not need to be forwarded to the MC / HB16. Switch 28 is shown as a separate logical unit, but it can be integrated into the MC / HB16.

[0020] In this embodiment, PCI link 20c connects MC / HB16 to service processor interface 30 to allow communication between I / O device 24a and service processor 32. Service processor 32 is connected to processors 12a and 12b via JTAG interface 34 and uses line of interest 36 to interrupt the actions of processors 12a and 12b. Service processor 32 may have its own local memory 38 and is connected to read-only memory (ROM) 40 storing various program instructions for system startup. Service processor 32 can also access hardware operator panel 42 to provide system status and diagnostic information.

[0021] In alternative embodiments, computer system 10 may include modifications to these hardware components or their interconnections or additional components; therefore, the depicted examples should not be construed as implying any architectural limitations regarding the invention. The invention can also be implemented in an equivalent cloud computing network.

[0022] When computer system 10 is initially powered on, service processor 32 uses JTAG interface 34 to query system (host) processors 12a, 12b and MC / HB16. After completing the query, service processor 32 obtains the inventory and topology of computer system 10. Service processor 32 then performs various tests on the components of computer system 10, such as Built-in Self-Test (BIST), Basic Assurance Test (BAT), and memory tests. Service processor 32 reports any error messages about faults detected during testing to operator panel 42. If efficient configuration of system resources is still possible after removing any faulty components found during testing, computer system 10 is allowed to continue execution. Executable code is loaded into memory 18 and service processor 32 releases host processors 12a, 12b for executing program code, such as an operating system (OS), which is used to launch applications and, in particular, the model training and scoring program of the present invention, the results of which can be stored in the system's hard disk drive (I / O device 24). While the main processors 12a and 12b are executing program code, the service processor 32 can enter a mode that monitors and reports any operating parameters or errors, such as cooling fan speed and operation, thermal sensors, power regulators, and recoverable and unrecoverable errors reported by any of the processors 12a and 12b, memory 18, and MC / HB 16. The service processor 32 can take further action based on the type of error or defined thresholds.

[0023] This invention can be a system, method, and / or computer program product. A computer program product may include one or more computer-readable storage media having thereon computer-readable program instructions for causing a processor to perform aspects of the invention.

[0024] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0025] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0026] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0027] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0028] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0029] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0030] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0031] Computer system 10 executes program instructions for an operational monitoring process that uses novel computational techniques to manage the creation and evaluation of a very large number of performance models. Therefore, the program embodying the invention may additionally include conventional aspects of various performance modeling tools, and these details will become apparent to those skilled in the art upon reference to this disclosure. Training is crucial for the proper operation of performance models (particularly cognitive systems) and constitutes a technical field in itself. The present invention therefore represents a significant improvement in the field of cognitive system training technology.

[0032] In some embodiments, one or more aspects of the invention may be implemented using cloud computing. It should be understood that while this disclosure includes a detailed description of cloud computing, implementations of the teachings cited herein are not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0033] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model can include different features, service models, and deployment models.

[0034] Features may include, but are not limited to, on-demand service, broad network access, resource pooling, rapid elasticity, and measurement services. On-demand self-service refers to the ability of cloud consumers to unilaterally and automatically provide computing power (such as server time and network storage) on demand without requiring human interaction with the service provider. Broad network access refers to the ability available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs). Resource pooling occurs when a provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity means the ability to provide capacity quickly and elastically, automatically scaling down and up rapidly in some cases. For consumers, the capacity available for provisioning often appears unrestricted and can be purchased in any quantity at any time. Measuring services are the ability of cloud systems to automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the providers and consumers of the services being utilized.

[0035] Service models can include, but are not limited to, Software as a Service (SaaS), Platform as a Service (PAS), and Infrastructure as a Service (IaaS). Software as a Service (SaaS) refers to the ability provided to consumers to use applications from a provider running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with possible exceptions of limited user-specific application configuration settings. Platform as a Service (PaaS) refers to the ability provided to consumers to deploy applications created or acquired by consumers using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but have control over the deployed applications and, possibly, the application hosting environment configuration. Infrastructure as a Service (IaaS) refers to the ability provided to consumers to supply processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software (including operating systems and applications). Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0036] Deployment models can include, but are not limited to, private clouds, community clouds, public clouds, and hybrid clouds. A private cloud refers to cloud infrastructure that is solely for an organization's operations. It can be managed by the organization or a third party and can exist on-site or off-site. A community cloud has cloud infrastructure shared by several organizations and supports a specific community with shared concerns (e.g., missions, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-site or off-site. In a public cloud, the cloud infrastructure is available to the general public or a large industry group and is owned by the organization that sells cloud services. A hybrid cloud's cloud infrastructure is a combination of two or more clouds (private, community, or public) that maintain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0037] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. The core of cloud computing is its infrastructure, which includes a network of interconnected nodes. Figure 2 The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 52 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54a, desktop computers 54b, laptop computers 54c, and / or automotive computer systems 54d. Nodes 52 can communicate with each other. They may be physically or virtually grouped in one or more networks (not shown), such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 2 The types of computing devices 54a-54d shown are intended to be illustrative only, and computing node 52 and cloud computing environment 50 can (e.g., using a web browser) communicate with any type of computerized device via any type of network and / or network-addressable connection.

[0038] In the illustrative embodiments, certain aspects of the invention may be implemented by a cloud server or cloud computing system. A cloud computing system, for example, may include systems that communicate with clients via the Internet. Figure 2 Having similar Figure 1 The computer system 10 has an architecture or other architecture of nodes 52. The cloud computing system can host any number and type of applications. Figure 3 A cloud computing system 60 according to one embodiment of the present invention is shown. The cloud computing system 60 is deployed on platforms such as IBM Cloud. TM On the platform's cloud platform 62. IBM Cloud TMThe platform is a suite of cloud computing services from International Business Machines Corporation (IBM). It offers Platform as a Service (PaaS) and Infrastructure as a Service (IaaS). Further, for this example, the cloud platform 62 hosts database applications such as the Db2 database. Db2 is a family of data management products developed by IBM, including database servers. It was originally designed as a relational database management system but has been extended to support object-relational features and non-relational structures such as JSON and XML file formats.

[0039] In this implementation, database application 64 is embodied in a Kubernetes-type computing infrastructure, such as IBM Cloud. TM Kubernetes Service. This service is a managed provision that is built to create Kubernetes clusters for compute hosts on IBM Cloud. TM Kubernetes deploys and manages containerized applications. It defines a set of building blocks (primitives) that collectively provide mechanisms for deploying, maintaining, and scaling applications based on CPU, memory, or custom metrics. This service includes a master or controller (66) and multiple pods. A pod is the smallest deployable compute or scheduling unit that can be created and managed in Kubernetes. A pod is a group of one or more containers that share storage and network resources and specifications on how to run containers. The contents of a pod are always co-located and co-scheduled and run in a shared context. For a Db2 application, a pod may include a storage pod (67), a Db2 pod (68), and a model pod (70). The storage pod (67) contains the actual operand data as a subject for a specific database. The Db2 pod (68) handles database operations. The model pod (70) contains the execution model used to detect anomalies in Db2 database operations. Other pods, not shown, may exist. The controller (66) performs cluster resource management, such as increasing the number of pods as needed or deleting pods when they are no longer in use, and selecting pods for model training and scoring, as discussed further below. Controller 66 also provides a metrics collection service that measures the resource utilization of different pods or containers, such as CPU, memory, and I / O usage.

[0040] Further references are available. Figure 4 Understanding model training Figure 4 A model pod 70' according to an exemplary implementation is shown. Model pod 70' contains multiple models 72 (0 to N). This particular group of models are all of the same model type. A given model pod may be dedicated to only one group or may handle multiple model groups. Some models within a group are distributed across different pods to balance resource utilization, and Figure 4 It represents each of those pods.

[0041] Training service 74 is used to train various models 72. While training service 74 can reside in different pods, it is advantageously located within the same pod where the models are being trained. Multiple training services may exist for different pods or groups. Training service 74 performs a training process that first performs initial, limited training on all models 72 on different pods 70' using conventional training techniques. This initial training is limited because it involves a substantially smaller training dataset than reliable training is required. Following this initial training, as combined below... Figure 5 Further described, a single pod 70' is selected to complete the training. Once the optimal pod for training is selected, the given model in that pod is fully trained. This completed training is then applied to all models of that type, greatly simplifying the task of training a large number of models. Depending on the nature of the specific model involved, the completed training can be applied in different ways. For example, in models using neural network infrastructure, the completed training is embodied in the set of weights and biases of the neurons, and these parameters can be easily copied from the trained model and programmed into other models.

[0042] In a preferred embodiment, the pod for training is selected by taking into account resource usage over time. For example... Figure 4 As shown, for a given model i at time t, the model's CPU usage is denoted as C(i,t), the model's memory usage is denoted as C(i,t), and the model's I / O usage is denoted as I(i,t). Then, the Pod metric 80 can be calculated, as follows... Figure 5 As shown. The CPU usage S of a given pod. C (t) is calculated as Given the storage usage S of a pod M (t) is calculated as And given the pod's I / O usage S I (t) is calculated as Given The resource usage overview can then be represented as:

[0043] S(t) = w1S C (t)+w2S M (t)+w3S I (t),

[0044] Here, w1, w2, and w3 are weights set by the designer's preferences. The weights w1, w2, and w3 are typically determined by the model type and any resource constraints. For example, if most models require a lot of memory, w2 will be relatively large, while if the system lacks CPU power, w1 will be relatively large. The pod selected for training is the one with the smallest change in maximum resource utilization among all pods during the first time period before the start of new training and the second time period after the start of new training.

[0045] min pod (max t1 (S n (t1))-max t2 (S(t2))), (1)

[0046] Where, max t1 (S n (t1)) represents the maximum value at time t1 when training has started, and max t2 (S(t2)) represents the maximum value of time t2 before training begins. Equation (1) is further subject to the following constraint: S C (t), S M (t) and S I (t) must all be less than the corresponding maximum value based on resource availability.

[0047] See also Figure 6 The diagram further clarifies the training of the invention, illustrating a computer-implemented training process 90 according to one embodiment. Process 90 begins by receiving 92 details about the models to be used to detect anomalies caused by operations of the specific application involved. These details include the number and type of models and the metrics used for each model. The models are then grouped 94 according to type and distributed 96 among different pods to balance resource utilization. Limited training 98 is performed on all models in the pods. Container resource usage is calculated 100, and a pod is selected for further training 102 according to the above formula (1). Full training 104 is then completed for the models in this selected pod, and this training is applied 106 to the other models.

[0048] Once training is complete, it is necessary to score these models to evaluate their accuracy. The training process can therefore continue by selecting 108 individual pods for scoring purposes, further optimizing the computational efficiency of scoring a large number of performing models. The following section combines... Figure 7 The selection process is described further. After scoring, the models can be evaluated (110) to determine their accuracy, and processing (90) concludes the process. If a model scores poorly, further training can be performed.

[0049] In an exemplary implementation, a specific container within the pod is again selected to optimize the process, but this time it's used for scoring instead of training. In other words, the best pod for scoring can be different from the best pod for training. Figure 4 As shown, in some implementations, pod 70' may have a scoring service 76. The scoring service 76 may alternatively reside in a different pod, thus reducing the number of pods required after training. Figure 7 The scoring pod selection process 108, performed by scoring service 76, is illustrated. The scoring process 108 begins by balancing resource utilization through the allocation 120 of scoring requirements for pods. Initial scoring then begins 122 across all pods. As scoring progresses, resource usage of the scoring service is monitored 124. The pod with the highest resource usage is selected for continued scoring 126, as this pod is considered the most widely used for scoring across all scoring services. The trained model 128 can then be scored using the selected pods. This invention therefore provides a superior method for optimally regulating system resources to perform model training and scoring on a very large scale.

[0050] Although the invention has been described with reference to specific embodiments, this description is not intended to be limiting. Various modifications to the disclosed embodiments and alternative embodiments of the invention will become apparent to those skilled in the art upon reference to this description. Therefore, it is contemplated that such modifications may be made without departing from the scope of the invention as defined in the appended claims.

Claims

1. A computer-based method for training a monitoring system to detect anomalies in computational operations, comprising: Receive details about the multiple execution models that will be used in anomaly detection, including the multiple execution models, the types of execution models, and the metrics used for each execution model; A group is formed to form the execution models, wherein the group is a subset of the execution models that contains less than the total number of execution models; Selecting a specific execution model from the execution models in the group, wherein at least some of the execution models in the group are embodied in corresponding compute containers in a specific compute pod among multiple compute pods, and the selection of the specific compute pod includes: determining, among all compute pods containing the execution models in the group, that the specific compute pod has the least change in resource usage during a first time period before initial training compared to a second time period after initial training; Train the specific execution model in the specific computing pod; and The training is then applied to the remaining execution models in the group. The metric referred to here means measuring the resource utilization of different pods or containers.

2. The computer-implemented method according to claim 1, wherein, Machine learning is used to train the execution model in the group.

3. The computer-implemented method according to claim 1, wherein, Each of the execution models in the group has the same model type.

4. The computer-implemented method according to claim 1, wherein: The corresponding computing container provides shared storage, shared network resources, and a shared context for all containers within a given computing pod.

5. The computer-implemented method according to claim 1, further comprising: Start the initial scoring of the trained execution model in some compute pods; Monitor resource usage of certain compute pods during the initial scoring period; Based on the resource usage, a specific computing pod, other than the previously mentioned specific computing pod, is selected for further scoring; and Use the scoring service included in the specific compute pod to complete the scoring of at least one execution model.

6. The computer-implemented method according to claim 1, wherein, The selection of the specific compute pod includes determining that the specific compute pod has the highest resource utilization among all compute pods performing the initial scoring during the initial scoring period.

7. A computer system, comprising: A processor that processes one or more program instructions; A memory device connected to the one or more processors; as well as Program instructions residing in the memory device are configured to train a monitoring system for detecting anomalies in computational operations by: receiving details about multiple execution models to be used in anomaly detection, including the multiple execution models, the types of execution models, and metrics for each execution model; forming a group of the execution models, wherein the group is a subset of the execution models containing less than the total number of execution models; selecting a specific execution model from the execution models in the group, wherein at least some of the execution models in the group are embodied in corresponding compute containers in a specific compute pod among multiple compute pods; and the selection of the specific compute pod includes: determining, among all compute pods containing the execution models in the group, that the specific compute pod has the least change in resource usage during a first time period before initial training compared to a second time period after initial training; training the specific execution model in the specific compute pod; and applying the training to the remaining execution models in the group, wherein the metric represents a measurement of resource utilization of different pods or containers.

8. The computer system according to claim 7, wherein, Machine learning is used to train the execution model in the group.

9. The computer system according to claim 7, wherein, Each of the execution models in the group has the same model type.

10. The computer system according to claim 7, wherein: The corresponding computing container provides shared storage, shared network resources, and a shared context for all containers within a given computing pod.

11. The computer system according to claim 7, wherein, The program instructions further initiate the initial scoring of trained execution models in certain compute pods, monitor the resource usage of certain compute pods during the initial scoring, select a specific compute pod other than the specific compute pod for further scoring based on the resource usage, and complete the scoring of at least one execution model using a scoring service contained in the specific compute pod.

12. The computer system according to claim 7, wherein, The selection of the specific compute pod includes determining that the specific compute pod has the highest resource utilization among all compute pods performing the initial scoring during the initial scoring period.

13. A computer program product comprising: One or more computer-readable storage media; as well as Program instructions residing jointly in the one or more computer-readable storage media are configured to train a monitoring system for detecting anomalies in computational operations by: receiving details about multiple execution models to be used in anomaly detection, including multiple execution models, types of execution models, and metrics for each execution model; forming a group of the execution models, wherein the group is a subset of the execution models containing less than the total number of execution models; selecting a specific execution model from the execution models in the group, wherein at least some of the execution models in the group are embodied in corresponding compute containers in a specific compute pod among multiple compute pods; and the selection of the specific compute pod includes: determining, among all compute pods containing the execution models in the group, that the specific compute pod has the least change in resource usage during a first time period before initial training compared to a second time period after initial training; training the specific execution model in the specific compute pod; and applying the training to the remaining execution models in the group, wherein the metric represents a measurement of resource utilization of different pods or containers.

14. The computer program product according to claim 13, wherein, Machine learning is used to train the execution model in the group.

15. The computer program product according to claim 13, wherein, Each execution model in the group has the same model type.

16. The computer program product according to claim 13, wherein The corresponding computing container provides shared storage, shared network resources, and a shared context for all containers within a given computing pod.

17. The computer program product according to claim 13, wherein, The program instructions further initiate the initial scoring of trained execution models in certain compute pods, monitor the resource usage of certain compute pods during the initial scoring, select a specific compute pod other than the specific compute pod for further scoring based on the resource usage, and complete the scoring of at least one execution model using a scoring service contained in the specific compute pod.

Images