Automated runtime preemptive alert method and system

Abstract

An automated runtime preemptive alerting method and system are provided. For a given set of candidate factors, a classification model is trained to predict the computer problem probability, the accuracy score of the classification model is determined based on model validation, and the factor weights of the candidate factors in that set are adjusted based on the accuracy score. This process is performed for all sets of candidate factors. Low-accuracy classification models with accuracy scores below a threshold are selected. Higher-accuracy classification models with accuracy scores higher than the accuracy scores of the low-accuracy classification models are selected. The set of candidate factors used to train the low-accuracy classification models is updated using one or more of the candidate factors used to train the higher-accuracy classification models. The low-accuracy classification models are updated based on the updated set of candidate factors.

Description

Background Art

[0001] The present application relates generally to computers and computer applications, and more specifically to distributed systems, machine learning, and the provision of automatic proactive alerts during runtime of, for example, computer systems.

Summary of the Invention

[0002] The summary of the present disclosure is provided, for example, to facilitate understanding of computer systems and methods that automatically provide proactive (predictive) alerts for system issues during runtime, and is not intended to limit the present disclosure or the present invention. It should be understood that various aspects and features of the present disclosure can, in some instances, be used separately to advantage, or in other instances, be used in combination with other aspects and features of the present disclosure to advantage. Accordingly, variations and modifications may be made to the computer systems and / or their methods of operation to achieve different effects.

[0003] In some embodiments, a computer implementation method includes the step of selecting a set of candidate factors, wherein the candidate factors are indicators of computer performance, and each set of the set of candidate factors has a different combination of candidate factors than another set of the set of candidate factors. The method also includes the step of training a classification model for one of the sets of candidate factors to predict the likelihood of problems during computer runtime using a training dataset associated with the set, validating the classification model using a validation dataset associated with the set, determining an accuracy score associated with the classification model, and adjusting the factor weights associated with the candidate factors in the set based on the accuracy score. The training of the classification model, the validation of the classification model, and the adjustment of the factor weights are repeated for all of the sets of candidate factors, so that multiple classification models are trained corresponding to each of the sets of candidate factors. The method also includes the step of selecting low-accuracy classification models from the multiple classification models that have an accuracy score below a threshold criterion. The method also includes the step of selecting from the plurality of classification models a higher-accuracy classification model having a higher accuracy score than the accuracy score of the low-accuracy classification model. The method also includes the step of updating the set of candidate factors used to train the low-accuracy classification model by replacing at least one candidate factor in the set of candidate factors used to train the low-accuracy classification model with a candidate factor in the set of candidate factors used to train the higher-accuracy classification model. The method also includes the step of updating the low-accuracy classification model by repeating the training of the classification model, the validation of the classification model, and the adjustment of the factor weights using the updated set of candidate factors.

[0004] In some embodiments, the system includes at least one computer processor. The system also includes at least one memory device coupled with the computer processor. The at least one computer processor is configured to select a plurality of sets of candidate factors, the candidate factors being indicators of computer performance, and each set of the plurality of candidate factors having a different combination of candidate factors than another set of the plurality of candidate factors. The at least one computer processor is also configured to train a classification model for a certain set of the plurality of candidate factors using a training dataset associated with the set to predict the likelihood of problems during computer runtime, validate the classification model using a validation dataset associated with the set, determine an accuracy score associated with the classification model, and adjust the factor weights associated with the candidate factors in the set based on the accuracy score, wherein the computer processor is configured to repeat the training of the classification model, the validation of the classification model, and the adjustment of the factor weights for all sets of the plurality of candidate factors, wherein a plurality of classification models are trained corresponding to each of the plurality of candidate factors. The at least one computer processor is also configured to select a low-precision classification model having a precision score lower than a threshold criterion from the plurality of classification models. The at least one computer processor is also configured to select a higher-precision classification model having a precision score higher than the precision score of the low-precision classification model from the plurality of classification models. The at least one computer processor is also configured to update the set of candidate factors used to train the low-precision classification model by replacing at least one candidate factor in the set of candidate factors used to train the low-precision classification model with a candidate factor in the set of candidate factors used to train the higher-precision classification model.The at least one computer processor is also configured to update the low-accuracy classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated set of candidate factors.

[0005] A computer-readable storage medium may be provided for storing a program of machine-executable instructions for carrying out one or more of the methods described herein.

[0006] The following sections will describe in detail further features and the structure and operation of various embodiments, with reference to the attached drawings. In these drawings, the same reference numerals indicate the same or functionally similar elements. [Brief explanation of the drawing]

[0007] [Figure 1] An example of a computing environment in which an automated proactive alert for system problems during runtime can be implemented in one embodiment is shown.

[0008] [Figure 2-1] This figure shows an architecture for providing proactive alerts in one or more computer systems in several embodiments. [Figure 2-2] This figure shows an architecture for providing proactive alerts in one or more computer systems in several embodiments.

[0009] [Figure 3] Examples of time-series data associated with exemplary candidate factors in several embodiments are shown.

[0010] [Figure 4] The transformer architecture in several embodiments is shown.

[0011] [Figure 5]This figure shows classification models in several embodiments.

[0012] [Figure 6] This section describes the evaluation of factor weights using classification models in several embodiments.

[0013] [Figure 7] This section describes how to update candidate factors used in building classification models in several embodiments.

[0014] [Figure 8] This flowchart illustrates methods for predictive alerts and forecasts of potential computer system and / or application problems in several embodiments.

[0015] [Figure 9] This figure shows components of a system that can provide proactive alerts for system and / or application problems during runtime in several embodiments. [Modes for carrying out the invention]

[0016] In some embodiments, a computer implementation method includes the step of selecting a set of candidate factors, where the candidate factors are indicators of computer performance, and each set of candidate factors has a different combination of candidate factors than another set of candidate factors. The method also includes the step of training a classification model to predict the likelihood of problems during computer runtime using a training dataset associated with a particular set of candidate factors, validating the classification model using a validation dataset associated with a particular set, determining an accuracy score associated with the classification model, and adjusting the factor weights associated with the candidate factors in that set based on the accuracy score. Training the classification model, validating the classification model, and adjusting the factor weights are repeated for all sets of candidate factors, so that multiple classification models are trained corresponding to each set of candidate factors. The method also includes the step of selecting a low-accuracy classification model from multiple classification models that has an accuracy score below a threshold criterion. The method also includes the step of selecting a higher-accuracy classification model from multiple classification models that has an accuracy score higher than the accuracy score of the low-accuracy classification model. The method also includes a step of updating the set of candidate factors used to train a low-accuracy classification model by replacing at least one candidate factor in the set of candidate factors used to train a low-accuracy classification model with a candidate factor in the set of candidate factors used to train a higher-accuracy classification model. The method also includes a step of updating the low-accuracy classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated set of candidate factors.

[0017] In some embodiments, the method may be able to identify problems in distributed and / or multi-cloud environments, even when symptoms are buried in a flood of irrelevant data, vast amounts, speed, and overwhelming diversity. In some embodiments, the method may be able to proactively prevent potential underlying issues that could lead to high levels of symptoms. In some embodiments, the method can automatically provide proactive alerts for problems in computer systems and / or applications during runtime.

[0018] One or more of the following features may be separable from each other or optional. In some embodiments, the method also includes a step of training a transformer having an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in a given set of multiple sets of candidate factors. The training of the transformer is repeated for all sets of multiple sets of candidate factors, along with iterative training of the classification model, validation of the classification model, and adjustment of factor weights, so that multiple transformers are trained corresponding to each of the multiple sets of candidate factors. Updating the low-accuracy classification model includes updating the low-accuracy classification model by iteratively training the transformer, training of the classification model, validation of the classification model, and adjustment of factor weights using the updated set of candidate factors. For example, training the transformer makes it possible to use the transformer for predictions using new unknown data that may be used in the selection of candidate factors.

[0019] In some embodiments, the method further includes passing a new set of feature values ​​corresponding to an updated set of candidate factors to a transformer trained with the updated set of candidate factors to generate future values ​​corresponding to the updated set of candidate factors, and using the generated future values ​​as input to an updated low-accuracy classification model to predict future problem probabilities. For example, using a transformer for data prediction can improve the accuracy of the predictions.

[0020] In some embodiments, the method further comprises the step of repeating selection of a low-precision classification model, selection of a higher-precision classification model, update of a set of candidate factors used for training the low-precision classification model, and update of the low-precision classification model until all low-precision classification models among the plurality of classification models are updated. In this way, for example, all classification models can be improved to have an accuracy score exceeding a threshold criterion, for example.

[0021] In some embodiments, the candidate factors include factors having time-series data values that change over a certain period, and at least one factor having data values that remain static during the certain period. In this way, the classification model can handle various types of factors including, for example, time-series data and static data.

[0022] In some embodiments, the transformer is a multi-head transformer with self-attention and cross-attention, and the transformer generates future values based on self-attention of individual time-series data among the plurality of time-series data input to the transformer and cross-attention between the plurality of time-series data. In this way, for example, more accurate future values may be generated based on self-attention and cross-attention across the candidate factors in the set. For example, when generating future values for individual candidate factors, dependencies between the candidate factors can be taken into account.

[0023] In some embodiments, the replacement of at least one candidate factor in the set of candidate factors used to train the low-precision classification model comprises replacing the at least one candidate factor with the adjusted lowest factor weight, which is selected from the set of candidate factors used to train the low-precision classification model, with a candidate factor that has the adjusted highest factor weight selected from the set of candidate factors used to train the higher-precision classification and is not included in the set of candidate factors used to train the low-precision classification model. In this way, for example, one or more candidate factors that may contribute the least to the decision-making of the classification model can be replaced with one or more candidate factors that can contribute more to the decision-making of the classification model.

[0024] In some embodiments, adjusting the factor weights comprises adjusting the factor weights of the candidate factors in the certain set used in the classification model by multiplying the precision score and the factor weight of the candidate factor. In this way, the weight of each individual candidate factor used in the classification model can also be adjusted based on the overall performance precision of the classification model.

[0025] A system is also disclosed that includes at least one computer processor and at least one memory device coupled to the at least one computer processor, wherein the at least one computer processor is configured to perform one or more methods as described above. A computer program product comprising a computer-readable storage medium embodying program instructions is also disclosed, wherein the program instructions are readable by a device to cause the device to perform one or more methods as described above.

[0026] Various aspects of this disclosure are described by explanatory text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). With respect to any flowchart, operations may be performed in a different order than those shown in a given flowchart, depending on the technology involved. For example, again, depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or with at least partial time overlap.

[0027] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of one or more storage media (also called "mediums") that collectively comprise a set of one or more storage devices that collectively comprise machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. "Storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any preferred combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as pits / lands formed on the main surface of a punch card or disk), or any suitable combination of those described above. When the term "computer-readable storage medium" is used in this disclosure, it shall not be interpreted as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media.As those skilled in the art will understand, data is typically moved at several intermittent points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection; however, data is not transient while it is stored, and therefore the storage device is not transient.

[0028] The computing environment 100 includes an example of an environment for executing at least a portion of the computer code involved in carrying out the method of the present invention, such as the preemptive alert algorithm code 200. In addition to block 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including a processing circuit configuration 120 and a cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including an operating system 122 and block 200 as identified above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0029] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that is capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the execution of the computer implementation method may be distributed among multiple computers and / or across multiple locations. On the other hand, in this presentation concerning the computing environment 100, in order to keep the presentation as concise as possible, the detailed discussion focuses on a single computer, specifically computer 101. Computer 101 may be located in the cloud, although it is not shown in the cloud in Figure 1. On the other hand, computer 101 is not required to be located in the cloud, except to any extent that can be definitively shown.

[0030] The processor set 110 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuit configuration 120 may be distributed across multiple packages, for example, multiple coordinated integrated circuit chips. The processing circuit configuration 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels, depending on its relative proximity to the processing circuit configuration. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may operate using qubits and be designed to perform quantum computing.

[0031] Computer-readable program instructions typically cause the processor set 110 of computer 101 to execute a series of operational steps, thereby loading them onto computer 101 to implement a computer implementation method, and thereby the instructions thus executed instantiate the methods specified in the flowcharts and / or descriptions of the computer implementation methods contained herein (collectively referred to as the "Methods of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, e.g., a cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the Methods of the Invention. In the computing environment 100, at least some of the instructions for executing the Methods of the Invention may be stored in blocks 200 in persistent storage 113.

[0032] The communication fabric 111 is a signal conduction path that enables various components of the computer 101 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, physical input / output ports, and similar switches and conductive paths. Other types of signal communication paths, such as optical fiber communication paths and / or wireless communication paths, may be used.

[0033] The volatile memory 112 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, the volatile memory 112 is characterized by random access, but this is not required unless explicitly stated. In computer 101, the volatile memory 112 is located in a single package and resides inside computer 101, but alternatively or in addition, the volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0034] The persistent storage 113 is any form of non-volatile storage for a computer, currently known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained whether or not power is directly supplied to the computer 101 and / or the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 may take several forms, such as various known proprietary operating systems or open-source portable operating system interface (CSI) type operating systems employing a kernel. The code contained in block 200 typically includes at least a portion of computer code involved in performing the method of the present invention.

[0035] The peripheral device set 114 includes a set of peripheral devices for the computer 101. Data communication connections between the computer 101's peripheral devices and other components may be implemented in various ways, such as Bluetooth connections, Near-Field Communication (NFC) connections, connections made by cables (such as universal serial bus (USB) type cables), insert-type connections (e.g., secure digital (SD) cards), connections made via local area communication networks, and even connections made via wide area networks such as the internet. In various embodiments, the UI device set 123 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic devices. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing memory device for storing data in the form of qubits. In embodiments where computer 101 requires a large amount of storage (for example, when computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer, and another may be a motion detector.

[0036] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control and forwarding functions of the network module 115 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for performing the method of the present invention can typically be downloaded from an external computer or external storage device to computer 101 through a network adapter card or network interface included in the network module 115.

[0037] WAN102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any currently known or future-developed technology for transmitting computer data. In some embodiments, WAN102 may be replaced and / or supplemented by a local area network (LAN), such as a Wi-Fi network, designed to transmit data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0038] The end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101) and can take any of the forms discussed above in relation to computer 101. Typically, EUD 103 receives useful and valuable data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end-user, these recommendations would typically be communicated from the network module 115 of computer 101 to EUD 103 via WAN 102. In this way, EUD 103 can display, or otherwise present, the recommendations to the end-user. In some embodiments, EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0039] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 101. For example, in the hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0040] The public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computing power, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the host physical machine set 142, which is the universe of physical computers available in and / or to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs may be stored as images and are understood to be transferable either as images or after VCE instantiation, among and between various physical machine hosts. The cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instantiations, and manages active instantiations of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate over the WAN 102.

[0041] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature in the kernel that allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave like real computers in terms of the programs running in them. Computer programs running on a normal operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container; this feature is known as containerization.

[0042] The private cloud 106 is similar to the public cloud 105, except that its computing resources are available only for use by a single enterprise. Although the private cloud 106 is shown as communicating with the WAN 102, in other embodiments the private cloud may be completely isolated from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a separate discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configuration clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0043] In various embodiments, systems, methods, and / or techniques are disclosed that automatically provide proactive alerts for one or more problems in a computer system and / or application during runtime. In some embodiments, the systems, methods, and / or techniques randomly select and initialize time-series and static factors to monitor for individual systems across multiple systems. Factors are performance metrics that indicate the results of measurements of the performance or state of a computer system. In some embodiments, the systems, methods, and / or techniques analyze time-series data based on historical data by using a multi-head transformer employing self-attention and cross-attention. In some embodiments, the systems, methods, and / or techniques train a multi-head transformer model to generate one or more values ​​at future time. In some embodiments, the systems, methods, and / or techniques predict the likelihood of problems in a system by using a classification model for the current system. In some embodiments, the systems, methods, and / or techniques calculate weights for the current system and obtain adjusted weights for one particular factor in one particular model by comparing across systems. In some embodiments, the system, method, and / or technique selects candidate factors from relatively high-accuracy systems and updates candidate factors for relatively low-accuracy systems based on the ranking of factor weights. In some embodiments, the system, method, and / or technique iterates through training, prediction, and adjustment steps to improve the predictive accuracy of the classification model.

[0044] Figure 2 shows an architecture for providing proactive alerts in one or more computer systems in several embodiments. The components shown are components of a computer implementation. During initialization, for a system in a computing environment such as a distributed computer system environment, candidate factors are selected from various factors 202 (shown as candidate factors) as shown in 204. The selection in 204 may be performed in a random manner. For example, candidate factors may be randomly selected from a pool of candidate factors 202. The selection may be performed for each of multiple computer systems in the computing environment, e.g., 206, 208, 210. The selected candidate factors may include factors with time-series data 212, 214, 216 and static factors with static data 218, 220, 222. Candidate factors with time-series data have values ​​that change over time. Candidate factors with static data have values ​​that remain the same over time. In some embodiments, for example, since it may be costly for a single system to collect all performance factors, different factors or sets of factors may be selected for each system 204, 206, and 210.

[0045] Factors are performance or operational indicators of a computer system. Factors belonging to one or more performance or operational categories may be selected as candidates. For example, factors in the mutex (mutual exclusion) queue category may include semaphores, locks, condition variables, spinlocks, barriers, and / or others. Factors in the input / output (I / O) category may include file I / O, socket I / O, network I / O, disk I / O, and / or keyboard I / O. Factors in the malloc count category may include heap allocation, memory management, dynamic memory allocation, memory fragmentation, and / or memory leaks. Factors in the network connectivity category may include Transmission Control Protocol (TCP) connections, User Datagram Protocol (UDP) connections, socket connections, network protocols, and / or bandwidth. Factors in the signaling category may include interrupt signals, signal handlers, signal masks, signal transmission, and / or signal propagation. Factors in the critical section length category may include mutual exclusion, synchronization, deadlock thread synchronization, and / or concurrency control. Factors in the workload request frequency category may include request rate, throughput, response time, latency, and / or queue length. Factors in the central processing unit (CPU) category may include processor, cores, multithreading, instruction set, and / or cache hierarchy. Factors in the memory category may include random access memory (RAM), virtual memory, page faults, memory hierarchy, and / or memory management unit (MMU). Factors in the application trace category may include debugging, system hardware / software profiling, logging, performance monitoring, and / or tracing tools. Other factors or other factor categories may also be selected as candidate factors.

[0046] Factors with time-series data are passed through transformers that predict the next data in the time series. Transformers 224, 226, and 228 are trained using historical time-series data to predict future values ​​in the time series. For example, considering sys1 (system) in 206, the time-series data 212 associated with its candidate factor is input to transformer 224. The transformers have an encoder-decoder architecture. In some embodiments, multi-head self-attention and cross-attention may be used as encoders in training on multiple types of time-series data. For example, mutex wait queues, workload request frequency, and critical section length are different types of time-series data. For example, the mutex wait queue in the next time will be brought about by multiple time-series data. Transformer 224 predicts the factor value for the next time step or unit in the time series, as shown in 230.

[0047] Figure 3 shows examples of time series data associated with exemplary candidate factors in several embodiments. A sliding window of size k (where k may be predefined or pre-set), time series data 302, 304, and 306 are used as input to a transformer (e.g., 224 in Figure 2) to train the transformer to predict future time values ​​for each of the candidate factor time series data. Once trained, the transformer (e.g., 224 in Figure 2) accepts input containing time series data in a window of size k, such as [Q1, Q2, ..., Qk, T1, T2, ..., Tk, V1, V2, ..., Vk], where, for example, Q1, Q2, ..., Qk are time series data associated with candidate factors such as a mutex waiting queue, T1, T2, ..., Tk are time series data associated with candidate factors such as critical section length, and V1, V2, ..., Vk are time series data associated with candidate factors such as workload request frequency. Given these inputs, the transformer (e.g., 224 in Figure 2) predicts the following values ​​in the time series data: Qk+1, Tk+1, Vk+1, where k+1 is the next time step or unit.

[0048] Figure 4 shows the transformer architecture in several embodiments, for example, shown in Figure 2, p. 224. The transformer 406 is trained on the vectorized values ​​of input 402 or a set of sliding windows of size k, vector 404, where k is configurable or predefined. The transformer 406 uses self-attention encoding for each candidate factor time series and then cross-attention across all series. Different types of encoded time series data are concatenated for input to the fully connected layer 408. For example, the input to transformer 406 is a time series token sequence. For example, vectors Q, T, V 404 represent different types of time series data (shown as input 402). Each vector 404 is fed into self-attention, which captures the relationships between different tokens in the input sequence. In the decoder of transformer 406, the output vectors Q', T', V' are generated using cross-attention. For example, the decoder generates the target sequence one token at a time. To generate the next token, the decoder considers both the tokens generated so far and the relevant information from the input sequence. For example, the decoder generates the next tokens Qk+1, Tk+1, and Vk+1 at once. A fully connected layer 408, such as a softmax layer, outputs a prediction 410 of the next value for each candidate factor time series.

[0049] Referring back to Figure 2, the static candidate factor data 218 and the predicted time-series candidate factor values ​​230 are input to a classification model 236 that predicts the likelihood 242 of a problem occurring in the future. For example, the classification model is trained and used to predict possible system problems. The time-series prediction data 230 and other data 218 are used as input to the classification model 236. The classification model 236 outputs the problem likelihood. In some embodiments, the input data is preprocessed to ensure that the distribution of different data (e.g., time distribution or frequency) is uniform. The classification model 236 is trained with tagged training data to stabilize the model. For example, the model is considered stable if its predictions fall within a threshold of actual values ​​(e.g., ground truth data). Once the model is considered stable, weights (W) for one particular factor within a particular model are determined. The factor weights describe or indicate the importance of the features in the classification model 236, e.g., which features have the greatest impact on the model's predictions, or the importance of the features in the classification model's predictions. Known techniques can be used to identify or determine such weights. For example, existing software development tool libraries provide built-in methods for directly calculating feature importance scores from trained models, such as trained decision trees.

[0050] Figure 5 shows a classification model in several embodiments, for example, shown in 236 of Figure 2. Time-series forecast data 502 (e.g., mutex queue, workload requests, malloc count) and other data 504 (e.g., CPU, memory, signals) are used as input to the classification model 506. The classification model 506 outputs a problem probability. Exemplary results are shown in 510. For example, different inference runs of the classification model 506 with different input data may provide different results. Based on predicted future values ​​of the time-series data and other data (e.g., static data), the classification model 506 outputs a score or probability of the occurrence of a problem in a given computer system having conditions described or indicated by the input data. For example, a certain result may indicate that there is a 70% probability that the problem will occur. In some embodiments, the input data is preprocessed (508) to ensure that the distribution of different data (e.g., time distribution or frequency) is uniform. The classification model 506 is trained with tagged training data to stabilize the model. For example, a model is considered stable if its predictions fall within a threshold of actual values ​​(e.g., ground truth data). Examples of classification models 506 include decision tree models, random forest models, and / or others.

[0051] Referring back to Figure 2, the probability of a problem at a future time predicted by the classification model 236 242 is compared with actual known data, such as ground truth data, as shown in 248. For example, actual data or ground truth data may be obtained at that future time and compared with the predicted probability of a problem. In 254, factor weights are evaluated and ranked, as explained, for example, in relation to 626, 632, and 638 in Figure 6.

[0052] A similar process flow is performed for other computer systems, such as sys2 in 208 and sys3 in 210. For example, candidate factors are selected for system 208. Time series data 214 associated with the candidate factors are passed through a transformer 226, which is trained with historical time series data associated with these candidate factors, so that the transformer 226 predicts future values ​​(e.g., values ​​for the next time step or unit in the time series) (232), as described, for example, in relation to Figure 4. The static candidate factor data 220 and the predicted time series candidate factor values ​​232 are input to a classification model 238 that predicts problem probabilities 244, as described, for example, in relation to Figure 5. The problem probabilities 244 at future time predicted by the classification model 238 are compared with actual known data, such as ground truth data, as shown in 250. For example, actual data or ground truth data may be acquired at that future time and compared with the predicted problem probabilities. In 256, the factor weights are evaluated and ranked, for example, in relation to 628, 634, and 638 in Figure 6.

[0053] Similarly, candidate factors are selected for system 210. Time series data 216 associated with the candidate factors are passed through a transformer 228, which is trained on historical time series data associated with these candidate factors, so that the transformer 228 predicts future values ​​(e.g., values ​​for the next time step or unit in the time series) (234), as described, for example, in relation to Figure 4. The static candidate factor data 222 and the predicted time series candidate factor values ​​234 are input to a classification model 240 that predicts problem probabilities 246, as described, for example, in relation to Figure 5. The problem probabilities 246 at future time predicted by the classification model 240 are compared against actual known data, such as ground truth data, as shown in 252. For example, actual data or ground truth data may be acquired at that future time and compared against the predicted problem probabilities. In 258, factor weights are evaluated and ranked, as described, for example, in relation to 630, 636, and 638 in Figure 6.

[0054] Figure 6 shows the evaluation of factor weights using a classification model in several embodiments, for example, shown in 254 of Figure 2. Time series data 602 of candidate factors may be used. For example, for training, the candidate factors may be split into model training data 608 and model validation data 614. The model training data 608 may be used to train a classification model 620, for example, also described in relation to Figure 5. The model validation data 614 is used to predict problem probability values. For example, the model validation data 614 is input to the classification model 620, and the classification model 620 is run for inference. The classification model 620 outputs predictions of problem probability. The predicted problem probability values ​​and the actual problem probability (actual values ​​or ground truth values) are used as input to the loss function 626 to calculate the accuracy (A) of the classification model 620. The weights (W) of the candidate factors in this classification model 620 are also obtained. W and A are multiplied to obtain adjusted weights for one particular factor in one particular model, as shown in 632.

[0055] Similar processing is performed for other systems. For example, for another computer system, time-series data 604 of candidate factors selected for that computer system may be used. For example, for training, the candidate factors may be split into model training data 610 and model validation data 616. The model training data 610 may be used to train a classification model 622, as also described in relation to Figure 5. The model validation data 616 is used to predict problem probability values. For example, the model validation data 616 is input to the classification model 622, and the classification model 620 is run for inference. The classification model 622 outputs predictions of problem probability. The predicted problem probability values ​​and the actual problem probability (actual values ​​or ground truth values) are used as input to the loss function 628 to calculate the accuracy (A) of the classification model 622. The weights (W) of the candidate factors in this classification model 622 are also obtained. W and A are multiplied to obtain adjusted weights for one particular factor in one particular model, as shown in 634.

[0056] Similarly, for yet another computer system, time-series data 606 of candidate factors selected for that computer system may be used. For example, for training, the candidate factors may be split into model training data 612 and model validation data 618. Model training data 612 may be used to train a classification model 624, as also described in relation to Figure 5. Model validation data 618 is used to predict problem probability values. For example, model validation data 618 is input to classification model 624, which is then run for inference. Classification model 624 outputs predictions of problem probability. The predicted problem probability values ​​and actual problem probability (actual values ​​or ground truth values) are used as input to the loss function 630 to calculate the accuracy (A) of the classification model 620. The weights (W) of the candidate factors in this classification model 624 are also obtained. W and A are multiplied to obtain adjusted weights for one particular factor in one particular model, as shown in 636.

[0057] Adjusted weights 632, 634, and 636 from all models, in which a specific factor is included as an input factor, are combined to obtain the final weight 638 for this factor. The combination may be, for example, by taking a weighted average. For example, as shown in 638, F1, ..., F7 represent candidate factors and W1, ..., W7 represent the weights of candidate factors F1, ..., F7. For example, W1, ..., W7 shown in 638 is the weighted average of W1, ..., W7 shown in 632, 634, and 636. As an example, if three systems (system1, system2, system3) have a factor F1, the combined W1 may be W1 = (W1 of system1 × precision A of system1 + W1 of system2 × precision A of system2 + W1 of system3 × precision A of system3) / 3, where "×" represents multiplication, "+" represents addition, and " / " represents division.

[0058] Referring back to Figure 2, at 260, the factors are ordered according to their adjusted weights. For example, a unified factor ordering across all systems is performed based on the combined value (e.g., weighted mean) of the candidate factor weights (e.g., shown at 638 in Figure 6).

[0059] Factor weights and / or rankings of factor weights from each system, e.g., 254, 256, and 258, are used to update the factor selection in 262. For example, one or more classification models 236, 238, and 240 that may have low accuracy in predicting or classifying problem probabilities may be remodeled using different candidate factors. These different candidate factors may be selected based on the factor weights determined in 254, 256, and 258. Whether a classification model has low accuracy may be determined based on whether the classification meets a predefined threshold accuracy.

[0060] Figure 7 shows the updating of candidate factors used in building a classification model in several embodiments. For example, for a system with a low-accuracy classification model (such as one determined based on a threshold accuracy criterion), the classification model can be retrained using different or updated candidate factors. In some embodiments, one or more new candidate factors may be selected from systems with higher-accuracy models, for example, based on the rank order of factor weights (e.g., shown in Figure 2, 260). Given that the accuracy of system 1 in 702 is 0.7, this is considered low according to a predefined criterion. For example, according to the predefined criterion, an accuracy score or level less than 0.75 is considered low. The predefined criterion may be any other predefined criterion for determining whether the accuracy score or level is low. Also, consider system 2 in 704 to have a higher accuracy score or level, such as 0.9. One or more new candidate factors for system 1 702 are selected from system 2 704. For example, a candidate factor with the highest ranking weight in system2 704 and that is not currently used as a candidate factor in system1 702 may be selected by the factor selector 706. The selected candidate factor replaces the factor with the lowest weight in a low-precision system such as system1, as shown in 708. In some embodiments, more than one candidate factor may be replaced. For example, a predefined number of candidate factors may be replaced, or candidate factors with weights that meet a criterion (e.g., weights smaller than a predefined threshold weight, or weights relatively lower than other candidate factors) may be replaced.

[0061] Potential computer system problems can be detected early, for example, so that possible preventive measures or precautions can be taken before they occur. The occurrence of serious system problems in the future can be avoided by taking appropriate countermeasures to address potential issues. Alerts for potential computer system and / or application problems can be generated in a timely manner. This can save time and effort spent on root cause analysis after a problem has occurred.

[0062] In some embodiments, the methods described herein may be used directly as standalone tools or solutions used by computer operating systems, observability systems, and monitoring systems in both real-world production and development environments. The methods described herein may be used by all types of computer operating systems and runtime environments.

[0063] Figure 8 is a flowchart illustrating methods for predictive alerting and forecasting potential computer system problems, such as operating system and / or application problems during runtime, in several embodiments. The method may be implemented and / or run on one or more computer processors. For example, the method may predict computer system problems during runtime, such as process or thread hangs, deadlocks, and bottlenecks. Details of the method in various embodiments are also described in relation to Figures 2-7 above. In 802, several sets of candidate factors are selected. Candidate factors are indicators of computer performance. Examples of such factors are described above. Each set of the multiple sets of candidate factors has a different combination of candidate factors than another set of the multiple sets of candidate factors. In some embodiments, the candidate factors of a set may be randomly selected from a pool of computer system performance indicators.

[0064] For any set of candidate factors, the processes described in 804, 806, and 808 may be performed. These processes are also described in relation to Figure 6. In 804, the classification model is trained to predict the likelihood of problems occurring during computer runtime using the training dataset associated with the set. In 806, the classification model is validated using the validation dataset associated with the set, and an accuracy score associated with the classification model is determined. In 808, the factor weights associated with the candidate factors within the set are adjusted based on the accuracy score. Examples of classification models include, but are not limited to, decision trees, random forests, neural networks, and / or others.

[0065] Training, validating, and adjusting the factor weights of the classification model are repeated for all sets of candidate factors, and multiple classification models are trained to correspond to each set of candidate factors. For example, if there are more sets to process in 810, the method proceeds to 804, and the processes in 804, 806, and 808 are repeated for the next set of candidate factors. In some embodiments, different sets of candidate factors may be associated with different computer systems in a distributed system.

[0066] In 812, a low-precision classification model with an accuracy score lower than a threshold is selected from among several classification models. The threshold may be an absolute number, such as 0.7, within a score range, such as 0 to 10. Alternatively, the threshold may be a relative number, such as 50% lower than the highest score within a score range (e.g., 0 to 10). Other thresholds may be used or defined.

[0067] In 814, a higher-accuracy classification model having a higher accuracy score than the lower-accuracy classification model is selected from among multiple classification models. In some embodiments, the higher-accuracy classification model having the highest accuracy score among the multiple classification models is selected.

[0068] In step 816, the set of candidate factors used to train the low-accuracy classification model is updated by replacing at least one candidate factor in the set of candidate factors used to train the low-accuracy classification model with a candidate factor in the set of candidate factors used to train the higher-accuracy classification model. In step 818, the low-accuracy classification model is updated by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated set of candidate factors.

[0069] In some embodiments, the method may be able to identify problems in distributed and / or multi-cloud environments, even when symptoms are buried in a flood of irrelevant data, vast amounts, speed, and overwhelming diversity. In some embodiments, the method may be able to proactively prevent potential underlying issues that could lead to high levels of symptoms. In some embodiments, the method can automatically provide proactive alerts for problems in computer systems and / or applications during runtime.

[0070] In some embodiments, the method also includes a step of training a transformer having an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in a given set of multiple sets of candidate factors, as described, for example, in reference to Figure 4 and further in reference to Figure 2. The training of the transformer is repeated for all sets of multiple sets of candidate factors, along with iterative training of the classification model, validation of the classification model, and adjustment of factor weights, so that multiple transformers are trained corresponding to each of the multiple sets of candidate factors. Updating the low-accuracy classification model includes updating the low-accuracy classification model by iteratively training the transformer, training of the classification model, validation of the classification model, and adjustment of factor weights using the updated set of candidate factors.

[0071] In some embodiments, the method further includes passing a new set of feature values ​​corresponding to an updated set of candidate factors to a transformer trained using the updated set of candidate factors to generate future values ​​corresponding to the updated set of candidate factors, and using the generated future values ​​as input to an updated low-precision classification model to predict future problem probabilities.

[0072] In some embodiments, the method further includes the steps of selecting a low-accuracy classification model, selecting a higher-accuracy classification model, updating the set of candidate factors used to train the low-accuracy classification model, and updating the low-accuracy classification model, repeated until all of the low-accuracy classification models among the multiple classification models have been updated. In this way, for example, all classification models can be improved to have an accuracy score above a threshold criterion, for example.

[0073] In some embodiments, candidate factors include factors having time-series data values ​​that change over a certain period, and at least one factor having data values ​​that remain static over that period. In this way, the classification model can handle various types of factors, including, for example, time-series data and static data.

[0074] In some embodiments, the transformer is a multi-head transformer with self-attention and cross-attention, and the transformer generates future values ​​based on the self-attention of individual time series data and the cross-attention between multiple time series data input to the transformer. In this way, for example, more accurate future values ​​may be generated based on self-attention and cross-attention across candidate factors in a set. For example, when generating future values ​​for individual candidate factors, dependencies between candidate factors may be taken into consideration.

[0075] In some embodiments, the substitution of at least one candidate factor in a set of candidate factors used to train a low-accuracy classification model includes substituting at least one candidate factor with the lowest adjusted factor weight, selected from the set of candidate factors used to train a low-accuracy classification model, with a candidate factor with the highest adjusted factor weight, selected from the set of candidate factors used to train a higher-accuracy classification model, and not included in the set of candidate factors used to train a low-accuracy classification model. In this way, for example, one or more candidate factors that may contribute least to the classification model's decision-making may be replaced with one or more candidate factors that may contribute more to the classification model's decision-making.

[0076] In some embodiments, factor weight adjustment includes adjusting the factor weights of candidate factors within a given set used in the classification model by multiplying the accuracy score by the factor weight of the candidate factor. Thus, the weights of individual candidate factors used in the classification model may also be adjusted based on the overall performance accuracy of the classification model.

[0077] In some embodiments, the computer implementation method includes a step of selecting a set of candidate factors, which are indicators of computer performance. Each set of candidate factors has a different combination of candidate factors than another set of candidate factors. For example, the candidate factors of a set may be randomly selected from a pool of computer system performance indicators. For a given set of candidate factors, it may be possible to train a transformer having an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in that set; train a classification model to predict the likelihood of problems during computer runtime using the training dataset associated with that set; validate the classification model using the validation dataset associated with that set to determine the accuracy score associated with the classification model; and adjust the factor weights associated with the candidate factors in that set based on the accuracy score. The training of the transformer, training of the classification model, validation of the classification model, and adjustment of factor weights are repeated for all sets of candidate factors. Thus, multiple transformers and multiple classification models are trained corresponding to each of the multiple sets of candidate factors.

[0078] The method also includes a step of selecting a low-accuracy classification model from among several classification models that has an accuracy score lower than a threshold criterion. The threshold criterion may be an absolute number, such as 0.7, within a score range such as 0 to 10. Alternatively, the threshold criterion may be a relative number, such as a value 50% lower than the highest score within a score range (e.g., 0 to 10). Other threshold criteria may be used or defined.

[0079] The method also includes the step of selecting a higher-accuracy classification model from among several classification models that has a higher accuracy score than the accuracy score of a low-accuracy classification model. In some embodiments, the classification model with the highest accuracy score may be selected.

[0080] The method also includes a step of updating the set of candidate factors used to train a low-accuracy classification model by replacing at least one candidate factor in the set of candidate factors used to train a low-accuracy classification model with a candidate factor in the set of candidate factors used to train a higher-accuracy classification model.

[0081] The method also includes a step to update low-accuracy classification models by repeatedly training the transformer, training the classification model, validating the classification model, and adjusting the factor weights using an updated set of candidate factors.

[0082] A computer program product may be provided, comprising a computer-readable storage medium in which program instructions are embodied, wherein the program instructions are readable by a device, causing the device to execute one or more of the methods described herein. A system may be provided comprising at least one computer processor and at least one memory device coupled to the computer processor, wherein the at least one computer processor is configured to perform one or more of the methods described herein.

[0083] Figure 9 shows components of a system that can provide proactive alerts for system and / or application problems during runtime in several embodiments. One or more hardware processors 902, such as a central processing unit (CPU), a graphics processing unit (GPU), and / or a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or another processor, may be coupled with a memory device 904. The memory device 904 may include random access memory (RAM), read-only memory (ROM), or another memory device, and may store data and / or processor instructions for implementing various functions associated with the method and / or system described herein. One or more processors 902 may execute computer instructions stored in the memory 904 or received from another computer device or medium. The memory device 904 may store instructions and / or data for the functions of one or more hardware processors 902, and may include an operating system and other programs of instructions and / or data. One or more hardware processors 902 may select multiple sets of candidate factors. For a given set of candidate factors, one or more hardware processors 902 may train a classification model using the training dataset associated with that set to predict the likelihood of a problem during computer runtime, validate the classification model using the validation dataset associated with that set, determine the accuracy score associated with the classification model, and adjust the factor weights associated with the candidate factors in that set based on the accuracy score. One or more hardware processors 902 may repeat the training of the classification model, validation of the classification model, and adjustment of factor weights for all sets of candidate factors. In this way, multiple classification models are trained, each corresponding to a set of candidate factors.One or more hardware processors 902 may select a low-precision classification model from among several classification models that has a precision score lower than a threshold criterion. One or more hardware processors 902 may select a higher-precision classification model from among several classification models that has a precision score higher than the precision score of the low-precision classification model. One or more hardware processors 902 may update the set of candidate factors used to train the low-precision classification model by replacing at least one candidate factor in the set of candidate factors used to train the low-precision classification model with a candidate factor in the set of candidate factors used to train the higher-precision classification model. One or more hardware processors 902 may update the low-precision classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated set of candidate factors. In one embodiment, candidate factor data, such as training data and validation data, may be stored in a memory device 906 or received from a remote device via a network interface 908 and temporarily loaded into a memory device 904 to generate a classification model. The learned classification model may be stored on a memory device 904 for execution by, for example, one or more hardware processors 902. One or more hardware processors 902 may be coupled with interface devices, for example, a network interface 908 for communicating with a remote system over a network, and an input / output interface 910 for communicating with input and / or output devices such as a keyboard, mouse, display, and / or other.

[0084] In some embodiments, systems and methods may be able to identify problems in distributed and / or multi-cloud environments, even when symptoms are buried in a flood of irrelevant data, vast amounts, speed, and overwhelming diversity. In some embodiments, systems and methods may be able to proactively prevent potential underlying issues that could lead to high levels of symptoms. For example, in some embodiments, systems and methods may be able to automatically provide proactive alerts for problems in computer systems and / or applications during runtime.

[0085] The terms used herein are intended to describe only specific embodiments and are not intended to limit the invention. Where used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless the context explicitly indicates otherwise. Where used herein, the term “or” is an inclusive operator and may mean “and / or” unless the context explicitly indicates otherwise. Where used herein, the terms “comprise,” “comprises,” “comprising,” “include,” “includes,” “including,” and / or “having” may specify the presence of a described feature, integer, step, operation, element, and / or component, but not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Where used herein, the phrases “in one embodiment” or “in several embodiments” may, but may not, refer to the same embodiment. As used herein, the phrase "in one embodiment" may, but not necessarily, refer to the same embodiment. As used herein, the phrase "in another embodiment" may, but not necessarily, refer to a different embodiment. Furthermore, embodiments and / or components of embodiments may be freely combined with one another, provided they are not mutually exclusive.

[0086] All means-plus-function elements or step-plus-function elements (if any) in the following claims are intended to include any structures, materials, or actions for performing a function in combination with other claimed elements, as specifically claimed. Although the description of the present invention is expressed for illustrative and explanatory purposes, it is not intended to be exhaustive or limited to the present invention in the disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the present invention. These embodiments have been selected and described to best illustrate the principles and practical applications of the present invention and to enable other those skilled in the art to understand the invention in terms of various embodiments with various modifications suitable for the particular use intended.

Claims

1. In the step of selecting multiple sets of candidate factors, the candidate factors are indicators of computer performance, and each set of the multiple sets of candidate factors has a different combination of candidate factors than another set of the multiple sets of candidate factors; For a certain set of the multiple sets of candidate factors, A classification model is trained to predict the likelihood of problems during computer runtime using a training dataset associated with the aforementioned set. The classification model is validated using the validation dataset associated with the aforementioned set, and the accuracy score associated with the classification model is determined, and Based on the accuracy score, adjust the factor weights associated with the candidate factors within a given set. In this stage, the training of the classification model, the validation of the classification model, and the adjustment of the factor weights are repeated for all sets of the multiple sets of candidate factors, so that multiple classification models are trained corresponding to each of the multiple sets of candidate factors; A step of selecting a low-accuracy classification model from the plurality of classification models that has an accuracy score lower than a threshold criterion; A step of selecting a higher-accuracy classification model from the plurality of classification models that has a higher accuracy score than the accuracy score of the low-accuracy classification model; A step of updating the set of candidate factors used to train the low-accuracy classification model by replacing at least one candidate factor in the set of candidate factors used to train the low-accuracy classification model with a candidate factor in the set of candidate factors used to train the higher-accuracy classification model; and A step to update the low-accuracy classification model by repeating the training of the classification model, the validation of the classification model, and the adjustment of the factor weights using the updated set of candidate factors. A computer implementation method comprising the following:

2. With respect to one of the sets of candidate factors, A transformer having an encoder and decoder architecture is trained to generate future values ​​for at least some of the candidate factors within a certain set. At this stage, The training of the transformer is repeated for all sets of the multiple sets of candidate factors, along with the training of the classification model, the validation of the classification model, and the adjustment of the factor weights, so that multiple transformers are trained corresponding to each of the multiple sets of candidate factors; and The update of the low-accuracy classification model includes updating the low-accuracy classification model by repeatedly performing the training of the transformer, the training of the classification model, the validation of the classification model, and the adjustment of the factor weights using the updated set of candidate factors. The computer implementation method according to claim 1, further comprising the following:

3. The computer implementation method according to claim 2, further comprising the steps of passing a new set of feature values ​​corresponding to the updated set of candidate factors to the transformer, which is trained using the updated set of candidate factors, to generate future values ​​corresponding to the updated set of candidate factors, and using the generated future values ​​as input to the updated low-precision classification model to predict future problem probabilities.

4. The computer implementation method according to claim 2 or 3, further comprising the steps of selecting the low-precision classification model, selecting the higher-precision classification model, updating the set of candidate factors used to train the low-precision classification model, and repeating the updating of the low-precision classification model until all of the low-precision classification models among the plurality of classification models have been updated.

5. The computer implementation method according to any one of claims 2 to 4, wherein the candidate factors include factors having time-series data values ​​that change during a certain period, and at least one factor having data values ​​that remain static during the certain period.

6. The computer implementation method according to any one of claims 2 to 5, wherein the transformer is a multi-head transformer having self-attention and cross-attention, and the transformer generates the future value based on the self-attention of individual time-series data and the cross-attention between the plurality of time-series data input to the transformer.

7. The substitution of at least one candidate factor in the set of candidate factors used to train the low-accuracy classification model is: Replacing at least one candidate factor having the lowest adjusted factor weight, selected from the set of candidate factors used to train the low-accuracy classification model, with a candidate factor having the highest adjusted factor weight, selected from the set of candidate factors used to train the higher-accuracy classification model, and not included in the set of candidate factors used to train the low-accuracy classification model. A computer implementation method according to any of the preceding claims, including the method described above.

8. The computer implementation method according to any of the preceding claims, wherein the adjustment of the factor weights includes adjusting the factor weights of the candidate factors in a certain set used in the classification model by multiplying the accuracy score by the factor weights of the candidate factors.

9. A computer program product comprising a computer-readable storage medium in which program instructions are embodied, wherein the program instructions are readable by the device, and the device Selecting multiple sets of candidate factors, wherein the candidate factors are indicators of computer performance, and each set of the multiple sets of candidate factors has a different combination of candidate factors than another set of the multiple sets of candidate factors; For a certain set of the multiple sets of candidate factors, A classification model is trained to predict the likelihood of problems during computer runtime using a training dataset associated with the aforementioned set. The classification model is validated using the validation dataset associated with the aforementioned set, and the accuracy score associated with the classification model is determined, and Based on the accuracy score, adjust the factor weights associated with the candidate factors within a given set. The device causes the training of the classification model, the validation of the classification model, and the adjustment of the factor weights to be repeated for all sets of the multiple sets of candidate factors, where multiple classification models are trained corresponding to each of the multiple sets of candidate factors; Selecting a low-accuracy classification model from the multiple classification models that has an accuracy score lower than a threshold criterion; Selecting a higher-accuracy classification model from the plurality of classification models that has a higher accuracy score than the accuracy score of the low-accuracy classification model; Updating the set of candidate factors used to train the low-accuracy classification model by replacing at least one candidate factor in the set of candidate factors used to train the low-accuracy classification model with a candidate factor in the set of candidate factors used to train the higher-accuracy classification model; and Updating the low-accuracy classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated set of candidate factors. A computer program product that performs a certain action.

10. The aforementioned device further, With respect to one of the sets of candidate factors, A transformer having an encoder and decoder architecture is trained to generate future values ​​for at least some of the candidate factors within a certain set. Here, The device is made to repeat the training of the transformers for all sets of the multiple sets of candidate factors, along with the repeated training of the classification model, the validation of the classification model, and the adjustment of the factor weights, such that multiple transformers are trained corresponding to each of the multiple sets of candidate factors; and The device being made to update the low-accuracy classification model includes the device being made to update the low-accuracy classification model by repeatedly performing the training of the transformer, the training of the classification model, the validation of the classification model, and the adjustment of the factor weights using the updated set of candidate factors. A computer program product according to claim 9 that can be made to perform the following actions.

11. The computer program product according to claim 10, wherein the device further passes a new set of feature values ​​corresponding to the updated set of candidate factors to the transformer, which is trained using the updated set of candidate factors, to generate future values ​​corresponding to the updated set of candidate factors, and uses the generated future values ​​as input to the updated low-precision classification model to predict future problem probabilities.

12. The computer program product according to claim 10 or 11, wherein the device is further configured to repeat the selection of the low-precision classification model, the selection of the higher-precision classification model, the updating of the set of candidate factors used to train the low-precision classification model, and the updating of the low-precision classification model until all of the low-precision classification models among the plurality of classification models have been updated.

13. The computer program product according to any one of claims 10 to 12, wherein the candidate factors include factors having time-series data values ​​that change over a certain period, and at least one factor having data values ​​that remain static over the same period.

14. The computer program product according to any one of claims 10 to 13, wherein the transformer is a multi-head transformer having self-attention and cross-attention, and the transformer generates the future value based on the self-attention of individual time series data and the cross-attention between the plurality of time series data input to the transformer.

15. The device is to be made to replace at least one of the candidate factors in the set of candidate factors used to train the low-accuracy classification model. The device is made to replace at least one candidate factor having the lowest adjusted factor weight, selected from the set of candidate factors used to train the low-accuracy classification model, with a candidate factor having the highest adjusted factor weight, selected from the set of candidate factors used to train the higher-accuracy classification, and not included in the set of candidate factors used to train the low-accuracy classification model. A computer program product according to any one of claims 9 to 14, including the following:

16. The computer program product according to any one of claims 9 to 15, wherein the device is configured to adjust the factor weights, the device is configured to adjust the factor weights of the candidate factors in a certain set used in the classification model by multiplying the accuracy score by the factor weights of the candidate factors.

17. At least one computer processor; At least one memory device coupled to the computer processor; Equipped with, The aforementioned at least one computer processor comprises at least, Selecting multiple sets of candidate factors, wherein the candidate factors are indicators of computer performance, and each set of the multiple sets of candidate factors has a different combination of candidate factors than another set of the multiple sets of candidate factors; For a certain set of the multiple sets of candidate factors, A classification model is trained to predict the likelihood of problems during computer runtime using a training dataset associated with the aforementioned set. The classification model is validated using the validation dataset associated with the aforementioned set, and the accuracy score associated with the classification model is determined, and Based on the accuracy score, adjust the factor weights associated with the candidate factors within a given set. Herein, the computer processor is configured to repeat the training of the classification model, the validation of the classification model, and the adjustment of the factor weights for all sets of the multiple sets of candidate factors, where multiple classification models are trained corresponding to each of the multiple sets of candidate factors; Selecting a low-accuracy classification model from the multiple classification models that has an accuracy score lower than a threshold criterion; Selecting a higher-accuracy classification model from the plurality of classification models that has a higher accuracy score than the accuracy score of the low-accuracy classification model; Updating the set of candidate factors used to train the low-accuracy classification model by replacing at least one candidate factor in the set of candidate factors used to train the low-accuracy classification model with a candidate factor in the set of candidate factors used to train the higher-accuracy classification model; and Updating the low-accuracy classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated set of candidate factors. It is configured to do the following: system.

18. The aforementioned computer processor further, With respect to one of the sets of candidate factors, A transformer having an encoder and decoder architecture is trained to generate future values ​​for at least some of the candidate factors within a certain set. Here, The device is made to repeat the training of the transformers for all sets of the multiple sets of candidate factors, along with the repeated training of the classification model, the validation of the classification model, and the adjustment of the factor weights, such that multiple transformers are trained corresponding to each of the multiple sets of candidate factors; and The computer processor being configured to update the low-accuracy classification model includes being configured to update the low-accuracy classification model by repeatedly performing the training of the transformer, the training of the classification model, the validation of the classification model, and the adjustment of the factor weights using the updated set of candidate factors. The system according to claim 17, configured to perform the following:

19. The system according to claim 18, wherein the computer processor is further configured to pass a new set of feature values ​​corresponding to the updated set of candidate factors to the transformer, which is trained using the updated set of candidate factors, to generate future values ​​corresponding to the updated set of candidate factors, and to use the generated future values ​​as input to the updated low-precision classification model to predict future problem probabilities.

20. The system according to claim 18 or 19, wherein the computer processor is further configured to repeat the selection of the low-precision classification model, the selection of the higher-precision classification model, the updating of the set of candidate factors used to train the low-precision classification model, and the updating of the low-precision classification model until all of the low-precision classification models among the plurality of classification models have been updated.

Images