Automatic runtime preemptive alarm

Abstract

For a set of the set of candidate factors, the classification model is trained to predict a likelihood of a computer problem, an accuracy score for the classification model is determined based on model verification, and factor weights for the candidate factors in the set are adjusted based on the accuracy score. This process is performed with respect to all candidate factor sets. A low accuracy classification model having an accuracy score below a threshold criterion is selected. A higher accuracy classification model having an accuracy score that is higher than the accuracy score of the low accuracy classification model is selected. A set of candidate factors for training a low-accuracy classification model is updated using one or more candidate factors for training a higher-accuracy classification model. And updating the low-accuracy classification model based on the updated candidate factor set.

Description

Background Technology

[0001] This application relates generally to computers and computer applications, and more specifically to distributed systems, machine learning, and, for example, providing automatic preemptive alerts during the runtime of a computer system. Summary of the Invention

[0002] The overview of this disclosure is provided to aid in understanding computer systems and methods that automatically provide preemptive alerts for system problems, such as during runtime, and is not intended to limit the disclosure or the invention. It should be understood that in some cases, various aspects and features of this disclosure may be used advantageously alone, while in others, they may be used in combination with other aspects and features. Therefore, variations and modifications can be made to the computer system and / or its operating methods to achieve different effects.

[0003] In some embodiments, a computer-implemented method includes: selecting a plurality of candidate factor sets, the candidate factors being indicators of computer performance, and wherein each set of the plurality of candidate factor sets has a combination of candidate factors different from another set of the plurality of candidate factor sets. The method further includes: for one set of the plurality of candidate factor sets, training a classification model using a training dataset associated with said set to predict the probability of a problem during computer runtime; validating the classification model using a validation dataset associated with said set and determining an accuracy score associated with the classification model; and adjusting factor weights associated with the candidate factors in said set based on the accuracy scores. For all sets of the plurality of candidate factor sets, the training, validation, and factor weight adjustments of the classification model are repeated, wherein multiple classification models corresponding to the respective plurality of candidate factor sets are trained. The method further includes selecting low-accuracy classification models from the plurality of classification models that have accuracy scores below a threshold criterion. The method further includes selecting high-accuracy classification models from the plurality of classification models that have accuracy scores higher than those of the low-accuracy classification models. The method further includes updating the candidate factor set used to train the low-accuracy classification model by replacing at least one candidate factor in the candidate factor set used to train the low-accuracy classification model with a candidate factor from the candidate factor set used to train the high-accuracy classification model. The method also includes updating the low-accuracy classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated candidate factor set.

[0004] In some embodiments, a system includes at least one computer processor. The system also includes at least one memory device coupled to the computer processor. The at least one computer processor is configured to select a plurality of candidate factor sets, the candidate factors being indicators of computer performance, and wherein each of the plurality of candidate factor sets has a combination of candidate factors different from another set of the plurality of candidate factor sets. The at least one computer processor is further configured to: for one of the plurality of candidate factor sets, train a classification model using a training dataset associated with said set to predict the probability of a problem during computer runtime; validate the classification model using a validation dataset associated with said set and determine an accuracy score associated with the classification model; and adjust factor weights associated with the candidate factors in said set based on the accuracy scores, wherein the computer processor is configured to repeatedly train the classification model, validate the classification model, and adjust the factor weights for all sets in the plurality of candidate factor sets, wherein multiple classification models corresponding to the respective plurality of candidate factor sets are trained. The at least one computer processor is further configured to select a low-accuracy classification model from the plurality of classification models that has an accuracy score below a threshold criterion. The at least one computer processor is further configured to select a high-accuracy classification model from the plurality of classification models that has an accuracy score higher than that of the low-accuracy classification model. At least one computer processor is further configured to update the candidate factor set used for training the low-accuracy classification model by replacing at least one candidate factor in the candidate factor set used for training the low-accuracy classification model with a candidate factor from the candidate factor set used for training the high-accuracy classification model. The at least one computer processor is also configured to update the low-accuracy classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated candidate factor set.

[0005] A computer-readable storage medium may also be provided for storing a machine-executable program of instructions for performing one or more methods described herein.

[0006] Further features and the structure and operation of various embodiments are described in detail below with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same or similarly functional elements. Attached Figure Description

[0007] Figure 1 An example computing environment is shown that can implement automatic preemptive alerts for system problems during runtime in an embodiment.

[0008] Figure 2 This is a diagram illustrating the architecture for providing preemptive alerts in one or more computer systems in some embodiments.

[0009] Figure 3Examples of time-series data associated with example candidate factors are shown in some embodiments.

[0010] Figure 4 The transformer architecture is shown in some embodiments.

[0011] Figure 5 This is a diagram illustrating a classification model in some embodiments.

[0012] Figure 6 The use of a classification model to evaluate factor weights is shown in some embodiments.

[0013] Figure 7 The following examples illustrate updating factor candidates used to build a classification model.

[0014] Figure 8 This is a flowchart illustrating a method for predicting and providing predictive alerts for potential computer system and / or application problems in some embodiments.

[0015] Figure 9 This is a diagram illustrating components of a system that, in some embodiments, can provide preemptive alerts for system and / or application problems during runtime. Detailed Implementation

[0016] In some embodiments, a computer-implemented method includes: selecting a plurality of candidate factor sets, the candidate factors being indicators of computer performance, and wherein each set of the plurality of candidate factor sets has a combination of candidate factors different from another set of the plurality of candidate factor sets. The method further includes: for one set of the plurality of candidate factor sets, training a classification model using a training dataset associated with said set to predict the probability of a problem during computer runtime; validating the classification model using a validation dataset associated with said set and determining an accuracy score associated with the classification model; and adjusting factor weights associated with the candidate factors in said set based on the accuracy scores. For all sets of the plurality of candidate factor sets, the training, validation, and factor weight adjustments of the classification model are repeated, wherein multiple classification models corresponding to the respective plurality of candidate factor sets are trained. The method further includes selecting a low-accuracy classification model from the plurality of classification models that has an accuracy score below a threshold criterion. The method further includes selecting a high-accuracy classification model from the plurality of classification models that has an accuracy score higher than that of the low-accuracy classification model. The method further includes updating the candidate factor set used to train the low-accuracy classification model by replacing at least one candidate factor in the candidate factor set used to train the low-accuracy classification model with a candidate factor from the candidate factor set used to train the high-accuracy classification model. The method also includes updating the low-accuracy classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated candidate factor set.

[0017] In some embodiments, the method can identify problems in distributed and / or multi-cloud environments, even when their symptoms are buried in irrelevant data, massive, high-speed, and overwhelmingly diverse data deluges. In some embodiments, the method can proactively prevent potential underlying problems that could lead to high-level symptoms. In some embodiments, the method can automatically provide preemptive alerts about computer system and / or application problems during runtime.

[0018] One or more of the following features may be separate from or optional. In some embodiments, the method further includes: for one set of a plurality of candidate factor sets, training a transformer with an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in the set. For all sets of the plurality of candidate factor sets, repeatedly training the transformer, along with repeatedly training the classification model, validating the classification model, and adjusting the factor weights, wherein multiple transformers corresponding to the respective plurality of candidate factor sets are trained. Updating the low-accuracy classification model includes updating the low-accuracy classification model by repeatedly training the transformer, training the classification model, validating the classification model, and adjusting the factor weights using the updated candidate factor set. For example, training the transformer allows for prediction of new, unseen data using the transformer, which can be used for candidate factor selection.

[0019] In some embodiments, the method further includes: passing a new set of feature values ​​corresponding to the updated set of candidate factors to a transformer trained using the updated set of candidate factors to generate future values ​​corresponding to the updated set of candidate factors, and using the generated future values ​​as input to an updated low-accuracy classification model to predict the probability of future problems. For example, using a transformer for data prediction can improve the accuracy of the prediction.

[0020] In some embodiments, the method further includes repeatedly selecting low-accuracy classification models, selecting higher-accuracy classification models, updating the set of candidate factors used to train the low-accuracy classification models, and updating the low-accuracy classification models until all low-accuracy classification models among the multiple classification models have been updated. In this way, for example, all classification models can be improved, for example, by having accuracy scores above a threshold criterion.

[0021] In some embodiments, candidate factors include factors with time-series data values ​​that change over a period of time and at least one factor with data values ​​that remain static during that period. In this way, the classification model can handle various types of factors, including both time-series data and static data.

[0022] In some embodiments, the transformer is a multi-head transformer with self-attention and cross-attention, wherein the transformer generates future values ​​based on the self-attention of individual time series data points among multiple time series data points input to the transformer, as well as the cross-attention between the multiple time series data points. In this way, for example, more accurate future values ​​can be generated based on self-attention within a set and cross-attention across candidate factors. For example, dependencies between candidate factors can be considered when generating future values ​​for individual candidate factors.

[0023] In some embodiments, replacing at least one candidate factor in the candidate factor set used to train a low-accuracy classification model includes replacing at least one candidate factor selected from the candidate factor set used to train a low-accuracy classification model with a candidate factor selected from the candidate factor set used to train a high-accuracy classification model, which has the highest adjustment factor weight and is not also in the candidate factor set used to train a low-accuracy classification model. In this way, for example, one or more candidate factors that may contribute the least to the decision of the classification model can be replaced with one or more candidate factors that may contribute more to the decision of the classification model.

[0024] In some embodiments, adjusting factor weights includes adjusting the factor weights of candidate factors in the set used in the classification model by multiplying the accuracy score by the factor weight of the candidate factor. In this way, the weights of individual candidate factors used in the classification model can also be adjusted based on the overall performance accuracy of the classification model.

[0025] A system is also disclosed, comprising at least one computer processor and at least one storage device coupled to the at least one computer processor, wherein the at least one computer processor is configured to perform one or more of the methods described above. A computer program product is also disclosed, comprising a computer-readable storage medium having program instructions embodied therein, wherein the program instructions are readable by a device to cause the device to perform one or more of the methods described above.

[0026] Various aspects of this disclosure are described by narrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). Regarding any flowchart, depending on the technology involved, operations may be performed in a different order than that shown in a given flowchart. For example, again according to the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0027] Computer Program Product Embodiment (“CPP Embodiment” or “CPP”) is a term used in this disclosure to describe any collection of one or more storage media (also referred to as “media”) collectively included in a collection of one or more storage devices, the collection of one or more storage devices collectively including machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A “storage device” is any tangible device capable of holding and storing instructions used by a computer processor. Without limitation, a computer-readable storage medium can be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punch cards or pits / platforms formed in the main surface of the disk), or any suitable combination of the foregoing. Computer-readable storage media, as used in this disclosure, should not be construed as storing transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses through fiber optic cables, electrical signals transmitted through wires, and / or other transmission media. As those skilled in the art will understand, data is typically moved at certain incidental points in time during the normal operation of the storage device, such as during access, defragmentation, or garbage collection; however, this does not render the storage device transient, as the data is not transient when it is stored.

[0028] The computing environment 100 includes examples of environments for executing at least some of the computer code involved in executing the inventive methods, such as preemptive alarm algorithm code 200. In addition to block 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end-user equipment (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including processing circuitry 120 and a cache 121), a communication infrastructure 111, volatile memory 112, persistent storage 113 (including an operating system 122 and block 200, as described above), a peripheral device set 114 (including a user interface (UI) device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. Public cloud 105 includes gateway 140, cloud orchestration module 141, host physical machine set 142, virtual machine set 143, and container set 144.

[0029] Computer 101 can take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or to be developed in the future capable of running programs, accessing networks, or querying databases such as remote database 130. As is well known in the field of computer technology, and depending on that technology, the execution of computer-implemented methods can be distributed among multiple computers and / or multiple locations. On the other hand, in this presentation of computing environment 100, the detailed discussion focuses on a single computer, specifically computer 101, to keep the presentation as simple as possible. Computer 101 can reside in the cloud, even... Figure 1 It is not shown that it is in the cloud. On the other hand, unless explicitly instructed otherwise, computer 101 is not required to be in the cloud.

[0030] Processor set 110 includes one or more computer processors of any type now known or to be developed in the future. Processing circuitry 120 may be distributed across multiple packages, such as multiple cooperating integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by the threads or cores running on processor set 110. Cache memory is typically organized into multiple levels based on its relative proximity to the processing circuitry. Alternatively, some or all of the cache in the processor set may be located “off-chip.” In some computing environments, processor set 110 may be designed to work with qubits and perform quantum computing.

[0031] Computer-readable program instructions are typically loaded onto computer 101 to cause the processor set 110 of computer 101 to perform a series of operational steps to implement a computer-implemented method, such that the instructions thus executed instantiate the method specified in the flowcharts and / or narrative descriptions of the computer-implemented method included in this document (collectively, the “inventive method”). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the inventive method. In computing environment 100, at least some of the instructions for performing the inventive method may be stored in block 200 of persistent storage 113.

[0032] Communication structure 111 is a signal transmission path that allows various components of computer 101 to communicate with each other. Typically, this structure consists of switches and conductive paths, such as switches and conductive paths forming buses, bridges, physical input / output ports, etc. Other types of signal communication paths can be used, such as fiber optic communication paths and / or wireless communication paths.

[0033] Volatile memory 112 is any type of volatile memory now known or to be developed in the future. Examples include dynamically typed random access memory (RAM) or statically typed RAM. Typically, volatile memory 112 is characterized by random access, but this is not required unless explicitly indicated. In computer 101, volatile memory 112 is located in a single package and is internal to computer 101; however, alternatively or additionally, volatile memory may be distributed across multiple packages and / or located externally relative to computer 101.

[0034] Persistent storage 113 is any form of non-volatile storage for a computer, now known or to be developed in the future. The non-volatility of this storage means that the stored data is retained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. Persistent storage 113 may be read-only memory (ROM), but typically at least a portion of persistent storage allows for data writing, data deletion, and data rewriting. Some common forms of persistent storage include disks and solid-state storage devices. Operating system 122 may take several forms, such as various known proprietary operating systems or operating systems employing an open-source portable operating system interface type with a kernel. The code included in block 200 generally includes at least some of the computer code involved in performing the inventive methods described herein.

[0035] Peripheral device set 114 includes a collection of peripheral devices for computer 101. Data communication connections between peripheral devices and other components of computer 101 can be implemented in various ways, such as Bluetooth connectivity, near field communication (NFC) connectivity, connections made by cables (such as Universal Serial Bus (USB) type cables), plug-in connections (e.g., secure digital (SD) cards), connections made via local area communication networks, and even connections made via wide area networks such as the Internet. In various embodiments, UI device set 123 may include components such as displays, speakers, microphones, wearable devices (such as glasses and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 124 is an external storage device, such as an external hard drive, or a pluggable storage device, such as an SD card. Storage 124 can be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 requires a large amount of storage (e.g., where computer 101 locally stores and manages a large database), this storage can be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in IoT applications. For example, one sensor could be a thermometer, while another could be a motion detector.

[0036] Network module 115 is a collection of computer software, hardware, and firmware that allows computer 101 to communicate with other computers via WAN 102. Network module 115 may include hardware such as a modem or Wi-Fi transceiver, software for packetizing and / or depacketizing data for transmission over the communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control and network forwarding functions of network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN), the control and forwarding functions of network module 115 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for performing the methods of the invention can typically be downloaded to computer 101 from an external computer or external storage device via a network adapter card or network interface included in network module 115.

[0037] WAN 102 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances using any technology known now or developed in the future for transmitting computer data. In some embodiments, WAN 102 may be replaced by and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0038] End User Equipment (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating computer 101) and can take any of the forms discussed above in conjunction with computer 101. EUD 103 typically receives helpful and useful data from the operation of computer 101. For example, assuming computer 101 is designed to provide recommendations to the end user, these recommendations are typically transmitted from network module 115 of computer 101 to EUD 103 via WAN 102. In this way, EUD 103 can display or otherwise present recommendations to the end user. In some embodiments, EUD 103 can be a client device, such as a thin client, a thick client, a mainframe computer, a desktop computer, etc.

[0039] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 can be controlled and used by the same entity operating computer 101. Remote server 104 represents a machine that collects and stores helpful and useful data used by other computers, such as computer 101. For example, if computer 101 is designed and programmed to provide recommendations based on historical data, that historical data can be provided to computer 101 from a remote database 130 of remote server 104.

[0040] Public cloud 105 is any computer system that can be used by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities (especially data storage (cloud storage) and computing power) without direct active management by users. Cloud computing typically leverages resource sharing to achieve scalability consistency and economy. Direct and active management of the computing resources of public cloud 105 is performed by the computer hardware and / or software of cloud orchestration module 141. The computing resources provided by public cloud 105 are typically implemented by virtual computing environments running on various computers constituting host physical set 142, which is the entirety of physical computers in and / or available to the public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set 143 and / or containers from container set 144. It should be understood that these VCEs can be stored as images and can be transferred between various physical machine hosts as images or after the VCEs are instantiated. Cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of VCEs, and manages active instantiations of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that allow public cloud 105 to communicate via WAN 102.

[0041] Now, we will provide some further explanation of Virtualized Computing Environments (VCEs). A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from this image. Two common types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows multiple isolated user-space instances, called containers, to exist. From the perspective of the programs running within them, these isolated user-space instances typically appear as actual computers. Computer programs running on a regular operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running within a container can only use the contents of the container and the devices allocated to the container; this is a characteristic known as containerization.

[0042] Private cloud 106 is similar to public cloud 105, except that computing resources are available only to a single enterprise. While private cloud 106 is depicted as communicating with WAN 102, in other embodiments, private cloud may be completely disconnected from the Internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types) typically implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardization or proprietary technology that enables orchestration, management, and / or data / application portability across the multiple component clouds. In this embodiment, public cloud 105 and private cloud 106 are both part of a larger hybrid cloud.

[0043] In various embodiments, systems, methods, and / or techniques are disclosed that automatically provide preemptive alerts for one or more computer system and / or application problems during runtime. In some embodiments, the systems, methods, and / or techniques randomly select and initialize time-series factors and static factors to monitor, for example, a single system across multiple systems. Factors are performance indicators that display measurements of computer system performance or status. In some embodiments, the systems, methods, and / or techniques analyze time-series data based on historical data using a multi-head transformer employing self-attention and cross-attention. In some embodiments, the systems, methods, and / or techniques train a multi-head transformer model to generate one or more values ​​for future times. In some embodiments, the systems, methods, and / or techniques use a classification model to predict the probability of system problems in the current system. In some embodiments, the systems, methods, and / or techniques calculate weights for the current system and obtain adjusted weights within a specific model for a specific factor based on cross-system comparisons. In some embodiments, the systems, methods, and / or techniques select factor candidates from relatively high-accuracy systems based on factor weight ranking and update candidate factors for relatively low-accuracy systems. In some embodiments, the systems, methods, and / or techniques iterate training, prediction, and tuning steps to improve the predictive accuracy of the classification model.

[0044] Figure 2This is a diagram illustrating the architecture for providing preemptive alerts in one or more computer systems in some embodiments. The components shown are computer-implemented components. During initialization, candidate factors are selected from various factors 202 (shown as factor candidates), as shown at 204 for systems in a computing environment (e.g., a distributed computer system environment). The selection at 204 can be performed in a random manner. For example, candidate factors can be randomly selected from a pool of factor candidates 202. The selection can be performed for each of multiple computer systems in the computing environment (e.g., 206, 208, 210). The selected candidate factors may include factors with time-series data 212, 214, 216 and static factors with static data 218, 220, 222. Candidate factors with time-series data have values ​​that change over time. Candidate factors with static data have values ​​that remain constant over time. In one aspect, different factors or sets of factors can be selected for each system 204, 206, 210, for example, because collecting all performance factors for a single system may be expensive.

[0045] Factors are performance or operational indicators of a computer system. Factors can be selected as candidate factors from one or more performance or operational categories. For example, factors in the mutex (mutual exclusion) wait queue category may include semaphores, locks, condition variables, spinlocks, barriers, and / or others. Factors in the input / output (I / O) category may include file I / O, socket I / O, network I / O, disk I / O, and / or keyboard I / O. Factors in the malloc (dynamic memory allocation) quantity category may include heap allocation, memory management, dynamic memory allocation, memory fragmentation, and / or memory leaks. Factors in the network connectivity category may include Transmission Control Protocol (TCP) connections, User Datagram Protocol (UDP) connections, socket connections, network protocols, and / or bandwidth. Factors in the semaphore category may include interrupt signals, signal handlers, signal masks, signal delivery, and / or signal propagation. Factors in the critical section length category may include mutexes, synchronization, deadlock thread synchronization, and / or concurrency control. Factors in the workload request frequency category may include request rate, throughput, response time, latency, and / or queue length. Factors in the Central Processing Unit (CPU) category may include processor, core, multithreading, instruction set, and / or cache hierarchy. Factors in the Memory category may include Random Access Memory (RAM), virtual memory, page faults, memory hierarchy, and / or Memory Management Unit (MMU). Factors in the Application Tracing category may include debugging, system hardware / software analysis, logging, performance monitoring, and / or tracing tools. Other factors or factors from other categories may be selected as candidate factors.

[0046] Factors with time-series data are fed through a transformer that predicts the next data point in the time series. Transformers 224, 226, and 228 are trained using historical time-series data to predict future values ​​in the time series. For example, considering sys1 (system) 206, time-series data 212 associated with its candidate factors is input into transformer 224. The transformer has an encoder-decoder architecture. In some embodiments, multi-head self-attention and cross-attention can be used as encoders in training for multiple types of time-series data. For example, mutex wait queues, workload request frequencies, and critical section lengths are different types of time-series data. For example, the mutex wait queue for the next time step will be influenced by multiple time-series data. Transformer 224 predicts the factor value for the next time step or unit in the time series, as shown at 230.

[0047] Figure 3 Examples of time-series data associated with example candidate factors are shown in some embodiments. The sliding window size is k (where k can be predefined or preconfigured), and time-series data 302, 304, and 306 for each candidate factor are used as a transformer (e.g., Figure 2 The input (224) is used to train the transformer to predict future time values. Once trained, the transformer (e.g., ...) can predict future time values. Figure 2 (224) receives input, which includes time-series data of window size k, such as [Q1, Q2, ..., Qk, T1, T2, ..., Tk, V1, V2, ..., Vk], where, for example, Q1, Q2, ..., Qk are time-series data associated with candidate factors such as mutex wait queues, T1, T2, ..., Tk are time-series data associated with candidate factors such as critical section length, and V1, V2, ..., Vk are time-series data associated with candidate factors such as workload request frequency. Given these inputs, the transformer (e.g., Figure 2 224) Predict the next value in the time series data, Qk+1, Tk+1, Vk+1, where k+1 is the next time step or unit.

[0048] Figure 4 It is shown in some embodiments, for example in Figure 2The transformer architecture is shown at position 224. Transformer 406 is trained based on vectorized values ​​of vector 404 or input 402 (a set of sliding windows of size k), where k is configurable or predefined. Transformer 406 uses self-attention encoding on each candidate factor time series, then cross-attention across all sequences. Encoded time series data of different types are concatenated for input to fully connected layer 408. For example, the input to transformer 406 is a sequence of time series tokens. For example, vectors Q, T, and V404 represent different types of time series data (shown as input 402). Each vector 404 is fed into self-attention. This self-attention mechanism captures the relationships between different tokens in the input sequence. In the decoder of transformer 406, cross-attention is used to generate output vectors Q', T', and V'. For example, the decoder generates the target sequence for one token at a time. To generate the next token, the decoder considers the tokens it has generated so far and relevant information from the input sequence. For example, the decoder simultaneously generates the next labels Qk+1, Tk+1, and Vk+1. The fully connected layer 408 (e.g., a softmax layer) outputs a prediction of the next value for each candidate factor time series 410.

[0049] Back Figure 2 Static candidate factor data 218 and predicted time-series candidate factor values ​​230 are input into classification model 236, which predicts possible problems 242 that may occur in the future. For example, the classification model is trained and used to predict possible system problems. Time-series predicted data 230 and other data 218 are used as input to classification model 236. Classification model 236 outputs the probability of the problem. In some embodiments, the input data is preprocessed to ensure that the distribution of different data (e.g., time distribution or frequency) is uniform. Classification model 236 is trained using labeled training data to stabilize the model. For example, if the model's predictions are within a threshold range of their actual values ​​(e.g., real data), the model is considered stable. Once the model is considered stable, the weight (W) of a particular factor in a particular model is determined. Factor weights describe or indicate the importance of features in classification model 236, such as which features have the greatest impact on the model's predictions or how important a feature is in the classification model's predictions. Known techniques can be used to identify or determine such weights. For example, existing software development tool libraries provide built-in methods for directly computing feature importance scores from trained patterns (such as trained decision trees).

[0050] Figure 5 This is a diagram illustrating the classification model in some embodiments, for example in Figure 2As shown at 236. Time series prediction data 502 (e.g., mutex wait queues, workload requests, malloc counts) and other data 504 (e.g., CPU, memory, signals) are used as input to classification model 506. Classification model 506 outputs the probability of a problem. Example results are shown at 510. For example, classification model 506 running different inference operations with different input data can provide different results. Based on the predicted future values ​​of time series data and other data (e.g., static data), classification model 506 outputs a score or probability of a problem occurring in a given computer system with conditions described or indicated by the input data. For example, the results may show a 70% probability of a problem, etc. In some embodiments, the input data is preprocessed 508 to ensure that the distribution of the different data (e.g., time distribution or frequency) is uniform. Classification model 506 is trained using labeled training data to make the model stable. For example, if the model's predictions are within a threshold range of their actual values ​​(e.g., real data), the model is considered stable. Examples of classification models 506 include decision tree models, random forest models, and / or other models.

[0051] Back Figure 2 The problem probabilities predicted by classification model 236 in future time 242 are validated using actual known data (e.g., real data), as shown at 248. For example, actual or real data can be obtained at that future time and compared with the predicted problem probabilities. At 254, the evaluation and ranking factor weights are calculated, for example, as referenced. Figure 6 As stated at 626, 632 and 638.

[0052] A similar process flow is performed for other computer systems, such as sys2 at 208 and sys3 at 210. For example, candidate factors are selected for system 208. Time series data 214 associated with the candidate factors is fed through a transformer 226 trained on historical time series data associated with those candidate factors, so that the transformer 226 can predict future values ​​(e.g., the value of the next time step or unit in the time series) 232, for example, as referenced. Figure 4 The static candidate factor data 220 and the predicted time-series candidate factor values ​​232 are input into a classification model 238 that predicts the probability of a problem 244, for example, as described in the reference. Figure 5 The problem probabilities predicted by classification model 238 in future time 244 are validated using actual known data (e.g., real data), as shown at 250. For example, actual or real data can be obtained at that future time and compared with the predicted problem probabilities. At 256, the evaluation and ranking factor weights are, for example, as referenced... Figure 6As stated at 628, 634 and 638.

[0053] Similarly, candidate factors are selected for system 210. Time series data 216 associated with the candidate factors is fed through a transformer 228 trained on historical time series data associated with these candidate factors, so that transformer 228 can predict future values ​​(e.g., the value of the next time step or unit in the time series) 234, for example, as referenced. Figure 4 As described, static candidate factor data 222 and predicted time-series candidate factor values ​​234 are input into a classification model 240 that predicts the probability of a problem 246, for example, as referenced. Figure 5 As described. The problem probability 246 in the future time predicted by classification model 240 is validated with actual known data (e.g., real data), as shown at 252. For example, actual or real data can be obtained at that future time and compared with the predicted problem probability. At 258, the evaluation and ranking factor weights are, for example, as referenced. Figure 6 As described at 630, 636, and 638.

[0054] Figure 6 This illustrates the use of classification models to evaluate factor weights in some embodiments, such as... Figure 2 As shown at position 254. Time series data 602 of the candidate factors can be used. For example, for training, the candidate factors can be divided into model training data 608 and model validation data 614. Model training data 608 can be used to train the classification model 620, for example, also refer to... Figure 5 The following description is provided. Model validation data 614 is used to predict the probability value of the question. For example, model validation data 614 is input into classification model 620, and classification model 620 is run for inference. Classification model 620 outputs its prediction of the probability of the question. The predicted probability value of the question and the true probability value of the question (actual or true value) are used as inputs to loss function 626 to calculate the accuracy (A) of classification model 620. In addition, the weights (W) of the candidate factors in classification model 620 are obtained. W and A are multiplied to obtain the adjusted weight for a specific factor within a specific model, as shown at 632.

[0055] Similar processing is performed on other systems. For example, for another computer system, time-series data 604 of candidate factors selected for that computer system can be used. For example, for training, the candidate factors can be divided into model training data 610 and model validation data 616. Model training data 610 is used to train classification model 622, for example, also referencing... Figure 5The following is described. Model validation data 616 is used to predict the probability value of the problem. For example, model validation data 616 is input into classification model 622, and classification model 620 is run for inference. Classification model 622 outputs its prediction of the problem probability. The predicted problem probability value and the true problem probability (actual or true value) are used as inputs to loss function 628 to calculate the accuracy (A) of classification model 622. In addition, the weights (W) of candidate factors in classification model 622 are obtained. W and A are multiplied to obtain the adjusted weights for a specific factor within a specific model, as shown at 634.

[0056] Similarly, for another computer system, time-series data 606 of candidate factors selected for that computer system can be used. For example, for training, the candidate factors can be divided into model training data 612 and model validation data 618. Model training data 612 is used to train classification model 624, for example, also referencing Figure 5 The following description is provided. Model validation data 618 is used to predict the probability value of the problem. For example, model validation data 618 is input into classification model 624, and classification model 624 is run for inference. Classification model 624 outputs its prediction of the problem probability. The predicted problem probability value and the true problem probability (actual or true value) are used as input to loss function 630 to calculate the accuracy (A) of classification model 620. Additionally, the weights (W) of candidate factors in classification model 624 are obtained. W and A are multiplied to obtain the adjusted weights for a specific factor within a specific model, as shown at 636.

[0057] The adjusted weights from all models 632, 634, and 636 (where a specific factor is included as an input factor) are integrated to obtain the final weight 638 for that factor. For example, the integration could be a weighted average. For example, as shown in 638, F1, ..., F7 represent candidate factors, and W1, ..., W7 represent the weights of candidate factors F1, ..., F7. For example, W1, ..., W7 as shown in 638 is a weighted average of W1, ..., W7 as shown in 632, 634, and 636. For instance, if three systems (System 1, System 2, System 3) have a factor F1, then the integrated W1 could be W1 = (W1 of System 1 * Accuracy A of System 1 + W1 of System 2 * Accuracy A of System 2 + W1 of System 3 * Accuracy A of System 3) / 3, where '*' indicates multiplication, '+' indicates addition, and ' / ' indicates division.

[0058] Reference Back Figure 2At position 260, the factors are ranked according to their adjusted weights. For example, the integrated value based on the candidate factor weights (e.g., a weighted average) (e.g., as...). Figure 6 (As shown at position 638) A uniform factor ranking is performed across all systems.

[0059] Factor weights and / or factor weight rankings from each system (e.g., 254, 256, and 258) are used to update the factor selection at 262. For example, one or more classification models 236, 238, and 240 (which may have low accuracy in their problem probability prediction or classification) can be remodeled using different candidate factors. These different candidate factors can be selected based on the factor weights determined at 254, 256, and 258. Whether a classification model has low accuracy can be determined based on whether the classification meets a threshold accuracy (which may be predefined).

[0060] Figure 7 The illustration shows how, in some embodiments, factor candidates used in building a classification model are updated. For example, for a system with a low-accuracy classification model (e.g., determined based on a threshold accuracy criterion), the classification model can be retrained using different or updated candidate factors. In some embodiments, one or more new factor candidates can be selected from a system with a high-accuracy model, for example, based on a ranking order of factor weights (e.g., such as...). Figure 2 (As shown at 760). Consider System 1 at 702, which has an accuracy of 0.7. This accuracy is considered low according to predefined criteria. For example, predefined criteria state that an accuracy score or level less than 0.75 is low. Note that other predefined criteria exist for determining whether an accuracy score or level is low. Also consider System 2 at 704, which has a higher accuracy score or level, for example, 0.9. One or more new factor candidates are selected from System 2 704 for System 1 702. For example, a factor selector 706 may select a candidate factor in System 2 704 that has the highest ranking weight and is not currently also used as a candidate factor in System 1 702. As shown at 708, the selected candidate factor replaces the lowest-weighted factor in the low-accuracy system, such as System 1. In some embodiments, more than one candidate factor may be replaced. For example, a predefined number of candidate factors may be replaced, or a candidate factor with a weight that meets the criteria (e.g., a weight less than a predefined threshold weight or a relatively low weight compared to other candidate factors) may be replaced.

[0061] Potential computer system problems can be detected early, for example, before they occur, allowing for possible prevention or preventative measures. This can prevent future serious system problems from occurring by enabling appropriate responses to potential issues. Alerts for potential computer system and / or application problems can be generated promptly. This can save time and effort spent on root cause analysis after a problem has occurred.

[0062] In some embodiments, the methods described herein can be used directly as a solution or as a standalone tool used by computer operating systems, observability systems, and monitoring systems, in both real production and development environments. The methods described herein can be used by all types of computer operating systems and runtime environments.

[0063] Figure 8 This is a flowchart illustrating a method for predicting and providing predictive alerts for potential computer system problems (e.g., operating system and / or application problems during runtime) in some embodiments. The method can be implemented and / or run on one or more computer processors. For example, the method can predict computer system problems during runtime, such as process or thread hangs, deadlocks, and bottlenecks. Also refer to the above... Figure 2-7 Details of the method in various embodiments are described. At 802, a plurality of candidate factor sets are selected. Candidate factors are indicators of computer performance. Examples of such factors have been described above. Each of the plurality of candidate factor sets has a combination of candidate factors that differs from another set of the plurality of candidate factor sets. In some embodiments, a set of candidate factors may be randomly selected from a pool of computer system performance indicators.

[0064] For one set of multiple candidate factors, processing options 804, 806, and 808 can be applied. See also... Figure 6 The process is described. At 804, a classification model is trained using the training dataset associated with the set to predict the probability of a problem during computer runtime. At 806, the classification model is validated using the validation dataset associated with the set, and an accuracy score associated with the classification model is determined. At 808, the factor weights associated with candidate factors in the set are adjusted based on the accuracy score. Examples of classification models include, but are not limited to, decision trees, random forests, neural networks, and / or others.

[0065] The method repeats the training, validation, and factor weight adjustment of classification models for all sets in a plurality of candidate factor sets, where multiple classification models are trained for each set of candidate factors. For example, at step 810, if more sets need to be processed, the method proceeds to step 804, and the processing at steps 804, 806, and 808 is repeated for the next set of candidate factors. In some embodiments, different sets of candidate factor sets may be associated with different computer systems in a distributed system.

[0066] At point 812, select the low-accuracy classification model from multiple classification models that has an accuracy score below a threshold criterion. This threshold criterion can be an absolute number, such as 0.7 from a score range such as 0-10. As another example, the threshold criterion can be a relative number, such as 50% lower than the highest score in the score range (e.g., 0 to 10). Another threshold criterion can be used or defined.

[0067] At point 814, a higher accuracy classification model is selected from multiple classification models, having an accuracy score higher than that of the lower accuracy classification model. In some embodiments, the higher accuracy classification model with the highest accuracy score among the multiple classification models is selected.

[0068] At point 816, the candidate factor set used to train the low-accuracy classification model is updated by replacing at least one candidate factor in the candidate factor set used to train the low-accuracy classification model with a candidate factor from the candidate factor set used to train the high-accuracy classification model. At point 818, the low-accuracy classification model is updated by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using the updated candidate factor set.

[0069] In some embodiments, the method can identify problems in distributed and / or multi-cloud environments, even when their symptoms are buried in irrelevant data, massive, high-speed, and overwhelmingly diverse data deluges. In some embodiments, the method can proactively prevent potential underlying problems that could lead to high-level symptoms. In some embodiments, the method can automatically provide preemptive alerts about computer system and / or application problems during runtime.

[0070] In some embodiments, the method further includes: for the set of a plurality of candidate factors, training a transformer with an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in the set, for example, as referenced Figure 4 as well as Figure 2As described. For all sets in the multiple candidate factor sets, along with repeatedly training the classification model, validating the classification model, and adjusting the factor weights, the transformer is repeatedly trained, where multiple transformers are trained for the corresponding multiple candidate factor sets. Updating the low-accuracy classification model involves repeatedly training the transformer, training the classification model, validating the classification model, and adjusting the factor weights using the updated candidate factor sets.

[0071] In some embodiments, the method further includes: passing a new set of feature values ​​corresponding to the updated set of candidate factors to a transformer trained with the updated set of candidate factors to generate future values ​​corresponding to the updated set of candidate factors, and using the generated future values ​​as input to an updated low-accuracy classification model to predict the probability of future problems.

[0072] In some embodiments, the method further includes repeatedly selecting low-accuracy classification models, selecting higher-accuracy classification models, updating the set of candidate factors used to train the low-accuracy classification models, and updating the low-accuracy classification models until all low-accuracy classification models among the multiple classification models have been updated. In this way, for example, all classification models can be improved, for example, by having accuracy scores above a threshold criterion.

[0073] In some embodiments, candidate factors include factors with time-series data values ​​that change over a period of time and at least one factor with data values ​​that remain static during that period. In this way, the classification model can handle various types of factors, including both time-series data and static data.

[0074] In some embodiments, the transformer is a multi-head transformer with self-attention and cross-attention, wherein the transformer generates future values ​​based on the self-attention of individual time series data points among multiple time series data points input to the transformer, as well as the cross-attention between the multiple time series data points. In this way, for example, more accurate future values ​​can be generated based on self-attention within a set and cross-attention across candidate factors. For example, dependencies between candidate factors can be considered when generating future values ​​for individual candidate factors.

[0075] In some embodiments, replacing at least one candidate factor in the candidate factor set used to train a low-accuracy classification model includes replacing at least one candidate factor selected from the candidate factor set used to train a low-accuracy classification model with a candidate factor selected from the candidate factor set used to train a high-accuracy classification model, which has the highest adjustment factor weight and is not also in the candidate factor set used to train a low-accuracy classification model. In this way, for example, one or more candidate factors that may contribute the least to the decision of the classification model can be replaced with one or more candidate factors that may contribute more to the decision of the classification model.

[0076] In some embodiments, adjusting factor weights includes adjusting the factor weights of candidate factors in the set used in the classification model by multiplying the accuracy score by the factor weight of the candidate factor. In this way, the weights of individual candidate factors used in the classification model can also be adjusted based on the overall performance accuracy of the classification model.

[0077] In some embodiments, a computer-implemented method includes selecting a plurality of candidate factor sets. Candidate factors are indicators of computer performance. Each set of the plurality of candidate factor sets has a combination of candidate factors that differs from another set of the plurality of candidate factor sets. For example, a set of candidate factors may be randomly selected from a pool of computer system performance indicators. For one set of the plurality of candidate factor sets, the following operations may be performed: training a transformer with an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in that set; training a classification model using a training dataset associated with that set to predict the probability of a problem during computer runtime; validating the classification model using a validation dataset associated with that set and determining an accuracy score associated with the classification model; and adjusting the factor weights associated with the candidate factors in that set based on the accuracy score. The training of the transformer, the training of the classification model, the validation of the classification model, and the adjustment of the factor weights are repeated for all sets of the plurality of candidate factor sets. Thus, multiple transformers and multiple classification models are trained corresponding to the respective plurality of candidate factor sets.

[0078] The method also includes selecting low-accuracy classification models from multiple classification models that have accuracy scores below a threshold criterion. This threshold criterion can be an absolute number, such as 0.7 from a score range such as 0-10. As another example, the threshold criterion can be a relative number, such as 50% lower than the highest score in the score range (e.g., 0 to 10). Another threshold criterion can be used or defined.

[0079] The method also includes selecting a higher accuracy classification model from among multiple classification models, which has an accuracy score higher than that of the lower accuracy classification model. In some embodiments, the classification model with the highest accuracy score can be selected.

[0080] The method further includes updating the candidate factor set for training the low-accuracy classification model by replacing at least one candidate factor in the candidate factor set for training the low-accuracy classification model with a candidate factor in the candidate factor set for training the high-accuracy classification model.

[0081] The method also includes updating the low-accuracy classification model by repeating the training of the transformer, the training of the classification model, the validation of the classification model, and the adjustment of the factor weights using an updated set of candidate factors.

[0082] A computer program product may be provided, comprising a computer-readable storage medium having program instructions embodied thereon, which can be read by a device to cause the device to perform one or more methods described herein. A system may be provided, comprising at least one computer processor and at least one memory device coupled to the computer processor, wherein the at least one computer processor is configured to perform one or more methods described herein.

[0083] Figure 9This is a diagram illustrating components of a system that, in some embodiments, can provide preemptive alerts for system and / or application problems during runtime. One or more hardware processors 902 (such as a central processing unit (CPU), graphics processing unit (GPU) and / or field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), and / or other processors) may be coupled to a memory device 904. The memory device 904 may include random access memory (RAM), read-only memory (ROM), or other memory devices, and may store data and / or processor instructions for implementing various functions associated with the methods and / or systems described herein. The one or more processors 902 may execute computer instructions stored in the memory 904 or received from another computer device or medium. The memory device 904 may, for example, store instructions and / or data for the functions of the one or more hardware processors 902, and may include operating system and other program instructions and / or data. The one or more hardware processors 902 may select from a plurality of candidate sets. For one of a plurality of candidate factor sets, one or more hardware processors 902 can train a classification model using a training dataset associated with that set to predict the probability of a problem during computer runtime, validate the classification model using a validation dataset associated with that set and determine an accuracy score associated with the classification model, and adjust the factor weights associated with the candidate factors in that set based on the accuracy scores. One or more hardware processors 902 can repeat the training, validation, and factor weight adjustment for all sets in the plurality of candidate factor sets. In this way, multiple classification models corresponding to the respective plurality of candidate factor sets are trained. One or more hardware processors 902 can select a low-accuracy classification model from the plurality of classification models that has an accuracy score below a threshold criterion. One or more hardware processors 902 can select a higher-accuracy classification model from the plurality of classification models that has an accuracy score higher than that of the low-accuracy classification model. One or more hardware processors 902 can update the candidate factor set used to train the low-accuracy classification model by replacing at least one candidate factor in the candidate factor set used to train the low-accuracy classification model with a candidate factor from the candidate factor set used to train the higher-accuracy classification model. One or more hardware processors 902 can update a low-accuracy classification model by repeatedly training the classification model, validating the classification model, and adjusting the factor weights using an updated set of candidate factors. In one aspect, candidate factor data, such as training data and validation data, can be stored in storage device 906 or received from a remote device via network interface 908, and can be temporarily loaded into memory device 904 to generate a classification model. The learned classification model can, for example, be stored on memory device 904 for execution by one or more hardware processors 902.One or more hardware processors 902 may be coupled to an interface device such as a network interface 908 for communicating with a remote system, for example, via a network, and to an input / output interface 910 for communicating with input and / or output devices such as a keyboard, mouse, or monitor.

[0084] Systems and methods in some embodiments can identify problems in distributed and / or multi-cloud environments, even when their symptoms are buried in irrelevant, massive, high-speed, and overwhelmingly diverse data deluges. Systems and methods in some embodiments can proactively prevent potential underlying problems that could lead to high-level symptoms. For example, in some embodiments, systems and methods can automatically provide preemptive alerts about computer system and / or application problems during runtime.

[0085] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context explicitly indicates otherwise. As used herein, the term “or” is an inclusive operator and may mean “and / or” unless the context explicitly or clearly indicates otherwise. It will be further understood that, as used herein, “comprising,” “including,” “containing,” “comprising,” and / or “having” may specify the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the phrase “in an embodiment” or “in some embodiments” does not necessarily refer to the same embodiment, although they may be. As used herein, the phrase “in one embodiment” does not necessarily refer to the same embodiment, although it may be. As used herein, the phrase “in another embodiment” does not necessarily refer to a different embodiment, although it may be. Furthermore, embodiments and / or components of embodiments can be freely combined with each other unless they are mutually exclusive.

[0086] The corresponding structures, materials, actions, and equivalents of all means or steps plus functional elements (if any) in the following claims are intended to include any structure, material, or action used in combination with other claim elements to perform the claimed function, as specifically claimed. The invention has been described for illustrative and descriptive purposes but is not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. Embodiments have been chosen and described to best explain the principles and practical application of the invention and to enable others skilled in the art to understand the invention to be adapted to various embodiments and to make various modifications to suit a particular intended use.

Claims

1. A computer-implemented method, comprising: Select multiple sets of candidate factors, where each set of candidate factors is an indicator of computer performance, and each set of the multiple sets of candidate factors has a combination of candidate factors that is different from another set of the multiple sets of candidate factors. For one set of the multiple candidate factor sets A classification model is trained using a training dataset associated with the set to predict the probability of a problem during computer runtime. The classification model is validated using the validation dataset associated with the set, and the accuracy score associated with the classification model is determined. The factor weights associated with the candidate factors in the set are adjusted based on the accuracy scores. For all sets in the plurality of candidate factor sets, the classification model is repeatedly trained, the classification model is validated, and the factor weights are adjusted, wherein the training is performed on the plurality of classification models corresponding to the plurality of candidate factor sets. Select a low-accuracy classification model from the multiple classification models, which has an accuracy score below the threshold standard. Select the higher accuracy classification model from the plurality of classification models, which has an accuracy score higher than that of the lower accuracy classification model. The candidate factor set for training the low-accuracy classification model is updated by replacing at least one candidate factor in the candidate factor set for training the low-accuracy classification model with a candidate factor in the candidate factor set for training the high-accuracy classification model. as well as The low-accuracy classification model is updated by repeatedly training and validating the classification model using the updated set of candidate factors and adjusting the factor weights.

2. The computer-implemented method according to claim 1 further includes: For the set of the plurality of candidate factors Train a transformer with an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in the set. For all sets of the plurality of candidate factor sets, along with repeatedly training the classification model, validating the classification model, and adjusting the factor weights, the transformer is repeatedly trained, wherein multiple transformers corresponding to the respective plurality of candidate factor sets are trained; and Updating a low-accuracy classification model involves repeatedly training the transformer, training the classification model, validating the classification model, and adjusting the factor weights using an updated set of candidate factors.

3. The computer-implemented method according to claim 2 further includes passing a new set of feature values ​​corresponding to the updated set of candidate factors to a transformer trained with the updated set of candidate factors to generate future values ​​corresponding to the updated set of candidate factors, and using the generated future values ​​as input to the updated low-accuracy classification model to predict the probability of future problems.

4. The computer-implemented method according to claim 2 or 3 further includes repeatedly selecting a low-accuracy classification model, selecting a higher-accuracy classification model, updating the candidate factor set used to train the low-accuracy classification model, and updating the low-accuracy classification model until all low-accuracy classification models among the plurality of classification models have been updated.

5. The computer-implemented method according to any one of claims 2 to 4, wherein the candidate factors include factors having time-series data values ​​that change over a period of time and at least one factor having data values ​​that remain static over said period of time.

6. The computer-implemented method according to any one of claims 2 to 5, wherein the transformer is a multi-head transformer with self-attention and cross-attention, wherein the transformer generates future values ​​based on the self-attention of each time series data in a plurality of time series data input to the transformer and the cross-attention between the plurality of time series data.

7. The computer-implemented method according to any one of the preceding claims, wherein replacing at least one candidate factor in the set of candidate factors used to train a low-accuracy classification model comprises: Replace at least one candidate factor selected from the candidate factor set for training the low-accuracy classification model with the candidate factor selected from the candidate factor set for training the high-accuracy classification model, which has the highest adjustment factor weight and is not also in the candidate factor set for training the low-accuracy classification model.

8. The computer-implemented method according to any of the preceding claims, wherein adjusting factor weights comprises adjusting the factor weights of candidate factors in the set used in the classification model by multiplying the accuracy score by the factor weights of the candidate factors.

9. A computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being readable by a device to cause the device to: Select multiple sets of candidate factors, where each set of candidate factors is an indicator of computer performance, and each set of the multiple sets of candidate factors has a combination of candidate factors that is different from another set of the multiple sets of candidate factors. For one set of the multiple candidate factor sets A classification model is trained using a training dataset associated with the set to predict the probability of a problem during computer runtime. The classification model is validated using the validation dataset associated with the set, and the accuracy score associated with the classification model is determined. The factor weights associated with the candidate factors in the set are adjusted based on the accuracy scores. The device repeatedly trains, validates, and adjusts the factor weights for all sets in the plurality of candidate factor sets, wherein the training is performed on the plurality of classification models corresponding to the plurality of candidate factor sets. Select a low-accuracy classification model from the multiple classification models, which has an accuracy score below the threshold standard. Select the higher accuracy classification model from the plurality of classification models, which has an accuracy score higher than that of the lower accuracy classification model. The candidate factor set for training the low-accuracy classification model is updated by replacing at least one candidate factor in the candidate factor set for training the low-accuracy classification model with a candidate factor in the candidate factor set for training the high-accuracy classification model. as well as The low-accuracy classification model is updated by repeatedly training and validating the classification model using the updated set of candidate factors and adjusting the factor weights.

10. The computer program product of claim 9, further comprising the device: For the set of the plurality of candidate factors Train a transformer with an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in the set. The device repeatedly trains a transformer for all sets of the plurality of candidate factor sets, together with repeatedly training the classification model, validating the classification model, and adjusting the factor weights, wherein multiple transformers are trained corresponding to the respective plurality of candidate factor sets; and Updating a low-accuracy classification model for a device involves repeatedly training the transformer, training the classification model, validating the classification model, and adjusting the factor weights to update the low-accuracy classification model using an updated set of candidate factors.

11. The computer program product of claim 10, wherein the device further causes the device to pass a new set of feature values ​​corresponding to the updated set of candidate factors to a transformer trained with the updated set of candidate factors to generate future values ​​corresponding to the updated set of candidate factors, and uses the generated future values ​​as input to an updated low-accuracy classification model to predict the probability of future problems.

12. The computer program product of claim 10 or 11, wherein the device is further configured to repeatedly select a low-accuracy classification model, select a higher-accuracy classification model, update the set of candidate factors for training the low-accuracy classification model, and update the low-accuracy classification model until all low-accuracy classification models among the plurality of classification models have been updated.

13. The computer program product according to any one of claims 10 to 12, wherein the candidate factors include factors having time-series data values ​​that change over a period of time and at least one factor having data values ​​that remain static over said period of time.

14. The computer program product according to any one of claims 10 to 13, wherein the transformer is a multi-head transformer with self-attention and cross-attention, wherein the transformer generates future values ​​based on the self-attention of each time series data in a plurality of time series data input to the transformer and the cross-attention between the plurality of time series data.

15. The computer program product according to any one of claims 9 to 14, wherein causing the device to replace at least one candidate factor in the set of candidate factors for training a low-accuracy classification model comprises: The device replaces at least one candidate factor selected from the candidate factor set for training the low-accuracy classification model with a candidate factor selected from the candidate factor set for training the high-accuracy classification model, which has the highest adjustment factor weight and is not also in the candidate factor set for training the low-accuracy classification model.

16. The computer program product according to any one of claims 9 to 15, wherein adjusting the factor weights of the device comprises adjusting the factor weights of the candidate factors in the set used in the classification model by multiplying the accuracy score by the factor weights of the candidate factors.

17. A system comprising: At least one computer processor; At least one memory device coupled to a computer processor; At least one computer processor is configured to at least: Select multiple sets of candidate factors, where each set of candidate factors is an indicator of computer performance, and each set of the multiple sets of candidate factors has a combination of candidate factors that is different from another set of the multiple sets of candidate factors. For one set of the multiple candidate factor sets A classification model is trained using a training dataset associated with the set to predict the probability of a problem during computer runtime. The classification model is validated using the validation dataset associated with the set, and the accuracy score associated with the classification model is determined. The factor weights associated with the candidate factors in the set are adjusted based on the accuracy scores. The computer processor is configured to repeatedly train the classification model, validate the classification model, and adjust the factor weights for all sets in the plurality of candidate factor sets, wherein the training is performed on the plurality of classification models corresponding to the respective plurality of candidate factor sets. Select a low-accuracy classification model from the multiple classification models, which has an accuracy score below the threshold standard. Select the higher accuracy classification model from the plurality of classification models, which has an accuracy score higher than that of the lower accuracy classification model. The candidate factor set for training the low-accuracy classification model is updated by replacing at least one candidate factor in the candidate factor set for training the low-accuracy classification model with a candidate factor in the candidate factor set for training the high-accuracy classification model. as well as The low-accuracy classification model is updated by repeatedly training and validating the classification model using the updated set of candidate factors and adjusting the factor weights.

18. The system of claim 17, wherein the computer processor is further configured to: For the set of the plurality of candidate factors Train a transformer with an encoder and decoder architecture to generate future values ​​for at least some of the candidate factors in the set. The device repeatedly trains a transformer for all sets of the plurality of candidate factor sets, together with repeatedly training the classification model, validating the classification model, and adjusting the factor weights, wherein multiple transformers are trained corresponding to the respective plurality of candidate factor sets; and The computer processor is configured to update the low-accuracy classification model by repeatedly training the transformer, training the classification model, validating the classification model, and adjusting the factor weights using an updated set of candidate factors.

19. The system of claim 18, wherein the computer processor is further configured to pass a new set of feature values ​​corresponding to the updated set of candidate factors to a transformer trained with the updated set of candidate factors to generate future values ​​corresponding to the updated set of candidate factors, and to use the generated future values ​​as input to an updated low-accuracy classification model to predict the probability of future problems.

20. The system of claim 18 or 19, wherein the computer processor is further configured to repeatedly select a low-accuracy classification model, select a higher-accuracy classification model, update the set of candidate factors used to train the low-accuracy classification model, and update the low-accuracy classification model until all the low-accuracy classification models among the plurality of classification models have been updated.

Images