Method, electronic device and computer program product for training a failure analysis model

Abstract

Embodiments of the present disclosure relate to a method for training a fault analysis model, an electronic device, and a computer program product. In the method for training a fault analysis model provided by embodiments of the present disclosure, at least one set of log files including a plurality of preprocessed log files is obtained, the at least one set of log files including an annotated fault cause of a storage system, and a preprocessed log file of the plurality of preprocessed log files including one or more potential fault causes of the storage system and a score associated with the potential fault cause; based on the potential fault cause and the score of the preprocessed log file, a fault cause of the storage system is predicted according to the fault analysis model; and parameters of the fault analysis model are updated based on a probability that the predicted fault cause is the annotated fault cause. In this way, the fault cause can be analyzed for a log package instead of a log file, improving the fault analysis efficiency.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of information processing, and more specifically, to methods, electronic devices, and computer program products for training fault analysis models. Background Technology

[0002] With the development of information technology, the number of companies providing information services is increasing. Especially for companies providing large-scale and complex information services, users are prone to encountering various problems. These problems may be caused by software defects, hardware or network issues, or operational errors. As a result, technical support engineering teams need to handle a large number of service requests related to problem reports. Summary of the Invention

[0003] In a first aspect of this disclosure, a method is provided for training a fault analysis model to determine the causes of storage system failures. The method includes acquiring at least one set of log files comprising multiple preprocessed log files, the at least one set of log files including labeled failure causes of the storage system, and the preprocessed log files including one or more potential failure causes of the storage system and scores associated with the potential failure causes. The method further includes predicting the failure causes of the storage system based on the potential failure causes and scores in the multiple preprocessed log files, according to the fault analysis model. The method also includes updating the parameters of the fault analysis model based on the probability that the predicted failure cause is a labeled failure cause.

[0004] In a second aspect of this disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor. The actions include acquiring at least one set of log files comprising a plurality of preprocessed log files, the at least one set of log files including labeled failure causes of a storage system, and the preprocessed log files including one or more potential failure causes of the storage system and scores associated with the potential failure causes. The actions also include predicting failure causes of the storage system based on the potential failure causes and scores in the plurality of preprocessed log files, according to a failure analysis model. The actions further include updating parameters of the failure analysis model based on the probability that the predicted failure cause is a labeled failure cause.

[0005] In a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the method according to the first aspect.

[0006] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0007] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings, in which like reference numerals generally denote like parts. In the drawings:

[0008] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure may be implemented is shown;

[0009] Figure 2 A flowchart illustrating an example method for training a fault analysis model according to an embodiment of the present disclosure is shown;

[0010] Figure 3 A schematic diagram of a preprocessed file used in some embodiments of this disclosure is shown;

[0011] Figure 4 The diagram illustrates fault characteristics according to some embodiments of the present disclosure, the fault characteristics being... Figure 3 It is generated by aggregating the scores from the preprocessed files; and

[0012] Figure 5 A block diagram of an example device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0013] The principles of this disclosure will now be described with reference to several exemplary embodiments illustrated in the accompanying drawings. While preferred embodiments of this disclosure are shown in the drawings, it should be understood that these embodiments are described merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0014] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0015] For large and complex storage services, technical support engineers always handle a large volume of customer service requests, including reports of problems caused by software defects, hardware or network issues, or operational errors. Inferring the root cause of a system failure from log packets (composed of several related log files, each containing consecutive log lines over a specific time interval) is a valuable task in the service request classification process, as clients are often asked to upload log packets (composed of the most recent log files) for further categorization of the cause of the failure. Most methods for analyzing failures from logs are developed for "log files" rather than "log packets."

[0016] Embodiments of this disclosure propose a scheme for training a fault analysis model to determine the causes of storage system failures. During model training, at least one set of log files, comprising multiple preprocessed log files, are acquired. These preprocessed log files include one or more potential causes of storage system failures and scores associated with those potential causes. Then, based on the potential causes and scores, the fault analysis model predicts the causes of storage system failures. The parameters of the fault analysis model are updated based on the probabilities of the predicted failure causes as labeled failure causes included in the at least one set of log files.

[0017] According to the embodiments described herein, a trained fault analysis model can be used to analyze the causes of faults in "log packets" uploaded by clients. This improves fault analysis efficiency and assists less experienced engineers in handling customer service requests.

[0018] The basic principles and several exemplary implementations of this disclosure will be described below with reference to the accompanying drawings. It should be understood that these exemplary embodiments are given only to enable those skilled in the art to better understand and implement the embodiments of this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0019] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure may be implemented is shown. (See diagram for reference.) Figure 1 As shown, environment 100 includes cloud server 110 and client 120. When client 120 fails, a log packet 130 is generated. Log packet 130 includes multiple log files. To analyze the cause of the client 120 failure, client 120 is required to upload log packet 130 to cloud server 110. Client 120 can be a user device or workstation. Examples of client 120 may include, but are not limited to, cellular mobile phones, tablet devices, laptops, etc.

[0020] The cloud server 110 can preprocess the log files in the log packet 130 uploaded by the client 120. Then, the preprocessed log packet 130 is input into the trained fault analysis model 150 to analyze and determine the cause of the system failure of the client 120. The training method of this fault analysis model 150 will be discussed later. Figure 2 Provide a detailed description.

[0021] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. For example, embodiments of this disclosure can also be applied to environments different from environment 100. Furthermore, although Figure 1 Only one log packet is shown, but it is not limited to this; there may be more or fewer log packets.

[0022] Figure 2 A flowchart of an example method 200 for training a fault analysis model according to an embodiment of the present disclosure is shown. Method 200 can be, for example, by... Figure 1 The cloud server 110 shown executes this. It should be understood that method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of this disclosure is not limited in this respect. The following is in conjunction with... Figure 1 Let me describe method 200 in detail.

[0023] At box 210, cloud server 110 acquires at least one set of log files comprising multiple preprocessed log files. The multiple preprocessed log files are obtained by cloud server 110 preprocessing the log files in log packet 130 uploaded by client 120. Preprocessing is performed on a single log file. The preprocessing of log files will be briefly described below. It should be understood that the preprocessing methods described below are merely illustrative, and other methods may be used to preprocess log files; the scope of this disclosure is not limited in this respect.

[0024] In preprocessing, each log file in log package 130 is segmented into multiple log segments, each segment comprising multiple log lines. By aggregating the log lines to form a log pattern, each log line can be identified as a Log DNA ID, and the log segment can be converted into a series of log identifiers. This series of log identifiers, as a feature representation of the log segment, is then input into a model to construct a classifier 140 for fault classification of the log segments. Using this classifier 140, the log file can be scanned, and a classification score for each log segment is output. If the classification score exceeds a fault classification threshold, it indicates that the log segment may indicate a fault of the type associated with that classification score. In some embodiments, other classifiers may also be used to scan the log file.

[0025] Afterward, technical support engineers can analyze the output of classifier 140. Based on their experience, engineers identify the causes of system failures and manually label them. Each log packet 130 corresponds to a cause of failure. Log files from multiple log packets 130 are preprocessed to prepare for training data.

[0026] Therefore, the log files in the preprocessed log package 130 are combined into a set of preprocessed log files. This set of log files includes the causes of failure in the storage system of client 120, which are manually labeled. Each of the preprocessed log files includes one or more potential causes of failure in the storage system and a score associated with that potential cause. Potential causes of failure may be represented, for example, by fields such as `starvation_detect` and `sc_be_fenced_out`.

[0027] In some embodiments, the preprocessed log file also includes a log file type associated with the potential cause of the failure, such as an ALERT warning, a MESSAGE message, etc. In some embodiments, the preprocessed log file also includes a storage node identifier associated with the log file type, such as storage node identifiers X1-SC1, X1-SC2, etc. It is worth noting that there is no storage node identifier associated with the log file type for an ALERT warning; this can be represented by an empty NULL.

[0028] Because it's difficult to perform fine-grained analysis on the entire log package—the time intervals for fine-grained analysis would be too long (typically greater than 24 hours), and the data from fine-grained analysis would be too large (typically greater than 1GB)—in some embodiments, multiple preprocessed log files can be divided into multiple time windows. Then, the divided preprocessed log files within each time window are obtained as at least one set of log files. The time window can be, for example, 1 hour, or from 0.5 hours to several hours.

[0029] At box 220, based on potential failure causes and scores from multiple preprocessed log files, the failure cause of the storage system is predicted according to failure analysis model 150. In some embodiments, scores from multiple preprocessed log files are aggregated based on the failure type of the potential failure cause to generate failure features. The failure features at least indicate the failure type and the aggregated score for the corresponding failure type. The failure cause of the storage system is then predicted based on the generated failure features according to failure analysis model 150.

[0030] In some embodiments, scores from multiple preprocessed log files can be aggregated based on the fault type of the potential fault cause and the log file types associated with the potential fault cause included in the preprocessed log files.

[0031] In some embodiments, scores from multiple preprocessed log files can be aggregated based on the fault type of the potential fault cause, the log file type, and the storage node identifier associated with the log file type included in the preprocessed log files. This will be discussed later in conjunction with... Figure 3 and Figure 4 Describe it.

[0032] At box 230, the parameters of the fault analysis model 150 are updated based on the probability that the predicted fault cause is a labeled fault cause. In some embodiments, the fault analysis model 150 is updated using the updated parameters based on the probability that the predicted fault cause is a labeled fault cause being less than a threshold. The threshold can be determined experimentally or set by engineers based on experience. For example, the threshold could be 70%. The algorithm used to train the fault analysis model 150 can be a random forest algorithm, a logistic regression algorithm, or a softmax algorithm, etc. The number of training iterations can also be set as needed, for example, 100 times.

[0033] In this way, training data is prepared for log packet 130 to train fault analysis model 150, which can then be used to analyze the causes of faults in log packet 130. Compared to predicting multiple potential fault causes for log files, this method requires less experience from engineers in fault analysis and improves fault analysis efficiency.

[0034] Figure 3 A schematic diagram of a preprocessed file according to some embodiments of the present disclosure is shown. Figure 4 The diagram illustrates fault characteristics according to some embodiments of the present disclosure, the fault characteristics being... Figure 3 It is generated by aggregating the scores from the preprocessed files. Figure 3 The diagram shows two fault types, `starvation_detect` and `sc_be_fenced_out`, two log file types, `ALERT` and `MESSAGE`, and two storage node identifiers, `X1-SC1` and `X1-SC2`, associated with the `MESSAGE` log file type. There is no storage node identifier associated with the `ALERT` log file type; therefore, in... Figure 3 The value is represented as NULL.

[0035] based on Figure 3The scores, after aggregating the fault types, log file types, and storage node identifiers shown, are as follows: Figure 4 The list of fault characteristics is shown. Each fault characteristic indicates the fault type, log file type, storage node identifier, and aggregated score.

[0036] It should be understood that, although Figure 3 The document shows that the preprocessed file includes the fault type, log file type, and storage node identifier; the preprocessed file may also include other additional information. Although Figure 4 It shows the basis Figure 3 The results of aggregating scores based on fault type, log file type, and storage node identifier can also be aggregated based on one or more of these factors. In some embodiments, scores can also be aggregated based on additional information from the preprocessed file.

[0037] Figure 5 A schematic block diagram of an example device 500 that can be used to implement embodiments of the present disclosure is shown. For example, such as Figure 1 The cloud server 110 shown can be implemented by device 500. For example... Figure 5 As shown, device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 502 or loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0038] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0039] The various processes and handling described above, such as method 200, can be executed by processing unit 501. For example, in some embodiments, method 200 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by CPU 501, one or more actions of method 200 described above can be performed.

[0040] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0041] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0042] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0043] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0044] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0045] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0046] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0047] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0048] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a fault analysis model, the fault analysis model being used to determine the causes of failures in a storage system, the method comprising: Obtain at least one set of log files comprising multiple preprocessed log files, the at least one set of log files including labeled failure causes of the storage system, and the preprocessed log files among the multiple preprocessed log files including one or more potential failure causes of the storage system and scores associated with the potential failure causes; Based on the potential causes of failure in the multiple preprocessed log files and the scores, the causes of failure in the storage system are predicted according to the failure analysis model. as well as The parameters of the fault analysis model are updated based on the probability of the predicted fault cause being the labeled fault cause. Each preprocessed log file in at least one subset of the preprocessed log files includes multiple fields for the following corresponding items: fault type, log file type, storage node identifier of the storage system, and score for the corresponding preprocessed log file; and The fault characteristics are determined, at least in part, by aggregating the scores of multiple preprocessed log files among the preprocessed log files based on one or more of the fault type, the storage node identifier, and the log file type.

2. The method of claim 1, wherein predicting the cause of failure in the storage system comprises: Based on the fault type of the potential fault cause, the scores from the multiple preprocessed log files are aggregated to generate fault features, which at least indicate the fault type and the aggregated score for the corresponding fault type; as well as Based on the fault characteristics, the cause of the storage system failure is predicted according to the fault analysis model.

3. The method of claim 2, wherein aggregating the scores from the plurality of preprocessed log files includes: The scores are aggregated based on the fault type of the potential fault cause and the log file types associated with the potential fault cause included in the preprocessed log files among the multiple preprocessed log files.

4. The method of claim 3, wherein aggregating the scores from the plurality of preprocessed log files further comprises: The score is aggregated based on the fault type of the potential fault cause, the log file type, and the storage node identifier associated with the log file type included in the preprocessed log files among the plurality of preprocessed log files.

5. The method according to claim 1, further comprising: The fault analysis model is updated using the parameters based on the probability being less than the threshold.

6. The method of claim 1, wherein obtaining at least one set of log files further comprises: The multiple preprocessed log files are divided into multiple time windows; as well as Obtain the multiple preprocessed log files divided within the time windows of the multiple time windows, as the at least one set of log files.

7. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, the instructions, when executed by the processor, cause the device to perform actions for training a fault analysis model, the fault analysis model being used to determine the causes of failures in the storage system, the actions including: Obtain at least one set of log files comprising multiple preprocessed log files, the at least one set of log files including labeled failure causes of the storage system, and the preprocessed log files among the multiple preprocessed log files including one or more potential failure causes of the storage system and scores associated with the potential failure causes; Based on the potential causes of failure in the multiple preprocessed log files and the scores, the causes of failure in the storage system are predicted according to the failure analysis model; and The parameters of the fault analysis model are updated based on the probability of the predicted fault cause being the labeled fault cause. Each preprocessed log file in at least a subset of the preprocessed log files includes multiple fields for the following corresponding items: fault type, log file type, storage node identifier of the storage system, and a score for the corresponding preprocessed log file; and The fault characteristics are determined, at least in part, by aggregating the scores of multiple preprocessed log files among the preprocessed log files based on one or more of the fault type, the storage node identifier, and the log file type.

8. The electronic device of claim 7, wherein predicting the cause of failure of the storage system includes: Based on the fault type of the potential fault cause, the scores from the multiple preprocessed log files are aggregated to generate fault features, which at least indicate the fault type and the aggregated score for the corresponding fault type; as well as Based on the fault characteristics, the cause of the storage system failure is predicted according to the fault analysis model.

9. The electronic device of claim 8, wherein aggregating the scores from the plurality of preprocessed log files includes: The scores are aggregated based on the fault type of the potential fault cause and the log file types associated with the potential fault cause included in the preprocessed log files among the multiple preprocessed log files.

10. The electronic device of claim 9, wherein aggregating the scores from the plurality of preprocessed log files further comprises: The score is aggregated based on the fault type of the potential fault cause, the log file type, and the storage node identifier associated with the log file type included in the preprocessed log files among the plurality of preprocessed log files.

11. The electronic device according to claim 7, wherein the action further includes: The fault analysis model is updated using the parameters based on the probability being less than the threshold.

12. The electronic device of claim 7, wherein obtaining at least one set of log files further comprises: The multiple preprocessed log files are divided into multiple time windows; as well as Obtain the multiple preprocessed log files divided within the time windows of the multiple time windows, as the at least one set of log files.

13. A computer program product tangibly stored on a computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 6.

Images