A fault automatic recovery method and system based on AIOps

Abstract

The present invention belongs to the field of computer technology, and provides an AIOps-based fault automatic recovery method and system. The method includes: extracting log templates and log parameters from received unstructured original log files, where log parameters are a set of variable values ​​when the program is running, and when extracting the log template, parsing the log entries in the original log file based on a dynamic analysis tree with a fixed depth to obtain the log template; detecting the obtained log template and log parameters, and during the detection, judging whether the log sequence of the log template is abnormal based on an LSTM network, and detecting the log parameters based on a parameter outlier algorithm and a parameter value first-order difference outlier algorithm to identify abnormal log parameters; based on a pre-created expert library, finding matching items corresponding to abnormal log templates and log parameters, and recovering the fault based on the matching item strategy; therefore, the present invention can solve the problem that it is difficult to automatically detect system anomalies through a large number of logs.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an AIOps-based automatic fault recovery method and system. Background Art

[0002] AIOps is the abbreviation of Artificial Intelligence for IT Operations, which is a method of using artificial intelligence (AI) and machine learning (ML) technologies to improve and simplify information technology operations. AIOps is currently in the exploratory stage of development, and its effective application in operations is relatively rare.

[0003] When operating and maintaining a network system, analyzing logs plays an important role. Logs are very important information storage carriers in production practice. Most software and systems implement log modules to record various key information during runtime, such as CPU utilization, memory usage, and error messages to help troubleshoot. With the rapid development of the information society, the number of logs that need to be analyzed is growing exponentially, and the log format is usually unstructured. It is particularly difficult to find key information from a large amount of log data to detect system or application anomalies. Summary of the invention

[0004] In view of the above technical problems, the present invention provides an AIOps-based automatic fault recovery method and system to solve the problem in the prior art that it is difficult to automatically detect system or application anomalies through a large number of logs.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to one aspect of the present invention, an AIOps-based fault automation recovery method is disclosed, the method comprising:

[0007] Extracting a log template and log parameters from a received unstructured original log file, wherein the log parameters are a set of variable values ​​when a program is running. When extracting the log template, the log entries in the original log file are parsed based on a dynamic analysis tree with a fixed depth to obtain the log template. Each level in the dynamic analysis tree is a judgment of an attribute of the original log file. Each node at a certain level corresponds to a different value of an attribute. Each leaf node stores attributes of multiple log entries. The path from the root node to the leaf node represents the common attributes of the log entries.

[0008] The obtained log template and log parameter are detected. During the detection, whether the log sequence of the log template is abnormal is determined based on the LSTM network, and the log parameter is detected based on the parameter abnormal value algorithm and the parameter value first-order difference abnormal value algorithm to identify the abnormal log parameter;

[0009] Based on the pre-created expert library, a matching item corresponding to the abnormal log template and the log parameter is found, and the fault is recovered based on the strategy of the matching item.

[0010] Furthermore, before extracting the log template and the log parameters in the original log file, based on regular expressions, custom fields and / or custom generalization rules, specific fields and / or generalized specific data of the original log file are extracted.

[0011] Furthermore, after obtaining the log template, the log template is clustered into a data set that can be processed by a deep learning model based on a clustering algorithm.

[0012] Furthermore, the LSTM network is learned based on the following method:

[0013] Slicing the input sequence of the log template to form a training data set;

[0014] Using the first entry of the sequence of the log template in the training data set as a label of the training data set;

[0015] The training data set is sent to the LSTM network for learning.

[0016] Further, after obtaining the training data set, based on the formula evaluating the quality of the training dataset, is the proportion of log sequences marked as abnormal in the sequence of the log template marked as normal, is the ratio of abnormal log sequences in the normal training data set, is the weight.

[0017] Furthermore, after the LSTM network training is completed, the sequence of the log template obtained in real time is input into the LSTM network to obtain a predicted sequence number. If the predicted sequence number is different from the actual sequence number of the log template, it is determined that the current log template is abnormal.

[0018] Further, the detecting the log parameter based on the parameter outlier algorithm and the parameter value first-order difference outlier algorithm to identify the abnormal log parameter includes:

[0019] The sequence of the log parameters obtained in real time is sampled in groups, and the mean is solved to form a new sequence;

[0020] Calculate whether the new series exceeds the confidence interval boundary of the mean at the preset confidence level to obtain outliers;

[0021] The first-order difference between the parameter value calculated by the parameter outlier algorithm and the parameter value first-order difference algorithm is the outlier value, the position of the outlier value is queried, and the log parameter corresponding to the outlier value is judged to be abnormal.

[0022] According to another aspect of the present disclosure, there is provided an AIOps-based automatic fault recovery system, comprising:

[0023] A front-end extraction module is used to extract a log template and log parameters from a received unstructured original log file, wherein the log parameters are a set of variable values ​​when the program is running. When extracting the log template, the log entries in the original log file are parsed based on a dynamic analysis tree with a fixed depth to obtain the log template. Each level in the dynamic analysis tree is a judgment of an attribute of the original log file. Each node at a certain level corresponds to a different value of an attribute. Each leaf node stores attributes of multiple log entries. The path from the root node to the leaf node represents the common attributes of the log entries.

[0024] A back-end computing module is used to detect the acquired log template and the log parameter. During the detection, it is determined whether the log sequence of the log template is abnormal based on the LSTM network, and the log parameter is detected based on the parameter abnormal value algorithm and the parameter value first-order difference abnormal value algorithm to identify the abnormal log parameter.

[0025] The processing module is used to find matching items corresponding to the abnormal log template and the log parameters based on a pre-created expert library, and recover the fault based on the strategy of the matching items.

[0026] The technical solution disclosed in this disclosure has the following beneficial effects:

[0027] Based on the automatic analysis of log files, automatic analysis of operation and maintenance faults is realized. Combined with the strategies specified by the expert database, faults that can be automatically operated are recovered, which greatly simplifies the work of operation and maintenance personnel. By combining the parameter outlier algorithm and the parameter value first-order difference outlier algorithm, end-to-end semi-supervised log anomaly detection is constructed, thereby reducing the difficulty of fault data collection and annotation in operation and maintenance and improving the detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1This is a flowchart of an AIOps-based automatic fault recovery method in an embodiment of this specification;

[0029] Figure 2 This is a structural block diagram of an AIOps-based automatic fault recovery system in an embodiment of this specification;

[0030] Figure 3 A computer-readable storage medium storing an AIOps-based automated fault recovery method in an embodiment of this specification. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, systems, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0032] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor systems and / or microcontroller systems.

[0033] like Figure 1 As shown, the embodiment of this specification provides an AIOps-based automatic fault recovery method, and the execution subject of the method can be a server. The method can specifically include the following steps S101~S103:

[0034] In step S101, a log template and log parameters are extracted from a received unstructured original log file, wherein the log parameters are a set of variable values ​​when a program is running. When extracting the log template, log entries in the original log file are parsed based on a dynamic analysis tree with a fixed depth to obtain the log template. Each level in the dynamic analysis tree is a judgment of an attribute of the original log file, and each node at a certain level corresponds to a different value of an attribute. Each leaf node stores attributes of multiple log entries, and the path from the root node to the leaf node represents the common attributes of the log entries.

[0035] Each record in the log file is printed by a printf statement. Restoring a record to the content of a printf statement is called log template extraction. The closer the extracted log template is to the content of the original printf statement, the better the extraction effect. The parameters in the log entry correspond to the dynamic values ​​of the variables in the original printf statement, that is, the log parameters. The log entry is parsed through a dynamic analysis tree. After multiple decisions from the root node to the leaf node of the dynamic analysis tree, a log entry can be classified into a log template.

[0036] In step S102, the acquired log template and the log parameters are detected. During the detection, whether the log sequence of the log template is abnormal is determined based on the LSTM network, and the log parameters are detected based on the parameter outlier algorithm and the parameter value first-order difference outlier algorithm to identify the abnormal log parameters.

[0037] Among them, the LSTM network is a deep learning network, which can obtain a detection model by inputting training data, so that the log sequence can be detected. In the judgment of log parameters, unsupervised learning is used to detect anomalies. In the operation and maintenance of power systems, it is usually difficult to obtain a large number of high-quality and labeled data sets, so unsupervised learning will be more advantageous. In the case of a large amount of data, the mean distribution of the data satisfies the central limit theorem and conforms to the normal distribution. Here, the sequence of the log parameters obtained in real time can be grouped and sampled, and the mean is solved to form a new sequence, which represents the mean distribution of the data; then, it is calculated whether the confidence interval boundary of the mean is exceeded under the preset confidence level in the new sequence to obtain anomalies, that is, to obtain outliers, and the parameter value calculated by the parameter outlier algorithm and the first-order difference algorithm of the parameter value are calculated to obtain the first-order difference as the outlier. In the first-order difference algorithm, the parameter value set at the same parameter position is respectively subtracted from the last value from the previous value according to the timestamp relationship. Therefore, the position of the outlier can be queried, located in the corresponding log entry, and the log parameter corresponding to the outlier value is judged as anomaly.

[0038] For a network device, the parameter value may be network bandwidth, CPU utilization, memory usage, hard disk space, etc. For an application, the parameter value includes: request response time, number of concurrent users, number of database queries, etc.

[0039] The parameter outlier algorithm and the parameter value first-order difference outlier algorithm are used to identify parameter anomalies. When the amount of data is large, the mean distribution of the data satisfies the central limit theorem and conforms to the normal distribution. The data is grouped and sampled, and the mean is solved to form a new sequence, which represents the mean distribution of the data. The outlier value can be obtained by judging whether the data exceeds the boundary of the data mean confidence interval with a certain confidence level. The outlier algorithm is used for both the parameter value and the first-order difference value of the parameter (the parameter value set at the same parameter position is subtracted from the last value from the previous value according to the timestamp relationship). If the parameter value or the first-order difference of the parameter is an outlier, the parameter anomaly is detected. By querying the position of the outlier value, the corresponding log entry can be located.

[0040] In step S104, based on the pre-created expert library, a matching item corresponding to the abnormal log template and the log parameter is found, and the fault is recovered based on the strategy of the matching item.

[0041] Among them, the expert library is a pre-prepared database or collection, which contains strategies, knowledge or rules for possible abnormal situations, and is used to guide the system to take appropriate measures when facing specific abnormal situations. When the system detects abnormal log templates and log parameters, it queries the pre-created expert library to find strategies or rules that match these abnormalities. When the items in the expert library that match the abnormal log templates and parameters are found, the relevant strategies or knowledge are extracted and appropriate measures are taken, such as troubleshooting, system restart, configuration changes, etc., so that the fault is eliminated, or an alarm can be issued to notify the operation and maintenance personnel to eliminate the fault.

[0042] In one embodiment, before extracting the log template and the log parameters in the original log file, specific fields and / or generalized specific data of the original log file are extracted based on regular expressions, custom fields and / or custom generalization rules.

[0043] Here, users can add specific rules to extract corresponding fields or generalize specific data. This can adapt to different systems and different log formats, ensure that the required information is correctly extracted, help users filter out unnecessary information, and focus on useful content. This can simplify data processing and analysis and improve efficiency.

[0044] In one embodiment, after obtaining the log template, the log template is clustered into a data set that can be processed by a deep learning model based on a clustering algorithm.

[0045] Since deep learning, neural networks, etc. cannot directly process unstructured raw log files, raw log files are usually unstructured and cannot be directly processed by neural networks. Through clustering, log entries can be used to train models.

[0046] As a supplement, when training the LSTM network model, the sequence of the input log template is sliced ​​to form a training data set; the first entry of the sequence of the log template in the training data set is used as the label of the training data set; and the training data set is sent to the LSTM network for learning. Among them, the first entry in the log sequence is used as the label, that is, slicing, so there is no need to specifically label the data, which greatly improves the training speed of the model, and the model granularity can be adjusted by selecting the length of the slice.

[0047] After obtaining the training data set, based on the formula evaluating the quality of the training dataset, is the proportion of log sequences marked as abnormal in the sequence of the log template marked as normal, that is, it measures the degree of repetition of log sequences. is the ratio of abnormal log sequences in the normal training data set, that is, it measures the repetition rate of log sequence slices. Represents the relatively coarse-grained features of the dataset, Represents more fine-grained features. is the weight, which is changed by adjusting The size of the feature can change the ratio of coarse-grained features to fine-grained features. Users can adjust the size according to the actual effect. . The higher the value, the less obvious the difference between the abnormal sequence and the normal sequence, or the less accurate the abnormal label, and the worse the quality of the data set. Therefore, when the Q value is high, it is necessary to re-collect the data or check whether there are many wrong labels.

[0048] After the LSTM network model training is completed, the sequence of the log template obtained in real time is input into the LSTM network to obtain a predicted sequence number. If the predicted sequence number is different from the actual sequence number of the log template, the current log template is judged to be abnormal.

[0049] Based on the same idea, Figure 2As shown, the exemplary embodiment of the present disclosure also provides an AIOps-based fault automatic recovery system, including: a front-end extraction module 201, used to extract log templates and log parameters from the received unstructured original log file, the log parameters are a set of variable values ​​when the program is running, and when extracting the log template, the log entries in the original log file are parsed based on a dynamic analysis tree with a fixed depth to obtain the log template, each level in the dynamic analysis tree is a judgment of an attribute of the original log file, each node at a certain level corresponds to a different value of an attribute, and each leaf node stores The method comprises the following steps: storing the attributes of multiple log entries, wherein the path from the root node to the leaf node represents the common attributes of the log entries; a back-end computing module 202 is used to detect the acquired log template and the log parameters. During the detection, the LSTM network is used to determine whether the log sequence of the log template is abnormal, and the log parameters are detected based on the parameter outlier algorithm and the parameter value first-order difference outlier algorithm to identify the abnormal log parameters; a processing module 203 is used to find matching items corresponding to the abnormal log template and the log parameters based on a pre-created expert library, and recover the fault based on the strategy of the matching items.

[0050] The present invention realizes automatic analysis of operation and maintenance faults based on automatic analysis of log files, and recovers faults that can be automatically operated in combination with the strategies specified by the expert library, which greatly simplifies the work of operation and maintenance personnel. By combining the parameter outlier algorithm and the parameter value first-order difference outlier algorithm, end-to-end semi-supervised log anomaly detection is constructed, thereby reducing the difficulty of fault data collection and annotation in operation and maintenance and improving the detection effect.

[0051] The specific details of each module in the above system have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0052] Based on the same idea, the exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above "Automated Fault Recovery Method Based on AIOps" section of the present specification.

[0053] refer to Figure 3As shown, a program product 300 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, system or device.

[0054] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0055] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, system, or device.

[0056] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0057] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0058] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal system, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0059] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0060] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0061] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. An automated fault recovery method based on AIOps, characterized in that: The method comprises: Extracting a log template and log parameters from a received unstructured original log file, wherein the log parameters are a set of variable values ​​when a program is running. When extracting the log template, the log entries in the original log file are parsed based on a dynamic analysis tree with a fixed depth to obtain the log template. Each level in the dynamic analysis tree is a judgment of an attribute of the original log file. Each node at a certain level corresponds to a different value of an attribute. Each leaf node stores attributes of multiple log entries. The path from the root node to the leaf node represents the common attributes of the log entries. The obtained log template and log parameter are detected. During the detection, whether the log sequence of the log template is abnormal is determined based on the LSTM network, and the log parameter is detected based on the parameter abnormal value algorithm and the parameter value first-order difference abnormal value algorithm to identify the abnormal log parameter. During the identification, the sequence of the log parameter obtained in real time is grouped and sampled, the mean is solved to form a new sequence, and whether the confidence interval boundary of the mean is exceeded under a preset confidence level in the new sequence to obtain the abnormal value. The first-order difference calculated by the parameter value calculated by the parameter abnormal value algorithm and the first-order difference abnormal value algorithm of the parameter value is the abnormal value, then the position of the abnormal value is queried, and the log parameter corresponding to the abnormal value is judged to be abnormal; the LSTM network is learned based on the following method: Slice the sequence of the input log template to form a training data set; use the first entry of the sequence of the log template in the training data set as the label of the training data set; send the training data set to the LSTM network for learning; after obtaining the training data set, based on the formula evaluating the quality of the training dataset, is the proportion of log sequences marked as abnormal in the sequence of the log template marked as normal, is the ratio of abnormal log sequences in the normal training data set, is a weight; after the LSTM network training is completed, the sequence of the log template obtained in real time is input into the LSTM network to obtain a predicted sequence number. If the predicted sequence number is different from the actual sequence number of the log template, the current log template is judged to be abnormal; Based on the pre-created expert library, a matching item corresponding to the abnormal log template and the log parameter is found, and the fault is recovered based on the strategy of the matching item.

2. The AIOps-based automatic fault recovery method according to claim 1, characterized in that: Before extracting the log template and the log parameter in the original log file, based on regular expressions, preset fields and preset generalization rules, corresponding fields and generalized data of the original log file are extracted.

3. According to the AIOps-based automatic fault recovery method of claim 1, it is characterized in that: After obtaining the log template, the log template is clustered into a data set that can be processed by a deep learning model based on a clustering algorithm.

4. An AIOps-based automatic fault recovery system, characterized in that: include: A front-end extraction module is used to extract a log template and log parameters from a received unstructured original log file, wherein the log parameters are a set of variable values ​​when the program is running. When extracting the log template, the log entries in the original log file are parsed based on a dynamic analysis tree with a fixed depth to obtain the log template. Each level in the dynamic analysis tree is a judgment of an attribute of the original log file. Each node at a certain level corresponds to a different value of an attribute. Each leaf node stores attributes of multiple log entries. The path from the root node to the leaf node represents the common attributes of the log entries. The back-end computing module is used to detect the acquired log template and log parameters. During the detection, it is determined whether the log sequence of the log template is abnormal based on the LSTM network, and the log parameters are detected based on the parameter outlier algorithm and the parameter value first-order difference outlier algorithm to identify the abnormal log parameters. During the identification, the sequence of the log parameters obtained in real time is grouped and sampled, the mean is solved to form a new sequence, and it is calculated whether the confidence interval boundary of the mean is exceeded under a preset confidence level in the new sequence to obtain the outlier. The first-order difference calculated by the parameter value calculated by the parameter outlier algorithm and the first-order difference outlier algorithm of the parameter value is the outlier, then the position of the outlier is queried, and the log parameter corresponding to the outlier is judged to be abnormal; the LSTM network is learned based on the following method: Slicing the input sequence of the log template to form a training data set; Using the first entry of the sequence of the log template in the training data set as a label of the training data set; Sending the training data set to the LSTM network for learning; After obtaining the training data set, based on the formula evaluating the quality of the training dataset, is the proportion of log sequences marked as abnormal in the sequence of the log template marked as normal, is the ratio of abnormal log sequences in the normal training data set, is a weight; after the LSTM network training is completed, the sequence of the log template obtained in real time is input into the LSTM network to obtain a predicted sequence number. If the predicted sequence number is different from the actual sequence number of the log template, the current log template is judged to be abnormal; The processing module is used to find matching items corresponding to the abnormal log template and the log parameters based on a pre-created expert library, and recover the fault based on the strategy of the matching items.

Images