A log anomaly detection method and system based on knowledge base and ensemble learning

Abstract

The present invention discloses a log anomaly detection method and system based on knowledge base and integrated learning, wherein the method comprises: obtaining a log; identifying the type of the log; based on the knowledge base, preprocessing the log differently for different log types; if the log is a first type of log, parsing it using a preset template and / or preset rules; if the log is a second type of log, processing it based on integrated learning; matching the preprocessed log with a corresponding log anomaly detection model based on the log type; and outputting the detection result. The present invention focuses on some significant challenges faced by log anomaly detection algorithms in actual production environments, and proposes a log anomaly detection system based on knowledge base and integrated learning, ensuring accuracy and interpretability; at the same time, the present invention designs and implements a suitable anomaly detection algorithm for each log anomaly type, and achieves good results in actual log data sets.

Description

Technical Field

[0001] The present invention belongs to the field of big data analysis and data mining, and in particular relates to a log anomaly detection method and system based on a knowledge base and integrated learning. Background Art

[0002] Logs are an important data source in software systems, which record the system's operating status and user behavior in detail. Log anomaly detection aims to discover abnormal behaviors in the system, which can help engineers find problems in the system in a timely manner and assist in related fault diagnosis. Therefore, log anomaly detection is of great significance to service reliability and fault management.

[0003] In traditional monitoring, engineers generally detect log anomalies by manually setting keywords and regular expressions based on domain knowledge. However, with the increasing scale and complexity of software services, the manual rule method has huge challenges, including the following reasons. 1) Large-scale software services contain multiple modules (such as hardware, virtual machines, databases, networks, etc.), which will generate massive and diverse logs, with about TB-level logs per day. Therefore, manual rule setting is time-consuming and labor-intensive; 2) Setting rules requires rich domain knowledge, but engineers with sufficient expert knowledge are preferred, and different engineers have different preferences when setting rules, so it is difficult to have a unified standard; 3) Software systems will face frequent software changes, resulting in the need for continuous maintenance and updating of these rules. In summary, detecting log anomalies through manual rule methods is time-consuming and error-prone.

[0004] In order to solve the pain points of manual rules, the academic community has proposed a lot of work on log anomaly detection. These works can be divided into five parts:

[0005] (1) Log collection. Log collection is usually the first step in subsequent log analysis.

[0006] (2) Template parsing. Logs are usually unstructured texts. Generally, template parsing is required to convert the unstructured original logs into structured information, including templates and variables.

[0007] (3) Log grouping. Since log data is context-dependent, it is difficult to determine whether a log is abnormal based on a single log. Therefore, logs need to be grouped based on time windows. Common strategies include sliding windows.

[0008] (4) Feature extraction. Existing work has designed a variety of methods to extract features from windows and perform anomaly detection based on these features, including the number of templates, template order, log semantic features, variable values, etc.

[0009] (5) Anomaly detection. The purpose of anomaly detection is to detect anomalies from the feature vector obtained in the previous step. There are two main categories of anomaly detection: 1) traditional statistical methods, such as PCA, clustering algorithms, etc.; 2) deep learning algorithms, such as the LSTM-based method that has been popular in recent years.

[0010] Although the existing work has been verified to be effective on public log datasets, we found that the results were not ideal when these algorithms were deployed in actual production environments. There are mainly the following challenges.

[0011] Diverse log types and log exception patterns. Every component in a software service generates logs during operation, from the hardware layer to the middleware to the application layer. Different types of logs will show different exception patterns. However, existing work has only verified the effectiveness on a few public data sets with a single type and simple exception patterns, and cannot cope with the complex and diverse log types and exception patterns in the actual production environment. Common exception patterns include keywords, number of templates, template order, variable values, variable distribution, time intervals, total log volume, etc.

[0012] Poor interpretability. Existing work has proposed a black box algorithm that directly gives the judgment results of anomaly detection, but it is difficult for engineers to understand the results given by the algorithm, and they do not understand why the current log is abnormal and what the normal mode should look like.

[0013] Lack of domain knowledge. Existing work provides anomaly detection results end-to-end, but in the face of complex situations in actual production, in order to ensure the accuracy and explainability of detection, it is necessary to integrate domain knowledge, such as which log variables are more important to be detected, what abnormal patterns will this log show, and what type of anomaly detection algorithm is suitable. All of these require domain knowledge. Summary of the invention

[0014] In view of the defects in the above-mentioned prior art, the present invention proposes to adopt appropriate anomaly detection algorithms for different log types. Specifically, for template quantity anomaly, the present invention proposes a template quantity anomaly detection algorithm based on multi-time series anomaly detection; for template sequence anomaly, the present invention adopts a template sequence anomaly detection algorithm based on LSTM; for log variable anomaly, the present invention proposes variable value anomaly detection based on time series anomaly detection and variable distribution anomaly detection algorithm based on distribution distance.

[0015] The present invention provides a log anomaly detection method based on knowledge base and integrated learning, comprising the following steps:

[0016] Get logs;

[0017] identifying the type of the log;

[0018] Based on the knowledge base, preprocessing the logs differently according to different log types;

[0019] If the log is a first type log, a preset template and / or preset rules are used for parsing;

[0020] If the log is a second type log, processing is performed based on ensemble learning;

[0021] Based on the log type, the preprocessed logs are matched with the corresponding log anomaly detection model;

[0022] Output the detection results.

[0023] The first type of logs include general logs, logs that are not suitable for parsing using templates, and logs with key variables;

[0024] The second type of log includes a log of an unknown type.

[0025] The log preprocessing includes processing the unstructured information of the log to extract the structured information.

[0026] If the log is a second type log, processing is performed based on ensemble learning, including:

[0027] Extract key features of logs of unknown types;

[0028] Based on the key features, a random forest is used as a classifier to generate a matching log anomaly detection model for the second type of log.

[0029] The key features include the number of templates, the number of content words in the templates, the distribution of the number of times the templates appear, and the similarity between the templates.

[0030] The log anomaly detection model includes: a template quantity anomaly detection model, a template sequence anomaly detection model, a log variable anomaly detection model and a keyword anomaly detection model.

[0031] Wherein, the template quantity anomaly detection model adopts an anomaly detection algorithm based on multiple time series;

[0032] The template sequence anomaly detection model adopts a template sequence anomaly detection algorithm based on LSTM;

[0033] The log variable anomaly detection model adopts a variable value anomaly detection algorithm based on time series anomaly detection and a variable distribution anomaly detection algorithm based on distribution distance;

[0034] The keyword anomaly detection model adopts a regular expression matching algorithm.

[0035] Among them, a variety of different anomaly detection models are used to detect the same log.

[0036] The method outputs the detection results including: log context information, historical rules of template quantity, workflow diagram of template sequence and temporal rules of variables.

[0037] The present invention also proposes a log anomaly detection system based on knowledge base and integrated learning, which includes:

[0038] An acquisition unit, which is used to acquire logs;

[0039] A type identification unit, used to identify the type of the log;

[0040] A knowledge base, which includes a log preprocessing unit and a model matching unit;

[0041] The log preprocessing unit is used to preprocess the log and convert the unstructured log text into structured information for subsequent anomaly detection;

[0042] A model matching unit, which is used to match the preprocessed logs with corresponding log anomaly detection models based on ensemble learning;

[0043] The output unit is used to output the detection result.

[0044] Compared with the prior art, the present invention focuses on some significant challenges faced by log anomaly detection algorithms in actual production environments, and proposes a log anomaly detection system based on knowledge base and ensemble learning to ensure accuracy and interpretability; at the same time, the present invention designs and implements a suitable anomaly detection algorithm for each log anomaly type, and achieves good results in actual log data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0046] Figure 1 is a flow chart showing a log anomaly detection method based on a knowledge base and integrated learning according to an embodiment of the present invention; and

[0047] Figure 2 The diagram is a schematic diagram showing a log anomaly detection system based on a knowledge base and integrated learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0050] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present invention, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present invention, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0051] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0052] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0053] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0054] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0055] Embodiment 1

[0056] like Figure 1 As shown, the present invention discloses a log anomaly detection method based on knowledge base and integrated learning, comprising the following steps:

[0057] Get logs;

[0058] identifying the type of the log;

[0059] Based on the knowledge base, preprocessing the logs differently according to different log types;

[0060] If the log is a first type log, a preset template and / or preset rules are used for parsing;

[0061] If the log is a second type log, processing is performed based on ensemble learning;

[0062] Based on the log type, the preprocessed logs are matched with the corresponding log anomaly detection model;

[0063] Output the detection results.

[0064] Embodiment 2

[0065] The present invention proposes a log anomaly detection method based on knowledge base and ensemble learning, comprising the following steps:

[0066] Get logs;

[0067] identifying the type of the log;

[0068] Based on the knowledge base, preprocessing the logs differently according to different log types;

[0069] If the log is a first type log, a preset template and / or preset rules are used for parsing;

[0070] If the log is a second type log, the log is processed using a preset algorithm based on ensemble learning;

[0071] Based on the log type, the preprocessed logs are matched with the corresponding log anomaly detection model;

[0072] Output the detection results.

[0073] When accessing logs for anomaly detection, the log type is usually known. For example, if you want to access Oracle logs for anomaly detection, then Oracle is a priori knowledge.

[0074] The first type of logs include general logs, logs that are not suitable for parsing using templates, and logs with key variables;

[0075] The second type of log includes a log of an unknown type.

[0076] Among them, some key variables (such as heap usage in JVM GC logs) can be extracted through the knowledge base and input into the subsequent anomaly detection process.

[0077] The log preprocessing includes processing the unstructured information of the log to extract the structured information.

[0078] If the log is a second type log, processing is performed based on ensemble learning, including:

[0079] Extract key features of logs of unknown types;

[0080] Based on the key features, a random forest is used as a classifier to generate a matching log anomaly detection model for the second type of log.

[0081] Here, because the log contains rich information, it will present a variety of log anomaly patterns, and different logs are suitable for different anomaly detection algorithms. For JVM GC logs, the template quantity anomaly detection model, keyword anomaly detection model, and variable value anomaly detection model are used; for DB2 logs, the template quantity anomaly detection model and keyword anomaly detection model are used; for HDFS logs, the template sequence anomaly detection model is used.

[0082] The key features include the number of templates, the number of content words in the templates, the distribution of the number of times the templates appear, and the similarity between the templates.

[0083] That is to say, for newly appeared logs, based on the existing data, a supervised classification model can be built to automatically assign a suitable algorithm to them. Intuitively, if the pattern of the unknown log is similar to the log (labeled data) that has been configured in the knowledge base, then their abnormal pattern and suitable algorithm should also be the same. For example, we extract the following features for unknown logs: the number of templates, the number of content words in the template, the distribution of the number of times the template appears, and the similarity between templates (using the jaccard distance). Based on these features, a random forest is used as a classifier to automatically select and assign a suitable algorithm to the new log. Of course, if the detection effect of the automatically assigned algorithm is not ideal, you can also manually select a suitable algorithm for it and add this knowledge to the knowledge base.

[0084] The log anomaly detection model includes: a template quantity anomaly detection model, a template sequence anomaly detection model, a log variable anomaly detection model and a keyword anomaly detection model.

[0085] Wherein, the template quantity anomaly detection model adopts an anomaly detection algorithm based on multiple time series;

[0086] The template sequence anomaly detection model adopts a template sequence anomaly detection algorithm based on LSTM;

[0087] The log variable anomaly detection model adopts a variable value anomaly detection algorithm based on time series anomaly detection and a variable distribution anomaly detection algorithm based on distribution distance;

[0088] The keyword anomaly detection model adopts a regular expression matching algorithm.

[0089] Among them, a variety of different anomaly detection models are used to detect the same log.

[0090] The method outputs the detection results including: log context information, historical rules of template quantity, workflow diagram of template sequence and temporal rules of variables.

[0091] In summary, using ensemble learning, for different data types in logs, it is necessary to configure appropriate anomaly detection algorithms in the system. The data types contained in different logs are different. Therefore, it is necessary to select the appropriate anomaly detection algorithm for each log:

[0092] (1) Keywords. Although the log keyword strategy is relatively simple, it is also the most direct way to discover faults. Usually, setting rare, specific, and more fatal keywords (such as "OutOfMemory" and "IOException") is more conducive to the accuracy of anomaly detection. The knowledge base can maintain some common keywords of common logs.

[0093] (2) Number of templates. Considering the interpretability of the algorithm, we use time series to characterize the number of templates. For historical data, the number of times each type of template appears per minute is counted as a time series. The anomaly detection of the number of templates can be converted into an anomaly detection problem of time series. Usually, a log has dozens to thousands of templates. In order to save system resources and reduce the number of invalid alarms, we treat all template sequences as a whole and use the LSTM-based multi-time series anomaly detection algorithm to detect the number of templates. The difference between the algorithm's predicted value and the actual value is used as the anomaly score.

[0094] (3) Template order. The order of templates usually describes the execution order of tasks. Once the task execution order violates the normal pattern, it is considered abnormal. In this invention, we use the LSTM algorithm to learn the normal template order. At the same time, in order to ensure interpretability, a finite state machine is constructed to clearly represent the workflow of task execution. In detail, the input of the LSTM algorithm is , where x i Indicates the template number corresponding to the i-th log, and the output is , that is, given the template of the previous w logs, predict what the next log will be. The algorithm will give the prediction result of the current arrival template, which is generally a set of lists And the corresponding probability, once the arriving log template is not within the prediction range of top-k, it is considered to be an anomaly.

[0095] (4) Variable values. The values ​​of variables can generally be aggregated into a time series. In the present invention, the autocorrelation system is first used to determine whether the variable time series has a period. If so, Holt-winters is used to detect anomalies; if not, 3-sigma is used to detect anomalies.

[0096] (5) Variable distribution. Some variables are not numerical continuous variables, but categorical variables (such as IP addresses, return codes). For such variables, changes in their distribution are usually considered. Generally speaking, the distribution of categories conforms to a normal pattern. Once the distribution changes, it may mean that there is a problem with the system. For example, if the proportion of a certain IP address increases, it may be that this IP address is trying to access it. In this invention, the problem of log variable distribution anomaly detection is proposed, and the JS divergence (Jensen-Shannon divergence) is used to characterize the distribution of variables. Once the distribution of the current window is significantly different from the normal time (the distance exceeds the threshold), it is considered to be abnormal.

[0097] Embodiment 3

[0098] In a certain embodiment of the present invention, at least three types of information are stored in the knowledge base.

[0099] (1) Templates corresponding to common logs: Taking the HDFS template as an example, when the system needs to access new HDFS logs in the future, the template can be directly read from the knowledge base, and there is no need to run the template extraction algorithm and manual tuning.

[0100] (2) Log parsing rules: For some special logs, regular parsing rules are stored in the knowledge base. For example, accesslog can be parsed into [time, IP, URL, return code] according to the regular expression.

[0101] 111.43.122.194 - - [20 / Nov / 2021:00:00:00 +0800] "GET / __local / 2 / A3 / 01 / 98E7F968084669CD9F7A4BB8304_FEC00CC4_D5AD5.jpg HTTP / 1.1" 200 875221 "https: / / www.tsinghua.edu.cn / " "Mozilla / 5.0 (Linux; U; Android 9; zh-cn; Redmi8A Build / PKQ1.190319.001) AppleWebKit / 537.36 (KHTML, like Gecko) Version / 4.0Chrome / 89.0.4389.116 Mobile Safari / 537.36 Xiaomi / MiuiBrowser / 15.5.12" After parsing, we can get

[0102] Time = 20 / Nov / 2021:00:00:00

[0103] IP=111.43.122.194

[0104] URL="https: / / www.tsinghua.edu.cn /

[0105] Return code = 200.

[0106] (3) Anomaly detection algorithm corresponding to the log type.

[0107] Embodiment 4:

[0108] like Figure 2 As shown, the present invention also proposes a log anomaly detection system based on knowledge base and integrated learning, which includes:

[0109] An acquisition unit, which is used to acquire logs;

[0110] A type identification unit, used to identify the type of the log;

[0111] A knowledge base, which includes a log preprocessing unit and a model matching unit;

[0112] The log preprocessing unit is used to preprocess the log and convert the unstructured log text into structured information for subsequent anomaly detection;

[0113] A model matching unit, which is used to match the preprocessed logs with corresponding log anomaly detection models using a preset algorithm based on ensemble learning;

[0114] The output unit is used to output the detection result.

[0115] Embodiment 5

[0116] An embodiment of the present disclosure provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the method steps described in the above embodiment.

[0117] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0118] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0119] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (AN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0120] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0122] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A log anomaly detection method based on knowledge base and ensemble learning, comprising the following steps: Get logs; identifying the type of the log; Based on the knowledge base, preprocessing the logs differently according to different log types; The knowledge base stores at least the following information: templates corresponding to common logs, log parsing rules, and anomaly detection algorithms corresponding to log types; Based on the log type, the preprocessed logs are matched with the corresponding log anomaly detection model; Output the test results; Wherein, if the log is a first type log, a preset template and / or preset rule is read from the knowledge base for parsing; If the log is a second type log, processing is performed based on ensemble learning, which includes: Extracting key features of logs of unknown types; the key features include the number of templates, the number of content words in the templates, the distribution of the number of times the templates appear, and the similarity between the templates; Based on the key features, a random forest is used as a classifier to generate a matching log anomaly detection model for the second type of log; The first type of logs include general logs, logs that are not suitable for parsing using templates, and logs with key variables; The second type of logs includes logs of unknown types.

2. The method as claimed in claim 1, wherein the log preprocessing comprises processing unstructured information of the log to extract structured information.

3. The method according to claim 1, wherein the log anomaly detection model comprises: Template quantity anomaly detection model, template sequence anomaly detection model, log variable anomaly detection model and keyword anomaly detection model.

4. The method as claimed in claim 3, wherein the template quantity anomaly detection model adopts an anomaly detection algorithm based on multiple time series; The template sequence anomaly detection model adopts a template sequence anomaly detection algorithm based on LSTM; The log variable anomaly detection model adopts a variable value anomaly detection algorithm based on time series anomaly detection and a variable distribution anomaly detection algorithm based on distribution distance; The keyword anomaly detection model adopts a regular expression matching algorithm.

5. The method as claimed in claim 3, wherein a plurality of different anomaly detection models are used to detect the same log.

6. The method of claim 1, wherein the method outputting the detection result comprises: Log context information, historical patterns of template quantity, workflow diagrams of template order, and temporal patterns of variables.

Images