An automated log anomaly detection method

Abstract

The present invention relates to an automated log anomaly detection method, comprising: S1, obtaining a log data set; the log data set comprising at least one log data item; S2, parsing each log data item in the log data set to obtain a log template and log parameters corresponding to the log data item; and S3, performing anomaly detection based on the log template and / or log parameters of each log data item in the log data set to obtain a detection result. The automated log anomaly detection method of the present invention improves the accuracy of identifying log anomalies compared to existing technologies because anomaly detection is performed based on the log template and / or log parameters corresponding to the log data item, and anomaly detection can be performed in different ways.

Description

Technical Field

[0001] The present invention relates to the technical field of network data security defense, and in particular to an automated log anomaly detection method. Background Art

[0002] The primary purpose of system logs is to record system status and important events at key points to aid in debugging system failures and performing root cause analysis. This log data is ubiquitous in nearly all computer systems. Log data is an important and valuable resource for understanding system status and performance issues; therefore, various system logs are a natural source of information for online monitoring and anomaly detection. Therefore, anomaly detection in system logs is a critical step in building a secure and reliable system. However, existing techniques for anomaly detection in logs are often inaccurate. Summary of the Invention

[0003] (1) Technical issues to be solved

[0004] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an automated log anomaly detection method, which solves the technical problem that log anomaly detection in the prior art is often not accurate enough.

[0005] (2) Technical solution

[0006] In order to achieve the above objectives, the main technical solutions adopted by the present invention include:

[0007] In a first aspect, an embodiment of the present invention provides an automated log anomaly detection method, characterized by comprising:

[0008] S1. Obtain log data set;

[0009] The log data set includes at least one log data;

[0010] S2. parse each log data in the log data set to obtain a log template and log parameters corresponding to the log data;

[0011] S3. Perform anomaly detection based on the log template and / or log parameters of each log data in the log data set to obtain a detection result.

[0012] Preferably, the S2 specifically includes:

[0013] A preset drain algorithm is used to parse each log data in the log data set to obtain a log template and log parameters corresponding to the log data.

[0014] Preferably,

[0015] The log template is an immutable text in the log data;

[0016] The log parameter is a variable text in the log data.

[0017] Preferably, the S3 specifically includes:

[0018] Using a plurality of pre-set regular expressions to perform regular filtering and identification detection on each log data in the log data set, and obtaining a first detection result;

[0019] Wherein, any of the preset regular expressions is formed by combining any preset first parameter and any type of log template in the log data set;

[0020] The log templates of the same type are exactly the same.

[0021] Preferably,

[0022] The log data set is a collection of log data collected within a first preset time period.

[0023] Preferably, before S3, the method further includes:

[0024] Obtain a first training log data set; wherein the first training log data set includes at least one training log data; the training log data is normal log data;

[0025] Obtain log templates of training log data within the same multiple time windows in the first training log data set, perform log template type merging processing on the training log data within the time windows, and generate first and second features of each type of log template in each time window;

[0026] The first feature of each type of log template includes the frequency of occurrence of the training log data of the log template of this type in the corresponding time window;

[0027] The second feature is the frequency at which the training log data of any two types of log templates appear in adjacent order in the corresponding time window;

[0028] An isolation forest model is trained using a time window having the first feature and the second feature of each type of log template to obtain a trained isolation forest model.

[0029] Preferably, the S3 specifically includes:

[0030] Obtain log templates of log data in the same multiple time windows in the log data set, perform log template type merging processing on the log data in the time windows, and generate the first feature and second feature of each type of log template in each time window;

[0031] The first feature of each type of log template includes the frequency of occurrence of log data of the log template of this type in the corresponding time window;

[0032] The second feature is the frequency at which log data of any two types of log templates appear in adjacent order in the corresponding time window;

[0033] Use the trained isolation forest model to perform anomaly detection on each time window to obtain the second detection result;

[0034] The second detection result is the first anomaly score corresponding to each time window;

[0035] It is determined whether the first abnormality score is less than a first preset value. If so, the time window corresponding to the first abnormality score is determined to be abnormal.

[0036] Preferably, before S3, the method further includes:

[0037] Obtaining a second training log data set; wherein the second training log data set includes at least one training log data; the training log data is normal log data;

[0038] Merging identical log templates in the second training log data set to obtain a first training log template set corresponding to the second training log data set;

[0039] The first training log template set includes multiple different types of log templates;

[0040] Sorting the preset codes of the log modules corresponding to the training log data in the second training log data set in sequence according to the time sequence of the training log data in the second training log data set to generate a training log template code sequence;

[0041] Using the training log template encoding sequence to train an LSTM deep learning model, and obtaining a trained LSTM deep learning model;

[0042] The top_k evaluation index of the trained LSTM deep learning model is higher than 95%.

[0043] Preferably, the S3 specifically includes:

[0044] Merging identical log templates in the log data set to obtain a first log template set corresponding to the log data set;

[0045] The first log template set includes multiple different types of log templates;

[0046] Sorting the preset codes of the log modules corresponding to the logs in the log data set in sequence according to the time sequence of the log data in the log data set to generate a log template code sequence;

[0047] Processing the log template encoding sequence using the trained LSTM deep learning model to obtain a third detection result;

[0048] The third detection result includes an output value corresponding to each log template code in the log template code sequence;

[0049] It is determined whether the output value corresponding to each log template code is less than a preset value.

[0050] Preferably, the S3 specifically includes:

[0051] Merging identical log templates in the log data set to obtain a first log template set corresponding to the log data set;

[0052] The first log template set includes multiple different types of log templates;

[0053] Obtain log parameters corresponding to each type of log template in the first log template set;

[0054] Using the trained OneClass anomaly detection model, perform anomaly detection on the log parameters corresponding to each type of log template in the first log template set to obtain a fourth detection result;

[0055] The fourth detection result includes a second abnormality score of the log parameter corresponding to each type of log template;

[0056] It is determined whether the second abnormality score is less than a second preset value. If so, the log data to which the log parameter corresponding to the second abnormality score belongs is determined to be abnormal.

[0057] (3) Beneficial effects

[0058] The beneficial effects of the present invention are as follows: an automated log anomaly detection method of the present invention performs anomaly detection based on a log template and / or log parameters corresponding to the log data, and at the same time, anomaly detection can be performed in different ways. Compared with the prior art, it can improve the accuracy of identifying log anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of an automated log anomaly detection method of the present invention;

[0060] Figure 2Schematic diagram of a log data set obtained in an embodiment of the present invention;

[0061] Figure 3 Schematic diagram of a log template in a log data collection process according to an embodiment of the present invention;

[0062] Figure 4 Schematic diagram of log parameters in log data collection according to an embodiment of the present invention;

[0063] Figure 5 A schematic diagram of a log template for log data within a time window counted in an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of the number of occurrences of different log templates within a time window counted in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0066] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Instead, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0067] See also Figure 1 、 one Automated log anomaly detection methods, including:

[0068] S1. Get the log data set, such as Figure 2 shown.

[0069] The log data set includes at least one piece of log data.

[0070] In this embodiment, the system's log data contains messages issued by several modules within the system and valuable system status information, such as device status and error conditions, and information about various tasks within the system, such as project name, execution path, including function name and parameters, and task completion status. For customers with remote support, the system collects this log data and transmits it to a central enterprise repository, where it is monitored for alerts, problem prediction, and troubleshooting.

[0071] The log data in this embodiment is simplified HDFS raw log data collected on the Amazon EC2 platform; the raw log data is unstructured data, including timestamps and original message content.

[0072] S2. Analyze each log data in the log data set to obtain a log template corresponding to the log data. Figure 2 As shown, and the log parameters are as Figure 3 shown.

[0073] S3. Perform anomaly detection based on the log template and / or log parameters of each log data in the log data set to obtain a detection result.

[0074] In the practical application of this embodiment, S2 specifically includes:

[0075] A preset drain algorithm is used to parse each log data in the log data set to obtain a log template and log parameters corresponding to the log data.

[0076] In practical applications of this embodiment, the log template is an immutable text in the log data.

[0077] The log parameter is a variable text in the log data.

[0078] Specifically, a distinction is made between a fixed portion and a variable portion of each raw log data.

[0079] The fixed part is usually unchanging text. It is a token that describes the system operation template (i.e. log event), for example Figure 2 "Receive block *src: *dest: *" in the command.

[0080] Variable text is the rest of the tokens that carry dynamic runtime system information (e.g., "blk 3587"). A typical structured log message contains the matching log event and the fields of interest (e.g., the HDFS block ID "blk 3587").

[0081] In the practical application of this embodiment, S3 specifically includes:

[0082] A plurality of pre-set regular expressions are used to perform regular filtering and identification detection on each log data in the log data set to obtain a first detection result.

[0083] Any of the preset regular expressions is formed by combining any preset first parameter and any type of log template in the log data set.

[0084] The log templates of the same type are exactly the same.

[0085] In practical applications of this embodiment, the log data set is a collection of log data collected within a first preset time period.

[0086] In practical applications of this embodiment, the following steps are also included before S3:

[0087] A first training log data set is obtained; wherein the first training log data set includes at least one piece of training log data; and the training log data is normal log data.

[0088] Obtain log templates of training log data in the same multiple time windows in the first training log data set, perform log template type merging processing on the training log data in the time windows, and generate first and second features of each type of log template in each time window.

[0089] The first feature of each type of log template includes the frequency at which the training log data of the log template of this type appears in the corresponding time window.

[0090] The second feature is the frequency at which the training log data of any two types of log templates appear in adjacent order in the corresponding time window.

[0091] An isolation forest model is trained using a time window having the first feature and the second feature of each type of log template to obtain a trained isolation forest model.

[0092] In the practical application of this embodiment, S3 specifically includes:

[0093] Get the log template of the log data in the same multiple time windows in the log data set, such as Figure 5 shown.

[0094] The log data in the time window is processed by log template type merging, and the first feature and second feature of each type of log template in each time window are generated, such as Figure 6 shown.

[0095] The first feature of each type of log template includes the frequency of occurrence of log data of the log template of this type in the corresponding time window;

[0096] The second feature is the frequency at which log data of any two types of log templates appear in adjacent order in the corresponding time window;

[0097] Use the trained isolation forest model to perform anomaly detection on each time window to obtain the second detection result;

[0098] The second detection result is the first anomaly score corresponding to each time window;

[0099] It is determined whether the first abnormality score is less than a first preset value. If so, the time window corresponding to the first abnormality score is determined to be abnormal.

[0100] In practical applications of this embodiment, the following steps are also included before S3:

[0101] A second training log data set is obtained; wherein the second training log data set includes at least one piece of training log data; and the training log data is normal log data.

[0102] The same log templates in the second training log data set are merged to obtain a first training log template set corresponding to the second training log data set.

[0103] The first training log template set includes multiple different types of log templates.

[0104] The preset codes of the log modules corresponding to the training log data in the second training log data set are sequentially sorted according to the time sequence of the training log data in the second training log data set to generate a training log template code sequence.

[0105] The training log template encoding sequence is used to train an LSTM deep learning model to obtain a trained LSTM deep learning model. Specifically, five encodings {K1, K2, K3, K4, K5} in the encoding sequence are used as X, a encoding K6 after the above five encodings is used as a prediction target, and top_k is used as an evaluation indicator to evaluate the LSTM deep learning model.

[0106] The top_k evaluation index of the trained LSTM deep learning model is higher than 95%.

[0107] In the practical application of this embodiment, S3 specifically includes:

[0108] The same log templates in the log data set are merged to obtain a first log template set corresponding to the log data set.

[0109] The first log template set includes multiple different types of log templates.

[0110] The preset codes of the log modules corresponding to the logs in the log data set are sorted in sequence according to the time sequence of the log data in the log data set to generate a log template code sequence.

[0111] The trained LSTM deep learning model is used to process the log template encoding sequence to obtain a third detection result.

[0112] The third detection result includes an output value corresponding to each log template code in the log template code sequence.

[0113] It is determined whether the output value corresponding to each log template code is less than a preset value.

[0114] In the practical application of this embodiment, S3 specifically includes:

[0115] The same log templates in the log data set are merged to obtain a first log template set corresponding to the log data set.

[0116] The first log template set includes multiple different types of log templates.

[0117] Obtain log parameters corresponding to each type of log template in the first log template set.

[0118] The trained OneClass anomaly detection model is used to perform anomaly detection on the log parameters corresponding to each type of log template in the first log template set to obtain a fourth detection result.

[0119] In practice, the log data corresponding to the same log template type describes the same event, but different parameters are used to record system status. For example, if the average access time for the same domain name on a website is 1 second, but a request takes 10 seconds to return, this indicates an anomaly. Alternatively, if a website normally returns a 200 status code but returns a different status code, this indicates an anomaly.

[0120] The fourth detection result includes a second abnormality score of the log parameter corresponding to each type of log template;

[0121] It is determined whether the second abnormality score is less than a second preset value. If so, the log data to which the log parameter corresponding to the second abnormality score belongs is determined to be abnormal.

[0122] An automated log anomaly detection method of the present invention performs anomaly detection based on a log template and / or log parameters corresponding to log data. At the same time, anomaly detection can be performed in different ways. Compared with the prior art, it can improve the accuracy of identifying log anomalies.

[0123] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions.

[0125] It should be noted that, in the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims enumerating several means, several of these means may be embodied by one and the same hardware. The use of the words first, second, third etc. is for convenience only and does not indicate any order. These words may be understood as part of the component name.

[0126] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0127] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments after learning the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0128] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the claims and their equivalents, the present invention shall also include such modifications and variations.

Claims

1. An automated log anomaly detection method, characterized in that: include: S1. Obtain log data set; The log data set includes at least one log data; S2. parse each log data in the log data set to obtain a log template and log parameters corresponding to the log data; S3. Perform anomaly detection based on the log template and / or log parameters of each log data in the log data set to obtain a detection result; The S2 specifically includes: Using a preset drain algorithm to parse each log data in the log data set to obtain a log template and log parameters corresponding to the log data; The log template is an immutable text in the log data; The log parameter is a variable text in the log data; The log data set is a collection of log data collected within a first preset time period; Before the S3, it also includes: Obtain a first training log data set; wherein the first training log data set includes at least one training log data; the training log data is normal log data; Obtain log templates of training log data within the same multiple time windows in the first training log data set, perform log template type merging processing on the training log data within the time windows, and generate first and second features of each type of log template in each time window; The first feature of each type of log template includes the frequency of occurrence of the training log data of the log template of this type in the corresponding time window; The second feature is the frequency at which the training log data of any two types of log templates appear in adjacent order in the corresponding time window; An isolation forest model is trained using a time window having the first feature and the second feature of each type of log template to obtain a trained isolation forest model.

2. The method according to claim 1, characterized in that The S3 specifically includes: Using a plurality of pre-set regular expressions to perform regular filtering and identification detection on each log data in the log data set, and obtaining a first detection result; Wherein, any of the preset regular expressions is formed by combining any preset first parameter and any type of log template in the log data set; The log templates of the same type are exactly the same.

3. The method according to claim 1, characterized in that The S3 specifically includes: Obtain log templates of log data in the same multiple time windows in the log data set, perform log template type merging processing on the log data in the time windows, and generate the first feature and second feature of each type of log template in each time window; The first feature of each type of log template includes the frequency of occurrence of log data of the log template of this type in the corresponding time window; The second feature is the frequency at which log data of any two types of log templates appear in adjacent order in the corresponding time window; Use the trained isolation forest model to perform anomaly detection on each time window to obtain the second detection result; The second detection result is the first anomaly score corresponding to each time window; It is determined whether the first abnormality score is less than a first preset value. If so, the time window corresponding to the first abnormality score is determined to be abnormal.

4. The method according to claim 1, wherein Before the S3, it also includes: Obtaining a second training log data set; wherein the second training log data set includes at least one training log data; the training log data is normal log data; Merging identical log templates in the second training log data set to obtain a first training log template set corresponding to the second training log data set; The first training log template set includes multiple different types of log templates; Sorting the preset codes of the log modules corresponding to the training log data in the second training log data set in sequence according to the time sequence of the training log data in the second training log data set to generate a training log template code sequence; Using the training log template encoding sequence to train an LSTM deep learning model, and obtaining a trained LSTM deep learning model; The top_k evaluation index of the trained LSTM deep learning model is higher than 95%.

5. The method according to claim 4, characterized in that The S3 specifically includes: Merging identical log templates in the log data set to obtain a first log template set corresponding to the log data set; The first log template set includes multiple different types of log templates; Sorting the preset codes of the log modules corresponding to the logs in the log data set in sequence according to the time sequence of the log data in the log data set to generate a log template code sequence; Processing the log template encoding sequence using the trained LSTM deep learning model to obtain a third detection result; The third detection result includes an output value corresponding to each log template code in the log template code sequence; It is determined whether the output value corresponding to each log template code is less than a preset value.

6. The method according to claim 1, characterized in that The S3 specifically includes: Merging identical log templates in the log data set to obtain a first log template set corresponding to the log data set; The first log template set includes multiple different types of log templates; Obtain log parameters corresponding to each type of log template in the first log template set; Using the trained OneClass anomaly detection model, perform anomaly detection on the log parameters corresponding to each type of log template in the first log template set to obtain a fourth detection result; The fourth detection result includes a second abnormality score of the log parameter corresponding to each type of log template; It is determined whether the second abnormality score is less than a second preset value. If so, the log data to which the log parameter corresponding to the second abnormality score belongs is determined to be abnormal.

Images