An unsupervised log anomaly detection method that does not rely on log parsers

Abstract

The present application discloses an unsupervised log anomaly detection method that does not rely on a log parser, wherein the method comprises three steps: a log collection step, in which the incremental logs generated by a computer system are efficiently collected and transmitted using a log collection tool Flume and a streaming transmission tool Kafka, thereby ensuring the efficiency of log data collection and transmission; a log preprocessing step, in which regular replacement and tokenization are used to enrich the semantic information of the collected logs and clean up invalid information, thereby effectively improving the data quality; a feature extraction and anomaly detection step, in which the BERT model is used to extract semantic features from log data, and the masked language model MLM is used for training, thereby encoding prior knowledge to capture the pattern of normal log sequences, thereby achieving unsupervised log anomaly detection.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an unsupervised log anomaly detection method that does not rely on a log parser. Background Art

[0002] Modern software systems often need to ensure uninterrupted service 24 / 7, and even the slightest failure can result in significant losses. Therefore, timely detection of abnormal events is crucial for ensuring stable system operation and enabling rapid fault diagnosis. Software systems often generate logs. Logs are text data used to assist system developers in debugging programs. Generated by embedded printout code within the program, logs record the program's operating status and important events at key points. Logs record valuable information in real time and are extremely important for anomaly detection. Log anomaly detection generally involves identifying patterns in log data that deviate from expected behavior. This is crucial for ensuring stable system operation and enabling rapid fault diagnosis.

[0003] Log anomaly detection has been widely studied in recent years. Traditional rule-based methods, while highly accurate and fast, can only identify predefined anomaly scenarios. As logs change, they require significant human input to provide domain knowledge, making them impractical in real-world scenarios. To reduce human intervention and improve the efficiency of anomaly detection, methods based on traditional machine learning, such as principal component analysis, have been proposed. While these methods can achieve automated anomaly detection, they also suffer from some issues, such as poor extraction of high-level and global features, particularly in the semantic representation and long-range dependencies of log text. To address these issues, current research focuses on methods based on deep learning, which excel at extracting high-level and global features and possesses strong expressive power.

[0004] However, previous research on deep learning-based log anomaly detection has mostly used log parsers to parse logs into templates, which are then used for anomaly detection. However, logs are unstable. For example, log formats change frequently during the software lifecycle. Using log parsers can lead to the loss of key information in the logs during the parsing process, affecting anomaly detection effectiveness. Furthermore, with the development of computer technology, computer systems have become large-scale and distributed, generating massive amounts of log data that are difficult to manually label, hindering the development of supervised learning-based log anomaly detection models.

[0005] Therefore, it is necessary to design an unsupervised log anomaly detection method that does not rely on log parsers to solve the technical difficulties in labeling massive logs in real scenarios and the possibility that log parsers may lose key information. Summary of the Invention

[0006] In response to the deficiencies of the prior art, the present invention proposes an unsupervised log anomaly detection method that does not rely on a log parser. The solution adopts a log preprocessing method based on regular matching and tokenization. The original log is processed by regular matching and tokenization, enriching the semantic information in the log and effectively filtering out invalid information in the log, thereby effectively improving the quality of the log. More importantly, the BERT model is introduced, and an unsupervised log anomaly detection model based on a pre-trained language model is proposed, which can simultaneously realize the extraction of log semantic features and the detection of anomalies. The extraction of semantic features greatly improves the robustness of the model and can adapt to the instability of logs to a certain extent. In addition, the model does not require labeled log data for training, which greatly improves the feasibility of the model in real-life scenarios with massive unlabeled logs.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] An unsupervised log anomaly detection method that does not rely on a log parser includes the following steps:

[0009] S1. Log Collection: Use the log collection tool Flume and the streaming tool Kafka to efficiently collect and transmit the incremental logs generated by the computer system to ensure the efficiency of log data collection and transmission;

[0010] S2. Log preprocessing: Use regular expression replacement and tokenization to enrich semantic information and clean invalid information in the collected logs, effectively improving data quality.

[0011] S3. Feature extraction and anomaly detection: The BERT model is used to extract semantic features from log data, and the masked language model (MLM) is used for training to encode prior knowledge and capture the patterns of normal log sequences, thereby achieving unsupervised log anomaly detection.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] 1. This paper adopts a log preprocessing method based on regular expression replacement and tokenization instead of using a log parser. This method avoids the possibility of key information loss during log parsing, retains as much key information as possible, enriches the semantic information of the log, and ensures the data quality of log anomaly detection.

[0014] 2. This paper leverages BERT's advantages in semantic representation of log messages to address issues such as polysemy and out-of-vocabulary, obtaining a more meaningful semantic feature representation. This significantly improves the robustness of the model and, to a certain extent, can adapt to the instability of logs.

[0015] 3. To address the problem of massive logs being difficult to label in real-world scenarios, the present invention uses the BERT model to extract semantic features from log data and uses the masked language model (MLM) for training, thereby encoding prior knowledge to capture the patterns of normal log sequences, thereby achieving unsupervised log anomaly detection and improving the accuracy of anomaly detection.

[0016] In summary, the method and device for unsupervised log anomaly detection based on a pre-trained language model, which does not rely on a log parser, not only ensures high accuracy in anomaly detection but also improves the robustness of the model. Furthermore, it is more feasible to apply in real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 This is a general framework diagram of the method and apparatus for unsupervised log anomaly detection based on a pre-trained language model that does not rely on a log parser, as provided in an embodiment of the present invention.

[0019] Figure 2 This is a flowchart of the log collection steps based on the Flume and Kafka architecture provided by an embodiment of the present invention.

[0020] Figure 3 This is a flowchart of log preprocessing steps based on regular expression replacement and tokenization provided by an embodiment of the present invention.

[0021] Figure 4 This is a structural diagram of the BERT-based unsupervised log anomaly detection model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to better understand the present technical solution, the method of the present invention is described in detail below with reference to the accompanying drawings.

[0023] The present invention proposes a method and device for unsupervised log anomaly detection based on a pre-trained language model that does not rely on a log parser. The method and device include three steps: log collection, log preprocessing, and feature extraction and anomaly detection. In log collection, we use the log collection tool Flume and the streaming tool Kafka to efficiently collect and transmit the incremental logs generated by the computer system, ensuring the efficiency of log data collection and transmission. In log preprocessing, we propose a log preprocessing method based on regular substitution and tokenization. Regular substitution and tokenization are used to enrich the semantic information and clean up invalid information in the collected logs, effectively improving data quality. In feature extraction and anomaly detection, we use the BERT model to extract semantic features from log data and train it using the masked language model (MLM). This method encodes prior knowledge to capture the patterns of normal log sequences, thereby achieving unsupervised log anomaly detection. The anomaly detection criterion is whether the hidden pattern of the log sequence to be detected matches the learned pattern. If it matches, it indicates that the log sequence is normal; otherwise, it is abnormal.

[0024] like Figure 1 As shown, it mainly includes the following three steps:

[0025] S1. Log Collection

[0026] Step S1 adopts the Flume+Kafka architecture. Its main task is to use the log collection tool Flume to collect various types of incremental logs generated during the operation of the computer system as comprehensively as possible, and then use the streaming tool Kafka to achieve the safe, efficient and stable transmission of the collected log data. Figure 2 The specific steps are as follows:

[0027] S11. Use the Exec Source component of the Flume agent, a log collection tool, to collect logs and generate events.

[0028] S12. Utilize the Spillable Memory Channel component of the Flume agent, a log collection tool, using memory and files as storage media to temporarily store events generated by the Exec Source. This ensures efficient transmission while minimizing data loss and generating events.

[0029] S13. Use the Kafka Sink component of the Flume agent, a log collection tool, to forward the event generated by the Spillable MemoryChannel, generating a message on a specific Kafka topic that flows into the streaming tool Kafka.

[0030] S14. Use the streaming tool Kafka to distribute the messages of the specific topic flowing into the Kafka Sink and deliver them to step S2.

[0031] S2. Log Preprocessing

[0032] In log preprocessing, this paper proposes a log preprocessing method based on regular matching and tokenization. Figure 3 The specific steps are as follows:

[0033] S21, establish a dictionary of valid words;

[0034] Based on an open-source corpus, we created a valid word dictionary D that covers most words in different log systems to further identify valid words. The specific process is as follows: We first segmented an English corpus containing 5.2 million sentences using spaces as delimiters to collect 578,054 unique words. Then, we set a threshold to filter out 18,335 valid words from these 578,054 unique words. These 18,335 valid words constitute the valid word dictionary D. When the log format in the system changes, business experts can incorporate new domain knowledge, that is, new valid words, as needed.

[0035] S22, regular matching;

[0036] Use pre-set regular expressions to replace common variables to enrich log semantics. For example, the IP address 192.168.0.1 can be matched with the regular expression "^(\d{1,3}\.){3}\d{1,3}$" and replaced with "IP." Numbers can be matched with the regular expression "^[0-9]*$" and replaced with "NUM."

[0037] S23, tokenization;

[0038] Using the tokenization technology WordPiece, words in logs are segmented into tokens. If a token is in the valid word dictionary D, it is considered a valid word; otherwise, it is filtered out as invalid information. For example, "trying" is segmented into two subwords: "try" and "##ing." By breaking words into smaller units, the model can better handle complex word morphology and structure, especially those composed of multiple subwords. This segmentation improves the model's ability to handle unregistered words, rare words, and uncommon variants, thereby enhancing the effectiveness and generalization of anomaly detection models in log anomaly detection tasks.

[0039] S3. Feature extraction and anomaly detection

[0040] In terms of feature extraction and anomaly detection, this paper proposes an unsupervised log anomaly detection model based on BERT. The model structure is as follows Figure 4 As shown. First, the definition of log data is as follows. For example, the log sequence is defined as L = {k1, k2, ..., k i ,...,k len}, where len is the length of the log sequence, that is, the number of log keys in the log sequence, k i The log key is a pre-processed log after step S2. Then the training set consisting of normal log sequences is defined as Among them L j Represents the training set The jth normal log sequence. From the perspective of the model data processing flow, the feature extraction and anomaly detection steps are divided into four sub-steps. The specific steps are as follows:

[0041] S31, log vectorized representation;

[0042] Through operations such as adding special tags, masking operations, word embedding, and position embedding, the text-type log sequence is converted into a vector of numerical type, which is convenient for the model to process. The special tag addition is to add a special token [CLS] as the first log key at the beginning of the log sequence L, representing the beginning of the entire log sequence. The masking operation is based on the special tag addition operation. For the log sequence L with the special token added, 15% of the log keys are randomly hidden, and the hidden log keys will be replaced with a special token [MASK]. Then there are the word embedding (token Embedding) operation and the position embedding (Position Embedding) operation. For the word embedding operation, we used Word2Vec technology to generate each log key k i Vectorized representation of For the position embedding operation, we randomly generated a matrix as the log-key embedding matrix And the position information of the log key in the log sequence is encoded using the sine function to generate the position embedding matrix Where d is the dimension of the log key vector representation. Then calculate the log key vector representation x i , the calculation process is as shown in formula 1. is the position embedding vector for the i-th log key, is the word embedding vector of the i-th log key.

[0043]

[0044] S32, semantic feature extraction;

[0045] The log vector of step S31 is used as input, and multiple Transformer encoders are used to learn the contextual relationship between log keys in the log sequence, thereby extracting bidirectional deep semantic features, which helps improve the robustness of the model. The calculation process is as shown in Formula 2. i is the log key k in the log sequence i The Transformer function represents the processing of multiple Transformer encoders stacked in sequence.

[0046] h i =Transformer(x i ) (2)

[0047] Each Transformer encoder consists of a self-attention layer and a feedforward neural network layer. Both the self-attention layer and the feedforward neural network layer are accompanied by a residual connection and layer normalization. The core of the self-attention layer is the attention mechanism. The main function of the attention mechanism is to enable the neural network to focus on a relatively important portion of the input, thereby achieving interaction with all contexts. Self-attention can further enhance the diversity of the attention mechanism by using multiple parallel self-attention layers to jointly capture different aspects of information at different positions in the input log sequence, thereby more effectively extracting information. The feedforward neural network layer consists of two fully connected layers, the first of which uses the ReLU activation function. These two fully connected layers are equivalent to performing two linear transformations, which enhances the model's expressive power. The residual connection adds the module's input and output together as the final output, making model training easier. Layer normalization is a standardization method that is equivalent to normalizing the nodes in a neural network layer to zero mean and one variance.

[0048] By extracting bidirectional deep semantic features, the model's generalization ability will be stronger, and it can effectively solve problems such as polysemy and foreign vocabulary, thereby improving the robustness of the model.

[0049] S33, mask probability distribution generation;

[0050] Based on the log semantic features obtained in step S32, the masked language model MLM is used to train the model, thereby fully capturing the contextual relationship and ultimately outputting the masked probability distribution. The masked probability distribution is the probability distribution of the hidden log key mapped to the entire log key set κ, that is, the model obtains the probabilities of all possible results by predicting what the hidden log key is. The masked probability distribution calculation process is shown in Formula 3. It is the semantic feature representation of the i-th hidden log key in the j-th log sequence. After being processed by the Softmax function, it will output the probability distribution of the entire log key set κ W C and bC are trainable parameters. is the output of step S33, ie, the mask probability distribution.

[0051]

[0052] The masked probability distribution obtained after processing in step S33 can be used for log anomaly detection for three reasons. First, the distribution of log data is severely unbalanced. The vast majority of log data is normal logs, providing a large amount of data for model training. This allows the capture of a large amount of contextual relationships during pre-training, facilitating further detection of log anomalies. Second, while normal logs and abnormal logs are structurally similar, the probability of occurrence of certain words in context varies significantly, making them a basis for anomaly detection. Third, in real-world scenarios, MLM only requires normal logs for training, not labeled data. This model is an unsupervised anomaly detection model, which is more practical than traditional supervised learning anomaly detection models and more feasible for log anomaly detection in real-world scenarios.

[0053] The result of MLM processing is a probability distribution. Considering that the cross-entropy loss function is sensitive to probability distributions and has advantages such as good gradient descent performance, good mathematical properties, and ease of calculation and optimization, the model's objective function uses the cross-entropy loss function. The definition of the objective function is shown in Equation 4.

[0054]

[0055] in Indicates the real log key originally corresponding to the i-th special token [MASK] in the j-th log sequence; M is the number of special token [MASK] in the j-th log sequence, and N is the number of log sequences.

[0056] S34, anomaly detection;

[0057] Based on the mask probability distribution obtained in step S33, a maximum likelihood candidate set is constructed for log anomaly detection. First, we assume that our model is trained on normal logs. If a normal log sequence is given as input and the hidden log key is predicted, theoretically, it is possible to accurately predict what the hidden log key is. The prediction result of the hidden log key prediction is the probability distribution of the hidden log key over the entire log key set κ. Based on this probability distribution, the anomaly score of the log sequence can be calculated. For example, a log sequence to be detected is first given, and then a portion of the log key in the log sequence is hidden. The input is fed into the model for processing, and then the prediction result of the hidden log key is obtained. Based on this prediction result, a specific anomaly detection standard can be adopted to identify anomalies in the log sequence to be detected. The specific anomaly detection criteria employed are as follows: First, based on the output of the masked probability distribution generation phase, a maximum likelihood candidate set of g log keys is constructed. This candidate set consists of the top g most likely log keys predicted by the model for a hidden log key. A conditional judgment is then performed: if the actual log key is in this candidate set, it is considered normal; otherwise, it is considered abnormal. Finally, if the total number of abnormal log keys in the log sequence exceeds n, the log sequence is considered abnormal. Both g and n are hyperparameters, adjusted using the validation set during model training.

Claims

1. An unsupervised log anomaly detection method that does not rely on a log parser, characterized in that: The following steps are involved: S1. Log collection: Use the log collection tool Flume and the streaming tool Kafka to collect and transmit the real-time logs generated by the computer system; S2, log preprocessing: using regular replacement to enrich the semantic information of the log and tokenization to identify valid words and remove invalid information; S3, feature extraction and anomaly detection: Use the BERT model to perform bidirectional deep semantic feature extraction on the log data. In step S33, use the masked language model MLM to learn the contextual relationship between log keys in the log sequence, output the masked probability distribution, and perform log anomaly detection based on the masked probability distribution. Wherein, step S1 includes the following four sub-steps: S11. Use the Exec Source component of the log collection tool Flume agent to collect logs and generate events; S12, using the Spillable Memory Channel component of the Flume agent, a log collection tool, using memory and files as storage media to temporarily store the event Event generated by the Exec Source, and generating the event Event; S13. Use the Kafka Sink component of the Flume agent, a log collection tool, to forward the event Event generated by the Spillable Memory Channel, and generate messages on a specific Kafka topic that flow into the streaming tool Kafka. S14. Use the streaming tool Kafka to distribute the messages of a specific topic flowing into the Kafka Sink and deliver them to the log preprocessing module; Step S2 includes the following three sub-steps: S21, establish a dictionary of valid words; S22, regular replacement; S23, tokenization; Step S3 includes the following four sub-steps: S31, log vectorization representation; S32, semantic feature extraction; S33, mask probability distribution generation; S34. Anomaly detection.

2. According to claim 1, the unsupervised log anomaly detection method that does not rely on a log parser is characterized in that: Step S21 creates a valid word dictionary D covering some words in different log systems based on the open source corpus.

3. According to claim 1, the unsupervised log anomaly detection method that does not rely on a log parser is characterized in that: Step S22 uses a pre-set regular expression to replace some common variables.

4. According to claim 1, the unsupervised log anomaly detection method that does not rely on a log parser is characterized in that: Step S23 uses the WordPiece algorithm to divide the words in the log into tokens. If the token is in the valid word dictionary D, it is a valid word. Otherwise, it is filtered out as invalid information.

5. According to claim 1, the unsupervised log anomaly detection method that does not rely on a log parser is characterized in that: Step S31 includes converting the text-type log sequence into a vector of numerical type by adding special tags, masking operations, word embedding and position embedding operations. Step S32 takes the log vector obtained in step S31 as input, uses multiple Transformer encoders to learn the contextual relationship between log keys in the log sequence, and extracts bidirectional deep semantic features.

6. The unsupervised log anomaly detection method according to claim 1, which is independent of the log parser, is characterized in that: Step S34 constructs a maximum likelihood candidate set based on the mask probability distribution obtained in step S33 to perform log anomaly detection.

Images