Log anomaly detection method based on pre-trained BERT sentence vector and Informer-encoder

Abstract

The present invention relates to a log anomaly detection method based on pre-trained BERT sentence vectors and an informer-encoder. The method comprises the following steps: step S1, obtaining raw log data and pre-processing it to obtain a log template after dividing the session; step S2, semantic vectorization of the log template based on the pre-trained BERT sentence vectors; step S3, constructing an anomaly detection model based on the informer-encoder, and training the anomaly detection model using normal log data processed by steps S1 to S2; step S4, processing the log data to be detected according to steps S1 to S2, and then using the trained anomaly detection model to perform log anomaly detection. Compared with the prior art, the method of the present invention reduces the time and space complexity of model training, reduces the training cost of the model, and at the same time, improves the accuracy of the anomaly detection method.

Description

Technical Field

[0001] The present invention relates to the technical field of log anomaly detection, and in particular to a log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder. Background Art

[0002] Anomaly detection is a crucial component of software system operations and maintenance, and crucial for maintaining high availability and reliability. Logs are a crucial data source for anomaly detection. Generated by embedded code within programs, log data provides multi-dimensional, fine-grained information about system operations. Furthermore, logs contain rich semantic features, providing an accurate and interpretable data source for anomaly detection. In summary, log-based anomaly detection methods offer a valuable insight into discovering system anomalies and ultimately locating system issues.

[0003] Existing log anomaly detection methods mainly construct a limited amount of key semantic information by extracting log templates, then construct template vectors, input the template vectors into a neural network model to learn the logical dependencies between templates, and determine whether there are anomalies in the log based on the learned logical dependencies. These methods include: the semi-supervised log anomaly detection method LogAnomaly, which retrains word vectors with synonym and antonym features for the target log, obtains the vector of the log template through weighted summation, and then uses LSTM to learn features in normal logs to detect anomalies; the supervised log anomaly detection method LogRobust, which obtains log template vectors through FastText pre-trained word vectors and TF-IDF, and then automatically learns the relationship between log sequence vectors and anomaly labels through Bi-LSTM; in addition, Transformer-encoder and attention mechanisms have also been applied in log anomaly detection.

[0004] Although the above mainstream log anomaly detection methods can model key semantic information in logs and learn the logical dependencies between log templates to a certain extent, they still have the following shortcomings:

[0005] 1) Log template vectors are usually based on discrete word vectors, which are insufficient in representing the semantic information of the entire template and between templates;

[0006] 2) The LSTM model has a capacity bottleneck when processing long-distance feature dependencies;

[0007] 3) Although Transformer-encoder can capture long-distance feature dependencies, it suffers from the problem of high spatiotemporal complexity. Summary of the Invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder. This method reduces the time and space complexity of model training, reduces the training cost of the model, and at the same time, improves the accuracy of the anomaly detection method.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] The present invention provides a log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder, which includes the following steps:

[0011] Step S1: Obtain the original log data and pre-process it to obtain a log template after dividing the session;

[0012] Step S2: semantically vectorize the log template based on the pre-trained BERT sentence vector;

[0013] Step S3: Build an informer-encoder-based anomaly detection model and train the anomaly detection model using the normal log data processed by S1 to S2;

[0014] Step S4: After processing the log data to be detected according to steps S1 to S2, the trained anomaly detection model is used to perform log anomaly detection.

[0015] Preferably, the preprocessing in step S1 includes performing log parsing and session division on the original log data to obtain a log template sequence.

[0016] Preferably, the log parsing is specifically as follows:

[0017] The original log data is divided into a constant part and a variable part. The constant part is the template information describing the system operation events, and the variable part is the remaining tag information carrying the system dynamic operation;

[0018] Build a parse tree of a set depth, continuously match the original log with the nodes in the parse tree layer by layer, dynamically update the parse tree and generate the parse results corresponding to the log to obtain the log template information.

[0019] Preferably, the session division is specifically: dividing the multiple log template information extracted by log analysis into multiple log template sequences according to the session, wherein the session division includes division according to the inherent association identifier in the log and division according to the time window.

[0020] Preferably, step S2 is specifically as follows: inputting the log template after dividing the session obtained in step S1 into the BERT language model pre-training to obtain the sentence vector corresponding to the log template; wherein the sentence vector corresponding to the log template sequence in a session is constructed as a session matrix, the row represents the dimension of the column vector, and the column represents the length of the template sequence.

[0021] Preferably, the informer-encoder-based anomaly detection model in step S3 includes an embedding layer, an attention layer, a residual connection and layer normalization module, a feedforward layer, a residual connection and layer normalization module, and a linear layer connected in sequence; the input of the anomaly detection model is the conversation matrix obtained by BERT pre-training in step S2, and the output is the probability of the next log template.

[0022] Preferably, step S3 includes the following sub-steps:

[0023] Step S31: Input the conversation matrix obtained by BERT pre-training in step S2 into the input Embedding layer;

[0024] Step S32: Set a sliding window of fixed length and slide it on the session matrix. Construct the vectors in each window as the input matrix X. The category of the next log template outside the window is the training label of the window.

[0025] Step S33: Convert the input matrix X into query Q, key K and value V through the parameter matrix, and calculate the dot product attention. The expression is:

[0026]

[0027] Where Q = XW q , K=XW k 、V=XW v , d x is the feature dimension of the vector, and d is the pre-set hidden layer feature hyperparameter;

[0028] Step S34: Set s heads and concatenate them. Each head corresponds to a set of query queryQ, key queryK and value queryV. The s heads are transformed into an attention matrix with the same dimension as the input matrix through concat operation and linear transformation. The expression is:

[0029]

[0030] Where head is the head, W att is the parameter matrix of the linear transformation;

[0031] Step S35: Use the residual structure to optimize the multi-head attention. The optimized multi-head attention expression is:

[0032] [X] att =LN(X+Dropout(X att ))

[0033] Where LN is the layer normalization operation, and Dropout is the inactivation operation, which randomly specifies that some neurons in the neural network stop working to improve the generalization ability of the network.

[0034] Step S36: Distill the optimized multi-head attention and then optimize it with the residual structure. After n layers, the final output X is obtained. f ; Among them, the distillation process expression is:

[0035] X j+1 =MaxPooling(Relu(Conv1d([X j ] att )))

[0036] In the formula, [X j ] att is the multi-head attention of the j-th layer, Conv1d represents the one-dimensional convolution operation, Relu represents the activation function, and MaxPooling represents the maximum pooling process;

[0037] Step S37: Due to X f The last element in corresponds to the position of the last session and is used to predict the next session. The last element is selected and the probability of the next template category is calculated through the linear layer and sigmoid function. The expression is:

[0038]

[0039] Where, For X f The last element in , W is the parameter matrix of the linear layer, and b is the bias of the linear layer;

[0040] By continuously training and reducing the loss function, the sequential pattern of the template in the normal log session is learned.

[0041] Preferably, the step S33 further comprises selecting the most critical part of the query by adopting a sampling strategy, specifically: calculating the criticality metric of each query To select, the expression is:

[0042]

[0043] Where qi is the i-th query in Q, k j is the jth value key in K; t is the window length, that is, the number of log templates contained in a window;

[0044] Given a fixed sampling factor c, the first c ln t queries are selected as the most critical part of the query; t is;

[0045] According to the matrix composed of sampled queries The expression of dot product attention is:

[0046]

[0047] Preferably, the loss function is a cross entropy loss function.

[0048] Preferably, step S4 specifically comprises: using the anomaly detection model based on the normal hypothesis constructed in step S3 to learn the sequence pattern of the template in the normal log session; when an anomaly occurs, the sequence pattern of the template in the log session deviates, generating a new pattern that does not exist in the normal pattern, and detecting the occurrence of the new pattern to perform anomaly detection using the following method:

[0049] 1) Using the method of step S1, obtain the log template after dividing the session;

[0050] 2) Using the inverse method of step S2, perform semantic vectorization on the log template sequence based on the BERT pre-trained sentence vector;

[0051] 3) Construct a sliding window with the same length as in the training process and slide it over the template sequence. The semantic vector matrix corresponding to the template sequence within the sliding window is used as the input data for model detection, and the category of the next template outside the sliding window is the true value.

[0052] 4) The anomaly detection model predicts the category of the next template based on the semantic vector matrix. When the predicted value of any window does not match the true value, it is considered that an anomaly has occurred in the current log session; when the predicted values ​​of all windows match the true values, the current log session is considered normal.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] 1) This paper uses the BERT pre-training model to obtain sentence vectors containing semantic attributes corresponding to log templates, thereby improving the expressiveness and accuracy of log template vectors;

[0055] 2) To address the inadequacy of log template vectors in representing the semantic information of the entire template and between templates, this paper first parses the log using a log parsing algorithm, extracting the core information (i.e., the template) from the log to reduce the input range. Then, using the BERT pre-trained model, it obtains the sentence vector containing semantic attributes corresponding to the log template, improving the expressiveness and accuracy of the log template vector.

[0056] 3) The informer-encoder-based anomaly detection model constructed by this invention uses the attention mechanism to replace the LSTM to eliminate the problem of insufficient ability to handle long-distance feature dependencies;

[0057] 4) The multi-head attention of the present invention is optimized through the residual structure to prevent the network degradation problem caused by the network structure being too deep;

[0058] 5) The present invention improves the sampling strategy to select the most critical part of the query, ignores the query with relatively small impact to reduce the amount of calculation, and reduces the computational complexity of the dot product attention;

[0059] 6) In view of the fact that the existing LSTM model has insufficient ability to handle long-distance feature dependencies and the high spatiotemporal complexity of the Transformer-encoder, the present invention constructs an anomaly detection model based on the Informer-encoder including a probabilistic sparse self-attention mechanism. While replacing the LSTM with the attention mechanism to eliminate the insufficient ability to handle long-distance feature dependencies, it reduces the spatiotemporal complexity of model training, improves the accuracy of the anomaly detection method, and reduces the training cost of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the method of the present invention;

[0061] Figure 2 A structural diagram of the informer-encoder-based anomaly detection model. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0063] Example

[0064] This example provides a log anomaly detection method based on pre-trained BERT sentence vectors and the Informer-encoder. The key points of this method are:

[0065] 1) Obtaining sentence vectors containing semantic attributes corresponding to the parsed log templates through the BERT pre-training model can better model the semantic information in the logs and provide a better input information source for the anomaly detection model.

[0066] 2) Building an anomaly detection model based on the informer-encoder can significantly reduce the time and space complexity of training while capturing long-distance dependencies, reducing training costs and enabling more accurate detection of anomalies in logs.

[0067] Next, the method of the present invention is described in detail.

[0068] Step S1: Obtaining original log data and performing data preprocessing, including the following sub-steps:

[0069] Step S11, Log Parsing: Utilize existing log parsing algorithms to parse the original logs, extracting core information from the logs and reducing the input range. Taking a typical log parsing method as an example, the parsing method divides the original log information into a constant portion and a variable portion. The constant portion is the template information describing system operation events, while the variable portion carries the remaining tag information of the system's dynamic operation. A parse tree of fixed depth is then constructed. By continuously matching the original logs with nodes in the parse tree layer by layer, the parse tree is dynamically updated and the corresponding parsing result (template) is generated, extracting the core information from the logs.

[0070] Step S12, Session Division: Log data is usually not isolated. A typical system event may generate multiple logs with related relationships. After log parsing, the multiple log template information extracted needs to be divided into multiple log template sequences according to the session to focus on the local relationship of the log and input it into the model. There are usually two ways to set up a session:

[0071] 1) Classification by inherent association identifiers in logs: Logs may contain identifiers that identify associations between logs. For example, in HDFS log datasets, the block_id field is used to identify related logs within a block. Using association identifiers, log templates corresponding to related logs can be grouped into the same session.

[0072] 2) Division by time window: Logs typically record the time at which events occurred. By setting a time window, you can group templates corresponding to temporally related logs into the same session. For example, you can set the time window to 1 minute. Starting from the start of logging, log information within 1 minute is captured and templates are extracted to form a template sequence within a session. As the time window continues to slide backward, the log data is eventually converted into multiple temporally related log template sequences.

[0073] Step S2: Semantic vectorization of log templates based on BERT pre-trained sentence vectors:

[0074] BERT is a pre-trained language model based on the Transformer encoder architecture, which is pre-trained using two unsupervised learning methods:

[0075] Masked Language Model (MLM): randomly removes some tokens from a sentence (masking operation) and then predicts what the removed tokens are.

[0076] Next Sentence Prediction (NSP): Given two sentences, predict whether one of them is the next sentence of the other.

[0077] By pre-training both tasks on a large corpus, we can develop a language model capable of representing rich semantic meaning. Using this pre-trained language model as a foundation, we input the log template constructed in the first step to generate the sentence vector corresponding to the log template. The sentence vectors corresponding to a sequence of templates within a session can be constructed as a matrix, where rows represent the dimensions of the column vectors and columns represent the length of the template sequence. This matrix serves as the input for the anomaly detection model in the next stage.

[0078] Step S3: Build an anomaly detection model based on Informer-encoder:

[0079] Since the number of anomalies in logs is relatively small and supervised methods require a lot of annotation work on logs, the anomaly detection model constructed in this paper is based on the normal assumption, that is, learning the sequence pattern of the template in the normal log session, and when the detection results deviate, it is considered an anomaly. The construction idea of ​​the anomaly detection model based on Informer-encoder is as follows Figure 2 As shown, it includes an embedding layer, an attention layer, a residual connection and layer normalization module, a feedforward layer, a residual connection and layer normalization module, and a linear layer connected in sequence; the input of the anomaly detection model is the conversation matrix obtained by BERT pre-training in step S2, and the output is the probability of the next log template.

[0080] The specific steps include:

[0081] The input embedding is constructed using the BERT pre-trained session matrix from the previous stage. A sliding window of fixed length x is set and slid from top to bottom along the columns of each session matrix. The vectors within each window can be constructed as the input matrix X. The category of the next template outside the window is the training label of this window. X is converted into the query queryQ = XW through the parameter matrix. q , key keyK=XW k Sum value valueV=XW v ,in, d x is the feature dimension of the vector, d is the pre-set hidden layer feature hyperparameter; the formula for dot product attention calculation is:

[0082]

[0083] In order to reduce the computational complexity of dot product attention, a sampling strategy is needed to select the most critical part of the query and ignore the queries with relatively small impact to reduce the amount of calculation. The sampling strategy calculates the criticality metric of each query. The implementation is as follows, where q i is the i-th query in Q, k j is the jth key in K. Given a fixed sampling factor c, the first c lnt queries will be selected to enter the next attention calculation. t is the window length, that is, the number of log templates contained in a window.

[0084]

[0085] According to the matrix composed of sampled queries The formula for dot product attention calculation is as follows:

[0086]

[0087] In order to better learn the relationship between templates in the window from multiple dimensions, s heads can be set and spliced. Each head corresponds to a set of query, key, and value, and the corresponding matrix is ​​obtained by calculating the above dot product attention. The s heads are converted into an attention matrix with the same dimension as the input matrix through concat operation and linear transformation:

[0088]

[0089] Where head is the head, W att is the parameter matrix of the linear transformation;

[0090] To prevent network degradation caused by an overly deep network structure, the obtained multi-head attention is optimized through a residual structure:

[0091] [X] att =LN(X+Dropout(X att ))

[0092] Where LN is the layer normalization operation, and Dropout is the inactivation operation, which randomly specifies that some neurons in the neural network stop working to improve the generalization ability of the network.

[0093] On this basis, the commonly used linear structure can be replaced by a distillation process. For the j-th layer X, the distillation process is as follows, where Conv1d represents a one-dimensional convolution operation (the convolution kernel size is 3), Relu represents the activation function, and MaxPooling represents the maximum pooling process (the step size is 2):

[0094] X j+1 =MaxPooling(Relu(Conv1d([X j ] att )))

[0095] The same residual structure as above is constructed for the output of this layer to obtain the output of the distillation layer. After n layers, the final output X f , due to X f Each element in contains information about other elements, and the last element corresponds to the position of the last session, which is suitable for predicting the next session. The last element is selected and the probability of the next template category is calculated through the linear layer and sigmoid function:

[0096]

[0097] Always, For X f The last element in , W is the parameter matrix of the linear layer, and b is the bias of the linear layer.

[0098] The loss function uses the cross-entropy loss function commonly used in classification tasks. By continuously training to reduce the loss, the sequential pattern of the template in the normal log session is learned, providing a basis for the next step of log anomaly detection based on normal assumptions.

[0099] Step S4: After processing the log data to be detected according to steps S1 to S2, the trained anomaly detection model is used to perform log anomaly detection:

[0100] The anomaly detection model constructed in the previous step is based on the normal assumption and learns the sequential patterns of templates in normal log sessions. When an anomaly occurs, the sequential patterns of templates in the log session often deviate, such as by generating new templates or swapping the order of templates, thus creating new patterns that do not exist in the normal pattern. The present invention uses the following method to detect the emergence of new patterns for anomaly detection:

[0101] 1) In the real-time anomaly detection process, the session is divided and the template sequence in the session is extracted in the same way as in the first two steps;

[0102] 2) Semantic vectorization of template sequences in the conversation based on BERT pre-trained sentence vectors;

[0103] 3) Construct a sliding window with the same length as in the training process and slide it over the template sequence. The semantic vector matrix corresponding to the template sequence within the sliding window is used as the input data for model detection, and the category of the next template outside the sliding window is the true value.

[0104] 4) The anomaly detection model predicts the category of the next template based on the semantic vector matrix. When the predicted value of any window does not match the true value, it is considered that an anomaly has occurred in the current log session. When the predicted values ​​of all windows match the true values, the current log session is considered normal.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder, characterized by: The method comprises the following steps: Step S1: Obtain the original log data and pre-process it to obtain a log template after dividing the session; Step S2: semantically vectorize the log template based on the pre-trained BERT sentence vector; Step S3: Build an informer-encoder-based anomaly detection model and train the anomaly detection model using the normal log data processed by S1 to S2; Step S4: After processing the log data to be detected according to steps S1 to S2, the trained anomaly detection model is used to perform log anomaly detection; The informer-encoder-based anomaly detection model includes an embedding layer, an attention layer, a residual connection and layer normalization module, a feedforward layer, a residual connection and layer normalization module, and a linear layer connected in sequence; the input of the anomaly detection model is the conversation matrix obtained by BERT pre-training in step S2, and the output is the probability of the next log template; The step S3 includes the following sub-steps: Step S31: Input the conversation matrix obtained by BERT pre-training in step S2 into the Embedding layer; Step S32: Set a sliding window of fixed length and slide it on the session matrix. Construct the vectors in each window as the input matrix X. The category of the next log template outside the window is the training label of the window. Step S33: Convert the input matrix X into query Q, key K and value V through the parameter matrix, and calculate the dot product attention. The expression is: Where Q = XW q , K=XW k 、 d x is the feature dimension of the vector, and d is the pre-set hidden layer feature hyperparameter; Step S34: Set s heads and concatenate them. Each head corresponds to a set of query queryQ, key queryK and value queryV. The s heads are converted into an attention matrix with the same dimension as the input matrix through concat operation and linear transformation. The expression is: Where head is the head, W att is the parameter matrix of the linear transformation; Step S35: Use the residual structure to optimize the multi-head attention. The optimized multi-head attention expression is: [X] att =LN(X+Dropout(X att )) Where LN is the layer normalization operation and Dropout is the inactivation operation; Step S36: Distill the optimized multi-head attention and then optimize it with the residual structure. After n layers, the final output X is obtained. f ; Among them, the distillation process expression is: In the formula, [X j ] att is the multi-head attention of the j-th layer, Conv1d represents the one-dimensional convolution operation, Relu represents the activation function, and MaxPooling represents the maximum pooling process; Step S37: Due to X f The last element in corresponds to the position of the last session and is used to predict the next session. The last element is selected and the probability of the next template category is calculated through the linear layer and sigmoid function. The expression is: Where, For X f The last element in , W is the parameter matrix of the linear layer, and b is the bias of the linear layer; By continuously training and reducing the loss function, the sequential pattern of the template in the normal log session is learned.

2. A log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder according to claim 1, characterized in that: The preprocessing in step S1 includes performing log parsing and session division on the original log data to obtain a log template sequence.

3. The log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder according to claim 2 is characterized in that: The log analysis is specifically as follows: The original log data is divided into a constant part and a variable part. The constant part is the template information describing the system operation events, and the variable part is the remaining tag information carrying the system dynamic operation; Build a parse tree of a set depth, continuously match the original log with the nodes in the parse tree layer by layer, dynamically update the parse tree and generate the parse results corresponding to the log to obtain the log template information.

4. The log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder according to claim 2 is characterized in that: The session division is specifically: dividing the multiple log template information extracted by log analysis into multiple log template sequences according to the session, wherein the session division includes division according to the inherent association identifier in the log and division according to the time window.

5. The log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder according to claim 4 is characterized in that: Step S2 specifically includes: inputting the log template after dividing the session obtained in step S1 into the BERT language model pre-training to obtain the sentence vector corresponding to the log template; wherein the sentence vector corresponding to the log template sequence in a session is constructed into a session matrix, where the rows represent the dimensions of the column vectors and the columns represent the length of the template sequence.

6. A log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder according to claim 1, characterized in that: The step S33 also includes selecting the most critical part of the query by using a sampling strategy, specifically: calculating the criticality metric of each query To select, the expression is: Where q i is the i-th query in Q, k j is the jth value key in K; t is the window length, that is, the number of log templates contained in a window; Given a fixed sampling factor c, select the first c ln t queries as the most critical part of the query; According to the matrix composed of sampled queries The expression of dot product attention is:

7. The log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder according to claim 1 is characterized in that: The loss function is a cross entropy loss function.

8. The log anomaly detection method based on pre-trained BERT sentence vectors and Informer-encoder according to claim 1 is characterized in that: Step S4 specifically includes: using the anomaly detection model based on the normal assumption constructed in step S3 to learn the sequence pattern of the template in the normal log session; when an anomaly occurs, the sequence pattern of the template in the log session deviates, generating a new pattern that does not exist in the normal pattern, and detecting the occurrence of the new pattern to perform anomaly detection using the following method: 1) Using the method of step S1, obtain the log template after dividing the session; 2) Using the inverse method of step S2, perform semantic vectorization on the log template sequence based on the BERT pre-trained sentence vector; 3) Construct a sliding window with the same length as in the training process and slide it over the template sequence. The semantic vector matrix corresponding to the template sequence within the sliding window is used as the input data for model detection, and the category of the next template outside the sliding window is the true value. 4) The anomaly detection model predicts the category of the next template based on the semantic vector matrix. When the predicted value of any window does not match the true value, it is considered that an anomaly has occurred in the current log session. When the predicted values ​​of all windows match the true values, the current log session is considered normal.

Images