A log sequence anomaly detection method based on out-of-stream regularization

Abstract

The present invention provides a log sequence anomaly detection method based on out-of-stream regularization, which optimizes the BERT model and uses it as a semantic feature extraction model. The optimization method is to use MixUp data enhancement as a form of out-of-stream regularization, impose linear constraints on the model input space outside the data manifold, and embed the generator and discriminator into different Encoder layers of the BERT model respectively. The discriminator is trained to detect whether the input embedding is outside the manifold, and the generator is optimized to generate an out-of-manifold embedding, so that it is easy to be recognized as an out-of-manifold embedding by the discriminator, so as to find embeddings that are useful for log sequence classification tasks but cannot be accessed by words, thereby improving the accuracy of anomaly detection. In addition, the present invention extracts log semantic features from two granularities, log entries and log sequences, respectively, which can not only extract the semantic features of the log sequence, but also extract the semantic information contained in the log entry itself, thereby enhancing the robustness of anomaly detection.

Description

Technical Field

[0001] The present invention relates to the technical field of log sequence anomaly detection, and in particular, to a log sequence anomaly detection method based on out-of-stream regularization. Background Art

[0002] As the scale of the system increases, its structure becomes increasingly complex. System logs are key indicators for recording system status. Analyzing information that may cause system abnormalities based on timestamps, log levels, log messages, etc. is an important measure to improve system reliability.

[0003] At present, many methods have been proposed for log-based anomaly detection at home and abroad, which are mainly divided into methods based on log event indexing and methods based on log template semantics.

[0004] In the anomaly detection method based on log event index, Xu et al. used data mining and machine learning PCA algorithm to unsupervisedly detect anomalies by analyzing system logs, counted the event states and occurrence times expressed in the logs, and used the state ratio vector and the practice count vector as the input of the model for anomaly detection. Although this method extracts complex features of logs from many aspects, it does not fully analyze the semantic information, which will lead to false alarms. Du et al. proposed DeepLog, which uses the LSTM neural network model in deep learning to learn the relationship between normal log sequences. Although the anomaly detection effect is good, and the detection model is incrementally updated with false positive logs through user feedback to adapt the model to the new log pattern. However, the log encoding index method cannot fully extract the semantic information of the log template, and cannot semantically process new logs, resulting in a lack of robustness and limited detection accuracy.

[0005] Among the anomaly detection methods based on log template semantics, Zhang et al. proposed LogRobust, which is an anomaly detection method based on unstable log data. It extracts the semantic information of log events and represents them as semantic vectors, and then uses the attention-based Bi-LSTM model to detect anomalies. Meng et al. proposed LogAnomaly, which models the log stream as a natural language sequence, uses template2vec to extract the hidden semantic information in the log template, and uses LSTM to predict the next log that may appear, and detects anomalies based on the predicted value. Although these methods can extract the semantic information of log templates and have certain robustness, and can appropriately reduce the impact of new logs on the detection results, the embedding space of the semantic feature extraction model is limited, and they all belong to the manifold embedding, which fails to fully utilize the manifold embedding that is useful for anomaly detection tasks, resulting in limited accuracy.

[0006] Currently, log-based anomaly detection has the following main disadvantages:

[0007] (1) Most methods based on log event indexing extract log templates from log messages and then represent them with log indexes. Although this simplifies the input of the feature extraction model, it cannot fully utilize the semantic information in the log templates, resulting in most of them being unable to handle log update situations and lacking robustness.

[0008] (2) Most methods based on log template semantics model the log template corresponding to the log message as a natural language sequence, use a pre-trained language model to convert the log template into a sentence vector, and then input the vector into a feature extraction model for feature extraction. Although the semantic information of the log template is retained, the embedding space of the semantic feature extraction model is not fully utilized, which reduces the accuracy of anomaly detection. Summary of the invention

[0009] According to the above-mentioned technical problem that the limited utilization of the embedding space of the semantic feature extraction model of the log sequence leads to the low accuracy of the detection method, considering the importance of the semantic information of the log entry and the embedding space of the semantic feature extraction model to anomaly detection, the present invention proposes a log sequence anomaly detection method based on out-of-Manifold regularization OMRLog (LogAnomaly Detection Method Based on out-of-Manifold Regularization). When extracting the semantic representation of the log sequence, the present invention optimizes the BERT model. The specific method is to use MixUp data enhancement as a form of out-of-flow regularization to discover and fully utilize the out-of-manifold embedding of BERT, and to train the discriminator to detect whether the input embedding is located in the manifold. At the same time, the generator is optimized to generate the out-of-manifold embedding, so that it is easy to be recognized as an out-of-manifold embedding by the discriminator, thereby improving the accuracy of log sequence anomaly detection. In addition, the present invention extracts the semantic representation of the log more abundantly from the two granularities of log entry and sequence, and can extract the semantic information contained in the log entry itself, thereby enhancing the robustness of anomaly detection.

[0010] The technical means adopted by the present invention are as follows:

[0011] A log sequence anomaly detection method based on out-of-stream regularization, comprising:

[0012] S1. Optimize the BERT model and use it as a semantic feature extraction model;

[0013] S2. Given a log set χ, train the semantic feature extraction model;

[0014] S3. Given a log set, use the trained semantic feature extraction model to perform anomaly detection on the log sequence.

[0015] Furthermore, in step S1, the method for optimizing the BERT model is as follows:

[0016] S11, MixUp data augmentation is used as a form of out-of-stream regularization, imposing linear constraints on the model input space outside the data manifold, embedding the generator and discriminator into different generation layers of the BERT model respectively;

[0017] S12, train the discriminator to detect whether the input embedding is outside the manifold;

[0018] S13. Generate the manifold outer embedding through the manifold outer embedding generated by the generation layer and the label optimization generator.

[0019] Furthermore, the specific implementation process of step S2 is as follows:

[0020] S21, log parsing, converting each log event into a specific event template associated with key parameters, selecting the Drain algorithm with good performance in extracting log templates, building a fixed-depth parsing tree based on log data, performing log parsing according to the template extraction rules contained in the tree, and extracting structured log keys from unstructured log events;

[0021] S22, manually selecting a framing method for the log sequence according to the characteristics of the log data set. If there is a Block_id in the data set, the log sequence is divided according to the session id; if not, the log sequence is framed with a sliding window of a fixed size;

[0022] S23, inputting the log entries in the log sequence into the BERT model to extract sentence vectors;

[0023] S24, determining whether there are any unprocessed log entries in the log sequence, if so, executing step S23 until all log entries are processed;

[0024] S25. Linearly concatenate all the sentence vectors in the sequence to obtain the initial sequence vector v seq Input into the Encoder layer of the optimized BERT model;

[0025] S26, linear interpolation of sequence vectors and their labels based on MixUp;

[0026] S27. Calculate the classification loss of the log sequence based on the embedding and label in the manifold. Perform binary classification on the log sequence by optimizing the pre-trained weight coefficient of the BERT model. Take the sequence label in the manifold as the output target and use KL divergence as the loss function. The objective function of the log sequence classification is as follows:

[0027]

[0028] Among them, ω f Represents the optimization of the trainable parameters of the BERT model pre-training and classifier, represents KL divergence, function f(seq batch ) is to convert seq batch Function to perform classification;

[0029] S28. Calculate the classification loss and optimize the discriminator based on the embedding and label inside and outside the manifold. Take the manifold label (inside and outside the manifold) as the output target, and use the binary cross entropy function as the loss function to optimize the discriminator. The objective function is as follows:

[0030]

[0031] Among them, ω g ,ω d Represent the trainable parameters of the generator and discriminator respectively, represents the binary cross entropy loss, for The vector output by the j-th layer encoding of the optimized BERT model, For seq batch The vector output by the j-th layer encoding of the optimized BERT model;

[0032] S29, take the sequence label outside the manifold as the output target, use KL divergence as the loss function to optimize the generator, the objective function is as follows:

[0033]

[0034] in, Represents the trainable parameters of the generation layer and subsequent Encoder layer, ω g represents the trainable parameters of the generator; the generator is optimized by the objective function in step S28 and the objective function in step S29, so that the generator generates an outer manifold embedding that is helpful for the log sequence classification task, and the final classification objective function is defined as:

[0035]

[0036] in, represents the classification loss of the log sequence, and Represent the losses of training the generator and discriminator to discriminate off-manifold embeddings, respectively, and e is responsible for regulating the two objectives of optimizing the generator.

[0037] Furthermore, the specific implementation process of step S23 is as follows:

[0038] S231. Take a log x∈χ in the set χ and split x into a word sequence w∈{0,...,|V|} m ;

[0039] S232, use the tokenizer of the vocabulary V to tokenize the word, where m represents the length of the log entry;

[0040] S233, map each word into a D-dimensional vector according to the embedding table, then the sentence vector corresponding to the log entry is After the l-layer Encoder, it is converted into

[0041] Furthermore, the specific implementation process of step S26 is as follows:

[0042] Determine the mixing coefficient λ∈[0,1] for linear interpolation, and embed the generator on the i-th layer Encoder of the optimized BERT model. The goal of the generator is to linearly interpolate the context embeddings within the two manifolds to synthesize the embedding outside the manifold;

[0043] The generator uses a two-layer fully connected network with SoftMax normalization to generate a lower bound α and an interval Δ based on the context embedding within the two manifolds, thus The mixing coefficients are sampled in

[0044] In order to avoid the huge computational overhead caused by the serial processing of two input sequences, a twin architecture is adopted to share weights on two different inputs to reduce the computational overhead;

[0045] Optimize the mixing coefficient λ sampled from a uniform distribution, using the reparameterization trick, introducing The mixing coefficient λ is calculated as follows:

[0046] λ=α+γ×Δ

[0047] Linear interpolation is performed based on the mixing coefficient λ. The linear interpolation is explained by taking a batch sequence vector and its label as an example, as shown below:

[0048]

[0049]

[0050] in, represents the sequence vector mapped from the 0th layer (embedding layer) of the optimized BERT to the i-th layer, is the sequence vector of the batch randomly perturbed from the 0th layer of the optimized BERT to the i-th layer, Represents the sequence vector label of the batch, Represents the sequence vector label after random perturbation of the batch, is the sequence vector after linear interpolation, Represents the label after linear interpolation.

[0051] Furthermore, the specific implementation process of step S3 is as follows:

[0052] S31, log parsing, converting each log event into a specific event template associated with key parameters, selecting the Drain algorithm with good performance in extracting log templates, building a fixed-depth parsing tree based on log data, performing log parsing according to the template extraction rules contained in the tree, and extracting structured log keys from unstructured log events;

[0053] S32, framing the log sequence with Block_id or a sliding window of a fixed size;

[0054] S33, inputting the log entries in the log sequence into BERT to extract sentence vectors;

[0055] S34, determining whether there are any unprocessed log entries in the log sequence, if so, returning to step S33 until all log entries are processed;

[0056] S35, inputting the initial sequence vector linearly concatenated from all sentence vectors into the Encoder layer of the trained optimized BERT model to extract the sequence vector;

[0057] S36, performing anomaly detection on the sequence vector based on MLP, and classifying the sequence vector as normal or abnormal;

[0058] S37, determine whether there are any undetected sequences in the log set, if yes, return to step S32, until all log sequences are detected, otherwise, end the detection.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] 1. System log is a key indicator for recording system status. Analyzing information that may cause system anomalies based on timestamp, log level, log message, etc. is an important measure to improve system reliability. This paper proposes a new log sequence anomaly detection method based on out-of-flow regularization, OMRLog, which applies the out-of-manifold embedding based on in-manifold embedding synthesis to the anomaly detection of log sequences.

[0061] 2. The log sequence anomaly detection method based on out-of-stream regularization provided by the present invention uses MixUp data augmentation as a form of out-of-stream regularization when extracting sequence representations, imposes linear constraints on the model input space outside the data manifold, and embeds the generator and discriminator into different Encoder layers of the BERT model. The two work together in an end-to-end manner, so that the out-of-manifold embedding can be used for anomaly detection tasks based on log sequences.

[0062] 3. The present invention extracts the semantic representation of logs from two perspectives: log entries and log sequences. It can extract the semantic information contained in the log entries themselves, thereby enhancing the robustness of anomaly detection.

[0063] 4. Experimental evaluation on large-scale system logs shows that compared with existing methods, the proposed method fully utilizes the embedding space of the semantic feature extraction model, and thus has better anomaly detection effect. In future work, the present invention will further explore feature extraction technology for sequence anomaly detection, integrate manifold embedding into other feature extraction models, and test the efficiency of the proposed method on more log data sets.

[0064] Based on the above reasons, the present invention can be widely promoted in fields such as log sequence anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0066] Figure 1 This is a framework diagram of the log sequence anomaly detection method based on out-of-stream regularization of the present invention.

[0067] Figure 2 This is an anomaly detection model of the log sequence anomaly detection method based on out-of-stream regularization of the present invention.

[0068] Figure 3 This is a training flow chart of the anomaly detection model of the present invention.

[0069] Figure 4 This is a flow chart of log sequence anomaly detection based on out-of-stream regularization of the present invention.

[0070] Figure 5 The impact of the embedding position of the generator and discriminator in BERT on the detection performance in the HDFS dataset provided by the embodiment of the present invention.

[0071] Figure 6The impact of the embedding position of the generator and discriminator in BERT on the detection performance in the BGL dataset provided by the embodiment of the present invention.

[0072] Figure 7 The performance of different methods provided by the embodiments of the present invention on the HDFS data set.

[0073] Figure 8 The performance of different methods provided by the embodiments of the present invention on the BGL dataset. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0075] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0076] The present invention provides a log sequence anomaly detection method based on out-of-stream regularization, such as Figure 1 As shown, including:

[0077] S1. Optimize the BERT model and use it as a semantic feature extraction model;

[0078] In specific implementation, as a preferred embodiment of the present invention, in step S1, the method for optimizing the BERT model is as follows:

[0079] S11, MixUp data enhancement is used as a form of out-of-stream regularization, linear constraints are imposed on the model input space outside the data manifold, and the generator and discriminator are respectively embedded in different generation layers of the BERT model; in this embodiment, the present invention makes full use of the self-attention mechanism of BERT, such as Figure 2As shown, first, the sentence vector v of each log entry in the log sequence is extracted based on BERT sn , each sentence vector v sn After linear concatenation, it is used as the initial sequence vector v seq ; Then, based on the 12-layer Encoder of the optimized BERT, the initial sequence vector v seq Further extraction; finally, the extracted sequence vector is classified into two categories based on MLP (Multilayer Perceptron) and Softmax function. In the process of extracting sequence vectors, the generator and discriminator are respectively embedded into different Encoder layers of BERT. The Encoder layer (e.g., the i-th layer) in which the generator is embedded is called the generation layer. In the generation layer, MixUp is performed based on the sequence vector to generate the manifold outer embedding and label, and the manifold outer embedding is input into the next Encoder layer of the generation layer.

[0080] S12. Train the discriminator to detect whether the input embedding is outside the manifold. For the discriminator, both the inside and outside manifold embeddings pass through the j-th layer encoder where the discriminator is located, so it can learn both inside and outside manifold embeddings. By optimizing the discriminator to identify whether the embedding is outside the manifold, the BERT model is optimized based on the outside manifold embedding, thereby improving the accuracy of log sequence anomaly detection.

[0081] S13. Generate the manifold outer embedding through the manifold outer embedding generated by the generation layer and the label optimization generator.

[0082] S2. Given a log set χ, train the semantic feature extraction model;

[0083] In specific implementation, as a preferred embodiment of the present invention, the specific implementation process of step S2 is as follows:

[0084] S21, log parsing, converting each log event into a specific event template associated with key parameters, and selecting the Drain algorithm with good performance in extracting log templates. The core idea is to build a fixed-depth parsing tree based on log data, perform log parsing according to the template extraction rules contained in the tree, and extract structured log keys from unstructured log events;

[0085] S22. Manually select a way to frame the log sequence according to the characteristics of the log data set. If there is Block_id in the data set (such as HDFS), the log sequence is divided according to the session id; if there is no Block_id (such as BGL), the log sequence is framed with a sliding window of a fixed size;

[0086] S23, inputting the log entries in the log sequence into the BERT model to extract sentence vectors;

[0087] S231. Take a log x∈χ in the set χ and split x into a word sequence w∈{0,...,|V|} m ;

[0088] S232, use the tokenizer of the vocabulary V to tokenize the word, where m represents the length of the log entry;

[0089] S233, map each word into a D-dimensional vector according to the embedding table, then the sentence vector corresponding to the log entry is After the l-layer Encoder, it is converted into

[0090] S24, determining whether there are any unprocessed log entries in the log sequence, if so, executing step S23 until all log entries are processed;

[0091] S25. Linearly concatenate all the sentence vectors in the sequence to obtain the initial sequence vector v seq Input into the Encoder layer of the optimized BERT model;

[0092] S26, linear interpolation of sequence vectors and their labels based on MixUp;

[0093] Determine the mixing coefficient λ∈[0,1] for linear interpolation, and embed the generator on the i-th layer Encoder of the optimized BERT model. The goal of the generator is to linearly interpolate the context embeddings within the two manifolds to synthesize the embedding outside the manifold;

[0094] The generator uses a two-layer fully connected network with SoftMax normalization to generate a lower bound α and an interval Δ based on the context embedding within the two manifolds, thus The mixing coefficients are sampled in

[0095] In order to avoid the huge computational overhead caused by the serial processing of two input sequences, a twin architecture is adopted to share weights on two different inputs to reduce the computational overhead;

[0096] Optimize the mixing coefficient λ sampled from a uniform distribution, using the reparameterization trick, introducing The mixing coefficient λ is calculated as follows:

[0097] λ=α+γ×Δ

[0098] Linear interpolation is performed based on the mixing coefficient λ. The linear interpolation is explained by taking a batch sequence vector and its label as an example, as shown below:

[0099]

[0100]

[0101] in, represents the sequence vector mapped from the 0th layer (embedding layer) of the optimized BERT to the i-th layer, is the sequence vector of the batch randomly perturbed from the 0th layer of the optimized BERT to the i-th layer, Represents the sequence vector label of the batch, Represents the sequence vector label after random perturbation of the batch, is the sequence vector after linear interpolation, Represents the label after linear interpolation.

[0102] S27. Calculate the classification loss of the log sequence based on the embedding and label in the manifold. Perform binary classification (normal or abnormal) on the log sequence by optimizing the pre-trained weight coefficient of the BERT model. Take the sequence label in the manifold as the output target and use KL divergence as the loss function. The objective function of the log sequence classification is as follows:

[0103]

[0104] Among them, ω f Represents the optimization of the trainable parameters of the BERT model pre-training and classifier, represents KL divergence, function f(seq batch ) is to convert seq batch Function to perform classification;

[0105] S28. Calculate the classification loss and optimize the discriminator based on the intra-manifold and extra-manifold embeddings and labels;

[0106] The discriminator is embedded on the j-th layer Encoder of the optimized BERT model, and its task is to learn the difference between the embedding inside the manifold and the embedding outside the manifold. Since both the embedding inside and outside the manifold are input into the discriminator, and the over-parameterized optimized BERT is more inclined to memorize the data used for training (the embedding inside the manifold), in this case, the generator is more likely to imitate the embedding inside the manifold (memorized by the optimized BERT model) rather than generate new embeddings. Therefore, the discriminator is needed to monitor the generator so that it generates the embedding outside the manifold that is useful for the log sequence classification task.

[0107] For the sequence vector input to the discriminator The probability score s∈[0,1] of the embedding outside the manifold is s∈[0,1]. In order to make the output of the discriminator correspond to the manifold label, the present invention takes the manifold label (inside the manifold and outside the manifold) as the output target and uses the binary cross entropy function as the loss function to optimize the discriminator. The objective function is as follows:

[0108]

[0109] Among them, ω g ,ω d Represent the trainable parameters of the generator and discriminator respectively, represents the binary cross entropy loss, for The vector output by the j-th layer encoding of the optimized BERT model, For seq batch The vector output by the j-th layer encoding of the optimized BERT model;

[0110] S29, take the sequence label outside the manifold as the output target, use KL divergence as the loss function to optimize the generator, the objective function is as follows:

[0111]

[0112] in, Represents the trainable parameters of the generation layer and subsequent Encoder layer, ω g represents the trainable parameters of the generator; the generator is optimized by the objective function in step S28 and the objective function in step S29, so that the generator generates an outer manifold embedding that is helpful for the log sequence classification task, and the final classification objective function is defined as:

[0113]

[0114] in, represents the classification loss of the log sequence, and Represent the losses of training the generator and discriminator to discriminate off-manifold embeddings, respectively, and e is responsible for regulating the two objectives of optimizing the generator.

[0115] S3. Given a log set, use the trained semantic feature extraction model to perform anomaly detection on the log sequence.

[0116] In specific implementation, as a preferred embodiment of the present invention, the specific implementation process of step S3 is as follows:

[0117] S31, log parsing, converting each log event into a specific event template associated with key parameters, selecting the Drain algorithm with good performance in extracting log templates, building a fixed-depth parsing tree based on log data, performing log parsing according to the template extraction rules contained in the tree, and extracting structured log keys from unstructured log events;

[0118] S32, framing the log sequence with Block_id or a sliding window of a fixed size;

[0119] S33, inputting the log entries in the log sequence into BERT to extract sentence vectors;

[0120] S34, determining whether there are any unprocessed log entries in the log sequence, if so, returning to step S33 until all log entries are processed;

[0121] S35, inputting the initial sequence vector linearly concatenated from all sentence vectors into the Encoder layer of the trained optimized BERT model to extract the sequence vector;

[0122] S36, performing anomaly detection on the sequence vector based on MLP, and classifying the sequence vector as normal or abnormal;

[0123] S37, determine whether there are any undetected sequences in the log set, if yes, return to step S32, until all log sequences are detected, otherwise, end the detection.

[0124] Example

[0125] In order to verify the effectiveness of the method of the present invention, a comparative experiment was conducted as follows:

[0126] Experimental environment: The experiments in this paper were all carried out on a server with NVIDIA TESLA V100 32G GPU. Python3.6 was used to build the model based on Keras. The Adam optimizer was used to train and optimize the BERT model. KL divergence and cross entropy functions were used as the loss functions during training. The training process was terminated after 10 iterations.

[0127] Dataset: This paper selects two representative log datasets, HDFS and BGL, and the details of the datasets are shown in Table 1. The HDFS dataset is collected by LogHub from a 203-node cluster on the Amazon EC2 platform. It is a common benchmark data for log-based anomaly detection. It contains a total of 11,175,629 original log messages, and 575,061 sessions are assigned corresponding labels to illustrate their normal and abnormal states. The BGL dataset is collected by LogHub from the BlueGene / L supercomputer system of Lawrence Livermore National Laboratory (LLNL) in Livermore, California, and contains a total of 4,747,963 original log messages, and each log is marked as an alarm or non-alarm message. In the experiment, for each dataset, 5,000 normal log sequences and 5,000 abnormal log sequences are selected from top to bottom according to the timestamp information of the logs. The first 80% are used as training data, and the remaining 20% ​​are used as test data. The sequences of the HDFS dataset are divided by Block_id, and the sequences of the BGL dataset are framed by a sliding window of size 20. Since the labels of the HDFS and BGL datasets are manually labeled, these labels are used as the factual basis for evaluation.

[0128] Table 1 Details of the dataset

[0129]

[0130] Baseline method: This paper selects PCA, IM, DeepLog, LogAnomaly and BERT-based anomaly detection method as the baseline method for comparative experiments.

[0131] PCA detects anomalies in an unsupervised manner by analyzing system logs, counting the event states and occurrence times expressed in the logs, and using the state ratio vector and the practice count vector as the input of the model for anomaly detection; IM is good at mining data rules and mining the invariant components in log events in the log count vector. Those log sequences that violate the invariance are considered anomalies; DeepLog uses LSTM to learn the patterns of normal log sequences and is a log key-based anomaly detection model; LogAnomaly models the log stream as a natural language sequence, uses template2vec to extract semantic information from the log template, and uses LSTM to predict the next possible log, and detects anomalies based on the predicted value.

[0132] Evaluation Metrics: Anomaly detection is a binary classification problem. This paper uses widely used metrics, namely precision, recall, and F1-score, to evaluate the accuracy of OMRLog and various benchmark methods in anomaly detection.

[0133] Accuracy It is the percentage of truly abnormal log sequences among all log sequences judged as abnormal by the model.

[0134] Recall Represents the percentage of all abnormal log sequences that are correctly identified as abnormal log sequences by the model.

[0135] F1 score Represents the harmonic mean of precision and recall. Among them, TP is the number of abnormal log sequences correctly detected by the model. FP is the number of normal log sequences incorrectly identified as abnormal by the model. FN is the number of abnormal log sequences not detected by the model.

[0136] Experimental parameter setting: By repeatedly testing different parameter combinations, the impact of the embedding position of the generator and discriminator in BERT on the detection performance in different data sets is shown in Figure 5 , 6 For HDFS datasets, Figure 5 It can be seen that when the generator is embedded below the 6th layer of BERT and the discriminator is embedded above the 6th layer of BERT, OMRLog has the best anomaly detection performance. When the generator and the discriminator are in the same layer, the gradient lost inside and outside the recognition manifold cannot guide the generator to generate an embedding outside the manifold, resulting in BERT being unable to learn the embedding outside the manifold, thus affecting the effect of anomaly detection. The best effect is achieved by embedding the generator and the discriminator into the 3rd and 12th layers of the BERT model respectively. Similarly, for the BGL dataset, when the generator is embedded below the 7th layer of BERT and the discriminator is embedded above the 7th layer of BERT, OMRLog has the best anomaly detection performance. The best embedding layers for the generator and the discriminator are the 4th and 12th layers.

[0137] This paper experimentally compares OMRLog with four baseline methods and a BERT-based anomaly detection method to verify the accuracy of OMRLog. Figure 7 and Figure 8 The comparison results of this method with the unoptimized BERT and four baseline methods on the HDFS and BGL datasets are shown respectively. Figure 7 and Figure 8It can be seen that on the two data sets of HDFS and BGL, OMRLog has the highest accuracy among the six methods, and the F1-score on the HDGS data set and the BGL data set is 0.98. Although PCA has a high accuracy on the HDFS data set, its recall rate is low, which affects the detection performance, and it is not ideal on the BGL data set. This is because it belongs to anomaly detection methods based on log indexes and does not fully analyze the log semantics; IM and DeepLog have achieved high recall rates on both data sets. For example, the recall rates of IM and DeepLog on the BGL data set are 0.99 and 0.90 respectively, but their high recall rates come at the cost of low accuracy (0.85 and 0.78), and lower accuracy means that more anomalies cannot be detected. Compared with the OMRLog proposed in the present invention, LogAnomaly achieves similar accuracy on the two data sets, but has a lower recall rate, resulting in more false positives. The accuracy of the BERT-based anomaly detection method on both datasets is 0.97. After optimizing BERT using off-manifold embedding, the accuracy of OMRLog is as high as 0.99 and 0.98 respectively. Compared with before optimization, the accuracy is significantly improved. OMRLog performs better in the task of sequence anomaly detection. Therefore, the semantic information obtained from different granularities and the off-manifold embedding discovered by the generator and discriminator play an important role in improving the performance of anomaly detection.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A log sequence anomaly detection method based on out-of-stream regularization, characterized in that: include: S1. Optimize the BERT model and use it as a semantic feature extraction model; In step S1, the method for optimizing the BERT model is as follows: S11, MixUp data augmentation is used as a form of out-of-stream regularization, imposing linear constraints on the model input space outside the data manifold, embedding the generator and discriminator into different generation layers of the BERT model respectively; S12, train the discriminator to detect whether the input embedding is outside the manifold; S13, generating a manifold outer embedding through the manifold outer embedding generated by the generation layer and the label optimized generator; S2. Given a log set χ, train the semantic feature extraction model; The specific implementation process of step S2 is as follows: S21, log parsing, converting each log event into a specific event template associated with key parameters, selecting the Drain algorithm with good performance in extracting log templates, building a fixed-depth parsing tree based on log data, performing log parsing according to the template extraction rules contained in the tree, and extracting structured log keys from unstructured log events; S22, manually selecting a framing method for the log sequence according to the characteristics of the log data set. If there is a Block_id in the data set, the log sequence is divided according to the session id; if not, the log sequence is framed with a sliding window of a fixed size; S23, inputting the log entries in the log sequence into the BERT model to extract sentence vectors; S24, determining whether there are any unprocessed log entries in the log sequence, if so, executing step S23 until all log entries are processed; S25. Linearly concatenate all the sentence vectors in the sequence to obtain the initial sequence vector v seq Input into the Encoder layer of the optimized BERT model; S26, linear interpolation of sequence vectors and their labels based on MixUp; S27. Calculate the classification loss of the log sequence based on the embedding and label in the manifold. Perform binary classification on the log sequence by optimizing the pre-trained weight coefficient of the BERT model. Take the sequence label in the manifold as the output target and use KL divergence as the loss function. The objective function of the log sequence classification is as follows: Among them, ω f Represents the optimization of the trainable parameters of the BERT model pre-training and classifier, represents KL divergence, function f(seq batch ) is to convert seq batch Function to perform classification; S28. Calculate the classification loss and optimize the discriminator based on the embedding and label inside and outside the manifold. Take the label inside the manifold and the label outside the manifold as the output target, and use the binary cross entropy function as the loss function to optimize the discriminator. The objective function is as follows: Among them, ω g ,ω d Represent the trainable parameters of the generator and discriminator respectively, represents the binary cross entropy loss, for The vector output by the j-th layer encoding of the optimized BERT model, For seq batch The vector output by the j-th layer encoding of the optimized BERT model; S29, take the sequence label outside the manifold as the output target, use KL divergence as the loss function to optimize the generator, the objective function is as follows: in, Represents the trainable parameters of the generation layer and subsequent Encoder layer, ω g represents the trainable parameters of the generator; the generator is optimized by the objective function in step S28 and the objective function in step S29, so that the generator generates an outer manifold embedding that is helpful for the log sequence classification task, and the final classification objective function is defined as: in, represents the classification loss of the log sequence, and Represent the losses of training the generator and discriminator to discriminate the off-manifold embedding, respectively. e is responsible for adjusting the two objectives of optimizing the generator. S3. Given a log set, use the trained semantic feature extraction model to perform anomaly detection on the log sequence.

2. The log sequence anomaly detection method based on out-of-stream regularization according to claim 1 is characterized in that: The specific implementation process of step S23 is as follows: S231. Take a log x∈χ in the set χ and split x into a word sequence w∈{0,...,|V} m ; S232, use the tokenizer of the vocabulary V to tokenize the word, where m represents the length of the log entry; S233, map each word into a D-dimensional vector according to the embedding table, then the sentence vector corresponding to the log entry is After the l-layer Encoder, it is converted into 3. The log sequence anomaly detection method based on out-of-stream regularization according to claim 1 is characterized in that: The specific implementation process of step S26 is as follows: Determine the mixing coefficient λ∈[0,1] for linear interpolation, and embed the generator on the i-th layer Encoder of the optimized BERT model. The goal of the generator is to linearly interpolate the context embeddings within the two manifolds to synthesize the embedding outside the manifold; The generator uses a two-layer fully connected network with SoftMax normalization to generate a lower bound α and an interval Δ based on the context embedding within the two manifolds, thus The mixing coefficients are sampled in In order to avoid the huge computational overhead caused by the serial processing of two input sequences, a twin architecture is adopted to share weights on two different inputs to reduce the computational overhead; Optimize the mixing coefficient λ sampled from a uniform distribution, using the reparameterization trick, introducing The mixing coefficient λ is calculated as follows: λ=α+γ×Δ Linear interpolation is performed based on the mixing coefficient λ. The linear interpolation is explained by taking a batch sequence vector and its label as an example, as shown below: in, represents the sequence vector mapped from the 0th layer of the optimized BERT, i.e., the embedding layer, to the i-th layer, is the sequence vector of the batch randomly perturbed from the 0th layer of the optimized BERT to the i-th layer, Represents the sequence vector label of the batch, Represents the sequence vector label after random perturbation of the batch, is the sequence vector after linear interpolation, Represents the label after linear interpolation.

4. The log sequence anomaly detection method based on out-of-stream regularization according to claim 1 is characterized in that: The specific implementation process of step S3 is as follows: S31, log parsing, converting each log event into a specific event template associated with key parameters, selecting the Drain algorithm with good performance in extracting log templates, building a fixed-depth parsing tree based on log data, performing log parsing according to the template extraction rules contained in the tree, and extracting structured log keys from unstructured log events; S32, framing the log sequence with Block_id or a sliding window of a fixed size; S33, inputting the log entries in the log sequence into BERT to extract sentence vectors; S34, determining whether there are any unprocessed log entries in the log sequence, if so, returning to step S33 until all log entries are processed; S35, inputting the initial sequence vector linearly concatenated from all sentence vectors into the Encoder layer of the trained optimized BERT model to extract the sequence vector; S36, performing anomaly detection on the sequence vector based on MLP, and classifying the sequence vector as normal or abnormal; S37, determine whether there are any undetected sequences in the log set, if yes, return to step S32, until all log sequences are detected, otherwise, end the detection.

Images