An unsupervised log anomaly detection method based on pre-trained model

Abstract

The unsupervised log anomaly detection method based on pre-training model disclosed in the present application includes: collecting log data sets generated from different systems; pre-processing the log data sets; calling LSTM and fully connected layer models of the source system log set to build a pre-training model; hybrid model voting pseudo-label: extracting the deep feature vector representation of the unlabeled target system log vector sequence through the LSTM in the pre-training model; the fully connected layer of the pre-training model assisted by the classifier of hierarchical clustering HDBSCAN, voting prediction to screen out the log vector sequence with high confidence as the target system log set with pseudo-label; fine-tuning the pre-training model; inputting the target system log vector sequence into the fine-tuned pre-training model to perform anomaly detection. The technical solution involved in the present invention can utilize the known source system data set when performing log sequence anomaly detection, and ensure the detection effect without using the target system log label, thereby reducing the label marking cost.

Description

Technical Field

[0001] The present application relates to the technical field of log anomaly detection, and more specifically, to an unsupervised log anomaly detection method based on a pre-trained model. Background Art

[0002] With the development of Internet technology, information data traffic is increasing day by day. Logs are an important carrier for recording the operating status and events of computer systems. They can record system status and important events at various key points to help debug system failures and perform root cause analysis. Therefore, system log anomaly detection is crucial.

[0003] Currently, the methods for log anomaly detection are:

[0004] 1. Supervised log anomaly detection method based on transformer structure: This method designs a log sequence encoder and a parameter value encoder to extract the corresponding deep feature representation, and assigns different weights to the two deep feature representations according to the attention mechanism, and uses the label data to train the model to obtain the anomaly probability;

[0005] 2. Unsupervised log anomaly detection method based on LSTM network: This method uses LSTM network to learn the sequence rules of log templates and parameters, and predicts the probability of the next template and parameter value respectively. If it does not appear in the first k predicted templates or parameters, it is considered an anomaly. The two anomaly detection methods are combined to comprehensively judge the anomaly of the entire system.

[0006] 3. Semi-supervised transfer learning method: Use the labeled data of the source system to train a supervised LSTM model as the original model, and use a small amount of labeled data of the target system through transfer learning to fine-tune the parameters of the original model, thereby optimizing the detection performance of the model for anomaly detection. In the prior art, the above three methods are often used for system log anomaly detection, but the above methods also have many disadvantages when used for detection, which affect the detection results or directly make it impossible to detect, including:

[0007] 1. Supervised methods require a large amount of labeled data: In the past, supervised methods required a large amount of labeled data to train the model, but the number of abnormal logs is relatively small. In order to obtain a good classification model, the amount of labeled data is particularly important. Within a certain range, the more labeled data there is, the better the trained detection model will be. The ideal supervised detection model must be trained with a large amount of labeled data, but under current realistic conditions, it is extremely difficult to obtain labeled data.

[0008] 2. Unsupervised detection methods require a large number of normal sequences: In the past, unsupervised learning methods based on LSTM in log anomaly detection required a large number of log sequences composed of normal logs. If there are a small number of abnormal log sequences in the training data, the model performance will drop significantly. Therefore, the ideal model must be composed of a large number of normal log sequences. In other words, this form of unsupervised learning method is actually a supervised learning method that only requires positive samples. In essence, it also requires a large amount of determined labeled data for training, and the difficulty of obtaining labeled data directly limits the normal implementation of this method and affects the detection effect.

[0009] 3. Semi-supervised transfer learning methods cannot be deployed automatically: The classification model of the semi-supervised transfer learning method requires a small amount of target data with real labels to fine-tune the model during model training. When the amount of data is small, the model migration effect is not ideal. In addition, the acquisition of real labels still requires professionals to manually annotate the data set. When the model usage system changes, it cannot be adjusted automatically and still requires personnel to assist in training, which undoubtedly increases labor costs and takes a lot of time.

[0010] Therefore, how to provide an unsupervised log anomaly detection method based on a pre-trained model, which can utilize the known source system data set when performing log sequence anomaly detection, ensure the detection effect without using the target system log label, and reduce the label marking cost, has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0011] To solve the above technical problems, the present application provides an unsupervised log anomaly detection method based on a pre-trained model. When performing log sequence anomaly detection, it can utilize known source system data sets to ensure detection results without using target system log labels, thereby reducing label marking costs.

[0012] The present application provides an unsupervised log anomaly detection method based on a pre-trained model, comprising the following steps: S1, log collection: collecting log data sets generated from different systems, the log data sets including a source system log set and a target system log set, wherein the source system log set already contains labels; S2, log preprocessing: performing log parsing, log vectorization and log serialization processing on the log data set in sequence; S3, building a pre-trained model: calling LSTM and a fully connected layer model, using the labeled source system log vector sequence as the input data of the LSTM and the fully connected layer model, performing model training, and using the trained model as the pre-trained model; S4, hybrid model casting Vote for pseudo-labels: effectively extract the deep feature vector representation of the unlabeled target system log vector sequence through the LSTM; then build a classifier based on hierarchical clustering HDBSCAN to assist the fully connected layer of the original pre-trained model, and vote to predict and select the log vector sequence with high confidence as the target system log set with pseudo-labels; S5, fine-tune the pre-trained model: use the target system log set with pseudo-labels and re-input it into the pre-trained model to fine-tune the parameters of the LSTM model; S6, online anomaly detection: input the target system log vector sequence into the fine-tuned pre-trained model, output the normal log message sequence and the abnormal log message sequence through the fully connected layer to complete the anomaly detection.

[0013] Furthermore, in a preferred embodiment of the present invention, in step S2, the operations of log parsing, log vectorization and log serialization processing are specifically as follows:

[0014] S201, log parsing processing: converting the semi-structured original log messages in the log data set into a structured log template form;

[0015] Among them, log parsing methods include: FT-TREE, FP-TREE, DRAIN and other data parsing methods;

[0016] S202, log vectorization processing: then mapping each original log to a log template, and converting the log template into a log template vector by means of log vectorization;

[0017] S203, log serialization processing: finally, the log data in the source system log set and the target system log set are converted into a labeled source system log vector sequence and an unlabeled target system log vector sequence respectively through a sliding window.

[0018] Furthermore, in a preferred embodiment of the present invention, in step S202, the specific steps of converting the log template vector are:

[0019] Use the vectorization method to convert the words in the log template into vector form, and then obtain the log template vector by summing and averaging the word vectors;

[0020] The vectorization method is the GLOVE method; wherein the GLOVE is a method for generating word vectors, and the specific operation is: using global statistical features and local context features to effectively capture words with similar meanings, so that the word vector similarity of their words is as close as possible to the word meaning similarity between words.

[0021] Furthermore, in a preferred embodiment of the present invention, the cost function of GLOVE is specifically:

[0022]

[0023] where v i 、v j Represent the word vector of word I and word J respectively; b i 、b j There are two different sizes of offsets; X i,j is the number of times word I and word J appear in the co-occurrence matrix; f is the weight function.

[0024] Furthermore, in a preferred embodiment of the present invention, the weight function expression formula is specifically:

[0025]

[0026] In order to prevent x from being too large and causing word co-occurrence frequency overload, a hyperparameter xmax is introduced for constraint; the hyperparameter xmax is set to 100.

[0027] Furthermore, in a preferred embodiment of the present invention, in step S4, the step of voting pseudo labels by the hybrid model is specifically as follows:

[0028] S401, input the unlabeled target system log vector sequence into the pre-trained model, use the LSTM of the pre-trained model to obtain the log depth feature vector, and obtain the pre-trained model label L through the trained fully connected layer a and probability P a ;

[0029] S402, while processing the log depth feature vector based on hierarchical clustering HDBSCAN to generate an unsupervised model label L b and probability P b ;

[0030] S403, filter out a target system log set with a pseudo label by comparing the pre-trained model labels and the unsupervised model labels and their probability values.

[0031] Furthermore, in a preferred embodiment of the present invention, the specific rules for the voting prediction screening are:

[0032] If the HDBSCAN is the same as the fully connected layer prediction, the log vector sequence is deemed to be credible and added to the pseudo-labeled target system log set;

[0033] If the HDBSCAN and the fully connected layer predictions are different, a high threshold TH is set for the classifier of the pre-trained model and the HDBSCAN respectively. a With TH b , by judging whether the predicted probability value is higher than the high threshold: if the predicted probability value is higher than the high threshold, the log vector sequence is added to the target system log set with pseudo labels.

[0034] Furthermore, in a preferred embodiment of the present invention, the high threshold TH a With TH b The preset standard is higher than the threshold standard originally set by the pre-trained model and the HDBSCAN.

[0035] Furthermore, in a preferred embodiment of the present invention, the voting expression formula is specifically:

[0036] L a = =L b ||P a >TH a ||P b >TH b .

[0037] Furthermore, in a preferred embodiment of the present invention, the HDBSCAN is used to construct a hierarchical model between points using a minimum spanning tree; an outlier score is calculated based on the distance of the outlier point, and the score is between 0 and 1. The higher the score, the greater the possibility that the point will become an outlier.

[0038] Compared with the prior art, the present invention provides an unsupervised log anomaly detection method based on a pre-trained model, which comprises the following steps: S1, log collection: collecting log data sets generated from different systems, the log data sets comprising a source system log set and a target system log set, wherein the source system log set already contains a label; S2, log pre-processing: performing log parsing, log vectorization and log serialization processing on the log data set in sequence; S3, building a pre-trained model: calling an LSTM and a fully connected layer model, taking the labeled source system log vector sequence as the input data of the LSTM and the fully connected layer model, performing model training, and taking the trained model as the pre-trained model; S4, mixing Combined model voting pseudo-labels: The deep feature vector representation of the unlabeled target system log vector sequence is effectively extracted through the LSTM; then a classifier based on hierarchical clustering HDBSCAN is constructed to assist the fully connected layer of the original pre-trained model, and the log vector sequence with high confidence is selected as the target system log set with pseudo-labels by voting prediction; S5, fine-tuning the pre-trained model: Using the target system log set with pseudo-labels, re-input it into the pre-trained model to fine-tune the parameters of the LSTM model; S6, online anomaly detection: The target system log vector sequence is input into the fine-tuned pre-trained model, and the normal log message sequence and the abnormal log message sequence are output through the fully connected layer to complete anomaly detection. The present invention provides an unsupervised log anomaly detection method based on a pre-trained model. The key lies in: firstly, extracting the deep feature representation of the log sequence through the pre-trained model; then, using the classifier of the pre-trained model and hierarchical clustering to construct a pseudo-label hybrid voter, assigning pseudo-labels to unlabeled log sequences, and updating the pre-trained model parameters; since the pre-trained model can be trained using a large amount of existing data, the problem of poor model classification effect caused by too little data is avoided; secondly, based on the LSTM pre-trained model, the potential features of the log time series sequence can be effectively extracted, and the common features can be extracted from the log sequence; compared with the method that most methods currently use a certain amount of labeled abnormal data to fine-tune the pre-trained model to update parameters, the method uses an unsupervised clustering method and a pre-trained model classifier to automatically label it in the form of hybrid voting, so as to use it as training data and update the pre-trained model parameters, thereby avoiding the time cost of manually labeling a data set and achieving an ideal anomaly detection effect. Compared with the prior art, when performing log sequence anomaly detection, the method can use a known source system data set to ensure the detection effect without using the target system log label, thereby reducing the label labeling cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A method flow chart of an unsupervised log anomaly detection method based on a pre-trained model provided in an embodiment of the present invention;

[0041] Figure 2 A flowchart of the steps of the unsupervised log anomaly detection method based on a pre-trained model provided in an embodiment of the present invention;

[0042] Figure 3 LSTM model framework diagram provided for an embodiment of the present invention;

[0043] Figure 4 A framework diagram of the LSTM log anomaly detection model provided in an embodiment of the present invention;

[0044] Figure 5 A bar chart showing the model performance evaluation of the wordcount dataset provided in an embodiment of the present invention;

[0045] Figure 6 A line chart of the model performance under different proportions of the WORDCOUNT dataset provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of this application.

[0047] It should be noted that when an element is referred to as being "fixed on" or "set on" another element, it can be directly on the other element or indirectly set on the other element; when an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0048] It should be understood that the terms "length", "width", "up", "down", "front", "back", "first", "second", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0049] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "multiple" and "several" mean two or more, unless otherwise clearly and specifically defined.

[0050] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which this application can be implemented. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in this application without affecting the effects and purposes that can be achieved by this application.

[0051] Compared with the prior art, the present invention provides an unsupervised log anomaly detection method based on a pre-trained model, which comprises the following steps: S1, log collection: collecting log data sets generated from different systems, the log data sets comprising a source system log set and a target system log set, wherein the source system log set already contains a label; S2, log pre-processing: performing log parsing, log vectorization and log serialization processing on the log data set in sequence; S3, building a pre-trained model: calling an LSTM and a fully connected layer model, taking the labeled source system log vector sequence as the input data of the LSTM and the fully connected layer model, performing model training, and taking the trained model as the pre-trained model; S4, mixing Combined model voting pseudo-labels: The deep feature vector representation of the unlabeled target system log vector sequence is effectively extracted through the LSTM; then a classifier based on hierarchical clustering HDBSCAN is constructed to assist the fully connected layer of the original pre-trained model, and the log vector sequence with high confidence is selected as the target system log set with pseudo-labels by voting prediction; S5, fine-tuning the pre-trained model: Using the target system log set with pseudo-labels, re-input it into the pre-trained model to fine-tune the parameters of the LSTM model; S6, online anomaly detection: The target system log vector sequence is input into the fine-tuned pre-trained model, and the normal log message sequence and the abnormal log message sequence are output through the fully connected layer to complete anomaly detection.

[0052] The present invention provides an unsupervised log anomaly detection method based on a pre-trained model, comprising the following steps: S1, log collection: collecting log data sets generated from different systems, the log data sets comprising a source system log set and a target system log set, wherein the source system log set already contains a label; S2, log preprocessing: performing log parsing, log vectorization and log serialization processing on the log data set in sequence; S3, building a pre-trained model: calling an LSTM and a fully connected layer model, taking the labeled source system log vector sequence as the input data of the LSTM and the fully connected layer model, performing model training, and taking the trained model as the pre-trained model; S4, hybrid model casting Vote pseudo-label: The deep feature vector representation of the unlabeled target system log vector sequence is effectively extracted through the LSTM; then a classifier based on hierarchical clustering HDBSCAN is constructed to assist the fully connected layer of the original pre-trained model, and the log vector sequence with high confidence is selected by voting prediction as the target system log set with pseudo-label; S5, fine-tuning the pre-trained model: using the target system log set with pseudo-label, re-input it into the pre-trained model to fine-tune the parameters of the LSTM model; S6, online anomaly detection: input the target system log vector sequence into the fine-tuned pre-trained model, output the normal log message sequence and the abnormal log message sequence through the fully connected layer, and complete the anomaly detection. Since the prior art takes a single system log as the processing object, forms a log vector sequence for detection, and the present invention takes the same domain multi-system log as the processing object, it is necessary to parse and vectorize the logs with different syntax and composition structures. Secondly, the prior art directly uses the real label to fine-tune the pre-trained model, and the present invention automatically labels it by unsupervised clustering method and pre-trained model classifier in the form of mixed voting, so it can reduce the training cost required for training the anomaly detection model. Compared with the prior art, the present invention provides an unsupervised log anomaly detection method based on a pre-trained model. When performing log sequence anomaly detection, it can utilize known source system data sets to ensure detection effect without using target system log labels, thereby reducing label marking costs.

[0053] Specifically, in an embodiment of the present invention, in the step S2, the operations of log parsing, log vectorization and log serialization processing are specifically as follows: S201, log parsing processing: converting the semi-structured original log messages in the log data set into a structured log template form; S202, log vectorization processing: then mapping each original log to a log template, and converting the log template into a log template vector by means of log vectorization; S203, log serialization processing: finally, through a sliding window, converting the log data in the source system log set and the target system log set into a labeled source system log vector sequence and an unlabeled target system log vector sequence, respectively.

[0054] In the log parsing process, each log message contains date, timestamp, level, process type, component (message generation location) and content.

[0055] Take the HADOOP log as an example, 2015-10-18,"18:01:51,963",INFO,MAIN,ORG.APACHE.HADOOP.HTTP.HTTPSERVER2,JETTY BOUND TO PORT 62267. The date is 2015-10-18, the timestamp is 18:01:51,963, the level is INFO, the type of the log process is MAIN, the location where the message is generated is ORG.APACHE.HADOOP.HTTP.HTTPSERVER2, and the content of the log message is JETTY BOUND TO PORT62267.

[0056] Specifically, in an embodiment of the present invention, the log parsing method includes: FT-TREE, FP-TREE, DRAIN and other data parsing methods.

[0057] In the present invention, FT-TREE is used for log parsing; after FT-TREE is used for log parsing, the log template is shown in the last row of Table 1 below, and the semi-structured original log message is converted into a structured log event. The variable part in the log message is deleted, while the constant part in the log message remains unchanged. Each log message has a unique corresponding log template. Taking the above log as an example, the log template is "INFO MAIN ORG APACHE HADOOP HTTPJETT Y BOUND TO PORT".

[0058]

[0059]

[0060] Table 1 Original log and log events

[0061] Specifically, in the embodiment of the present invention, in the step S202, the specific steps of converting the log template vector are: using a vectorization method to convert the words in the log template into a vector form, and then obtaining the log template vector by a word vector summing and averaging method;

[0062] The vectorization method is the GLOVE method; wherein the GLOVE is a method for generating word vectors, and the specific operation is: using global statistical features and local context features to effectively capture words with similar meanings, so that the word vector similarity of their words is as close as possible to the word meaning similarity between words.

[0063] Specifically, in an embodiment of the present invention, the cost function of GLOVE is:

[0064]

[0065] where v i 、v j Represent the word vector of word I and word J respectively; b i , b j There are two different sizes of offsets; X i,j is the number of times word I and word J appear in the co-occurrence matrix; f is the weight function.

[0066] Specifically, in an embodiment of the present invention, the weight function expression formula is specifically:

[0067]

[0068] In order to prevent x from being too large and causing word co-occurrence frequency overload, a hyperparameter xmax is introduced for constraint; the hyperparameter xmax is set to 100.

[0069] Specifically, in step S3, the LSTM is a special RNN that controls the transmission state through the gating state and can selectively record or forget the input information, so it can effectively process sequence data.

[0070] The pre-trained model architecture is a supervised LSTM log anomaly detection model; it is composed of the LSTM model and the fully connected layer model. The LSTM model extracts the deep feature representation of the log vector sequence, and the fully connected layer obtains the anomaly detection result of the log vector sequence. During the pre-training data, the model parameters are trained through the labels of the source system log vector sequence.

[0071] Specifically, in the embodiment of the present invention, in the step S4, the step of voting pseudo-labels by the hybrid model is specifically as follows: S401, inputting the unlabeled target system log vector sequence into the pre-trained model, using the LSTM of the pre-trained model to obtain the log depth feature vector, and obtaining the pre-trained model label L through the trained fully connected layer. a and probability P a ; S402, at the same time, the log depth feature vector is processed based on hierarchical clustering HDBSCAN to generate an unsupervised model label Lb and probability P b ; S403, filter out the target system log set with pseudo labels by comparing the pre-trained model labels and the unsupervised model labels and their probability values.

[0072] Among them, in the LSTM model, the LSTM has a powerful feature extraction capability and can effectively extract the deep feature representation of the log vector sequence. The distribution law of its deep feature representation has a certain correlation with the abnormal state. Therefore, a classifier based on hierarchical clustering HDBSCAN is constructed to assist the fully connected layer of the original pre-trained model, which can screen out log vector sequences with high confidence as pseudo-label log sets for subsequent pre-trained model network parameter updates.

[0073] Specifically, in an embodiment of the present invention, the specific rules for the voting prediction screening are: if the HDBSCAN is the same as the prediction of the fully connected layer, the log vector sequence is deemed to be credible and added to the pseudo-labeled target system log set; if the HDBSCAN is inconsistent with the prediction of the fully connected layer, a high threshold TH is set for the classifier of the pre-trained model and the HDBSCAN respectively. a With TH b , by judging whether the predicted probability value is higher than the high threshold: if the predicted probability value is higher than the high threshold, the log vector sequence is added to the target system log set with pseudo labels.

[0074] For example, when the log vector sequence is predicted to be 1 or 0 by both HDBSCAN and the fully connected layer, it is added to the target system log set with pseudo labels; if the log vector sequence is predicted to be 1 by HDBSCAN and 0 by the fully connected layer, it is determined whether the HDBSCAN prediction probability value is higher than the high threshold. If so, it is added to the target system log set with pseudo labels.

[0075] Specifically, in the embodiment of the present invention, the high threshold TH a With TH b The preset standard is higher than the threshold standard originally set by the pre-trained model and the HDBSCAN.

[0076] Different from the threshold set when the classifier classifies labels, in order to better filter out data with high reliability in the mixed voting stage, a high threshold TH is set for the classifier of the pre-trained model and HDBSCAN respectively. a With TH b These two high thresholds are higher than the thresholds shown in the original classifier. For example, when HDBSCAN classifies labels, the threshold is set to 0.95, while the high threshold TH in mixed voting is bThe quantile is set to 0.98. This method can effectively reduce the noise generated when allocating pseudo labels, improve the robustness of the model, and provide reliable data for subsequent pre-training model updates.

[0077] Specifically, in an embodiment of the present invention, the voting expression formula is:

[0078] L a = =L b ||P a >TH a ||P b >TH b .

[0079] Specifically, in an embodiment of the present invention, the HDBSCAN is used to construct a hierarchical model between points using a minimum spanning tree; an outlier score is calculated based on the distance of the outlier point, and the score is between 0 and 1. The higher the score, the greater the possibility that the point will become an outlier. Since the performance of HDBSCAN will drop sharply as the dimension of the input vector increases when the dimension of the vector exceeds 50, the FASTICA algorithm is used to reduce the dimensionality of the deep feature representation of the log vector sequence.

[0080] Specifically, in step S5, the pre-trained model is fine-tuned. Since the pre-trained model is obtained by training based on the source system log set, although the source system log set has a certain correlation with the target system log set, the model can detect anomalies in the target system log, but the accuracy needs to be improved. Fine-tuning the pre-trained model using a pseudo-label log set can retrain the pre-trained model through a pseudo-labeled log vector sequence without the need for the real label of the target system log set, so as to adjust some parameters of the model, better fit the data of the target system log set, and improve the model effect. Fixing the parameters of the fully connected layer is conducive to improving the robustness of the model to noise. Therefore, in the fine-tuning process, only the parameters of the LSTM are adjusted, and the parameters of the fully connected layer are not changed.

[0081] Example Evaluation:

[0082] In order to evaluate the performance of the method of the present application, the following is a verification explanation by listing Example 1:

[0083] In Example 1 of the present application, the WORDCOUNT and PAGERANK data sets generated by the application of the HADOOP architecture are used. As shown in Table 2 below, some basic information of the WORDCOUNT and PAGERANK data sets is shown; the PAGERANK data set contains 214,670 original logs, of which the number of abnormal log messages is 120,862 and the number of normal log messages is 93,808; the WORDCOUNT data set contains 81,948 original logs, of which the number of abnormal log messages is 4,564 and the number of normal log messages is 77,384. PAGERANK is used as the source system data set, and WORDCOUNT is used as the target system data set. The three performance evaluation indicators commonly used in machine learning are used to evaluate the quality of the model, namely, accuracy, recall, and F1-score.

[0084]

[0085] Table 2 Overview of Hadoop datasets

[0086] Specifically, in Example 1, the anomaly detection method in the present application is compared with LOGTRANSFER and LOGCLUSTERING respectively. LOGTRANSFER is a semi-supervised learning log anomaly detection method based on transfer learning, which uses a log template as input and is constructed with an LSTM model. The LSTM network is used to extract the log sequence pattern, and a transfer learning method in which the source system and the target system share a fully connected network is proposed. The features of the original log sequence are extracted through a sliding window and sent to the LSMT network to output the probability of the log sequence being abnormal. LOGCLUSTERING is a log anomaly detection method based on clustering and unsupervised learning. By converting the log sequence into a vector and clustering it into clusters, a representative sequence is extracted from each cluster, and it is checked whether it represents a previously checked cluster stored in the knowledge base.

[0087] First, we use WORDCOUNT as the test set and compare it with the unsupervised method LOGCLUSTERING and the semi-supervised transfer learning method LOGTRANSFER. We build models by comparing their performance using accuracy, recall, and F1-score. The results are as follows: Figure 5 shown.

[0088] Through the following example results, as an unsupervised learning method, the method provided by this application is significantly better than the baseline LOGCLUSTERING in terms of recall rate and F1 score. When compared with the transfer learning method LOGTRANSFER, although LOGTRANSFER has a higher accuracy rate, the recall rate is low, and the method provided by this application is also slightly better than LOGTRANSFER. It can be seen that our method can still obtain a relatively ideal anomaly detection model without the need for the real label of the target system, which significantly reduces the training cost of the model.

[0089] We divide the WORDCOUNT dataset into ratios of 1:9, 2:8, 4:6, 6:4, and 8:2, and build models based on the divided datasets to evaluate their performance.

[0090] By attaching Figure 6 From the experimental results, it can be seen that the unsupervised log anomaly detection method based on the pre-trained model only needs to train a small data set to obtain an ideal anomaly detection model, which significantly reduces the training cost of the model.

[0091] As described above, the unsupervised log anomaly detection method based on a pre-trained model involved in the embodiment of the present invention first extracts the deep feature representation of the log sequence through the pre-trained model; then, a pseudo-label hybrid voter is constructed using the classifier of the pre-trained model and hierarchical clustering, pseudo-labels are assigned to unlabeled log sequences, and the pre-trained model parameters are updated; since the pre-trained model can be trained using a large amount of existing data, the problem of poor model classification effect caused by too little data is avoided; secondly, the LSTM pre-trained model can effectively extract the potential features of the log time series sequence and extract common features from the log sequence; compared with the method in which most current methods use a certain amount of labeled abnormal data to fine-tune the pre-trained model to update parameters, the method uses an unsupervised clustering method and a pre-trained model classifier to automatically label it in the form of mixed voting, so as to use it as training data and update the pre-trained model parameters, which avoids the time cost of manually labeling data sets and achieves an ideal anomaly detection effect. Compared with the prior art, when performing log sequence anomaly detection, the method can use the known source system data set to ensure the detection effect without using the target system log label, thereby reducing the label labeling cost.

[0092] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An unsupervised log anomaly detection method based on a pre-trained model, characterized in that: The following steps are involved: S1. Log collection: Collect log data sets generated from different systems. The log data sets include source system log sets and target system log sets, where the source system log sets already contain tags. S2, log preprocessing: performing log parsing, log vectorization and log serialization processing on the log data set in sequence; S3. Build a pre-trained model: call the LSTM and fully connected layer models, use the labeled source system log vector sequence as input data of the LSTM and fully connected layer models, perform model training, and use the trained model as the pre-trained model; S4, hybrid model voting pseudo-label: effectively extract the deep feature vector representation of the unlabeled target system log vector sequence through the LSTM; then build a classifier based on hierarchical clustering HDBSCAN to assist the fully connected layer of the original pre-trained model, and vote to predict and select the log vector sequence with high confidence as the target system log set with pseudo-label; S5, fine-tuning the pre-trained model: using the pseudo-labeled target system log set, re-inputting it into the pre-trained model to fine-tune the parameters of the LSTM model; S6. Online anomaly detection: Input the target system log vector sequence into the fine-tuned pre-trained model, output the normal log message sequence and the abnormal log message sequence through the fully connected layer to complete anomaly detection.

2. The unsupervised log anomaly detection method based on a pre-trained model according to claim 1, characterized in that: In step S2, the operations of log parsing, log vectorization and log serialization processing are specifically as follows: S201, log parsing processing: converting the semi-structured original log messages in the log data set into a structured log template form; S202, log vectorization processing: then mapping each original log to a log template, and converting the log template into a log template vector by means of log vectorization; S203, log serialization processing: finally, the log data in the source system log set and the target system log set are converted into a labeled source system log vector sequence and an unlabeled target system log vector sequence respectively through a sliding window.

3. The unsupervised log anomaly detection method based on a pre-training model according to claim 2 is characterized in that: In step S202, the specific steps of converting the log template vector are: Use the vectorization method to convert the words in the log template into vector form, and then obtain the log template vector by summing and averaging the word vectors; The vectorization method is the GLOVE method; wherein the GLOVE is a method for generating word vectors, and the specific operation is: using global statistical features and local context features to effectively capture words with similar meanings, so that the word vector similarity of their words is as close as possible to the word meaning similarity between words.

4. The unsupervised log anomaly detection method based on a pre-training model according to claim 3 is characterized in that: The cost function of the GLOVE model is specifically: where v i 、v j Represent the word vector of word I and word J respectively; b i , b j There are two different sizes of offsets; X i,j is the number of times word I and word J appear in the co-occurrence matrix; f is the weight function.

5. The unsupervised log anomaly detection method based on a pre-trained model according to claim 4 is characterized in that: The weight function expression formula is specifically: In order to prevent x from being too large and causing word co-occurrence frequency overload, a hyperparameter xmax is introduced for constraint; the hyperparameter xmax is set to 100.

6. The unsupervised log anomaly detection method based on a pre-trained model according to claim 1, characterized in that: In step S4, the steps of voting pseudo labels by the hybrid model are specifically as follows: S401, input the unlabeled target system log vector sequence into the pre-trained model, use the LSTM of the pre-trained model to obtain the log depth feature vector, and obtain the pre-trained model label L through the trained fully connected layer a and probability P a ; S402, while processing the log depth feature vector based on hierarchical clustering HDBSCAN to generate an unsupervised model label L b and probability P b ; S403, filter out a target system log set with a pseudo label by comparing the pre-trained model labels and the unsupervised model labels and their probability values.

7. The unsupervised log anomaly detection method based on a pre-trained model according to claim 6, characterized in that: The specific rules for voting prediction screening are: If the HDBSCAN is the same as the fully connected layer prediction, the log vector sequence is deemed to be credible and added to the pseudo-labeled target system log set; If the HDBSCAN and the fully connected layer predictions are different, a high threshold TH is set for the classifier of the pre-trained model and the HDBSCAN respectively. a With TH b , by judging whether the predicted probability value is higher than the high threshold: if the predicted probability value is higher than the high threshold, the log vector sequence is added to the target system log set with pseudo labels.

8. The unsupervised log anomaly detection method based on a pre-training model according to claim 7 is characterized in that: The high threshold TH a With TH b The preset standard is higher than the threshold standard originally set by the pre-trained model and the HDBSCAN.

9. The unsupervised log anomaly detection method based on a pre-trained model according to claim 7, characterized in that: The voting expression formula is specifically: L a ==L b ||P a >TH a ||P b >TH b 。 10. The unsupervised log anomaly detection method based on a pre-training model according to claim 7, characterized in that: The HDBSCAN is used to construct a hierarchical model between points using a minimum spanning tree; an outlier score is calculated according to the distance of the outlier point, and the score is between 0 and 1. The higher the score, the greater the possibility that the point becomes an outlier.

Images