An automatic log parsing method based on RNN neural network and inverted index

Through the method based on RNN neural network and inverted index, the words in the log are divided into template classes and variable classes, which solves the problems of poor generalization and low parsing efficiency in the existing technology, and realizes the different log data sets efficient analysis and powerful generalization capabilities.

CN114969241BActive Publication Date: 2024-11-12WUHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210588074.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-26
Publication Date
2024-11-12
Estimated Expiration
2042-05-26

AI Technical Summary

Technical Problem

The existing log automation analysis technology has problems such as poor generalization and low parsing efficiency, especially in terms of migration and accuracy between different log data sets.

Method used

Using the method based on RNN neural network and inverted index, the words in the log are divided into log template classes and variable classes, and the RNN neural network is used for classification, and template extraction and matching is performed through inverted index, avoiding manual definition of regular expressions. The requirements of the method improve the mobility and accuracy of the method.

Benefits of technology

Improves the generalization ability and parsing accuracy of log parsing, can simply migrate between different log data sets, and efficiently and quickly match unmatched logs without the need for additional manual definition of regular expressions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114969241B_ABST
    Figure CN114969241B_ABST
Patent Text Reader

Abstract

The present invention discloses a log automatic parsing method based on RNN neural network and inverted index. The log automatic parsing technology of the present invention is divided into three stages: the first stage is to classify the words in the log into template class and variable class for the parsed log data, and perform standardization processing to obtain standardized data; the second stage is to learn the processed standardized data based on the RNN neural network model, learn to obtain a neural network classifier that can perform binary classification on the words in the log, and select the best model according to the accuracy, recall rate and F index; the third stage is to input the log message to be parsed according to the obtained RNN classification model, classify the words in the message, retain the template class words in the log message as the log template; and perform template matching on the log message by using the inverted index method. The model in the present invention has a strong generalization ability and can achieve high parsing accuracy on different log data sets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to an automatic log parsing method based on a RNN neural network and an inverted index. Background Art

[0002] Logs are essential in the development and maintenance of many software systems. They record detailed runtime information, allowing developers and maintenance engineers to monitor their systems and analyze abnormal behaviors and errors. The rich information contained in logs and the ubiquity of logs in the system make many system management and diagnostic tasks possible, such as analyzing usage, ensuring application security, identifying performance anomalies, and diagnosing system errors and crashes.

[0003] However, as the scale and complexity of modern software systems increase, the number of logs generated has also exploded. In many cases, traditional manual log inspection has become impractical. Many recent studies and tools used by the industry have used powerful text-based and machine learning tools to analyze solutions to the above problems. Due to the unstructured nature of logs, the first key step is to parse the logs into structured data in preparation for subsequent analysis.

[0004] The technologies for automated log parsing at home and abroad mainly include three types of methods based on frequent pattern mining, clustering, and heuristic methods.

[0005] Based on the frequent pattern mining method, the template of log events can be regarded as a set of unchanging tokens that frequently appear in the log. Frequent pattern mining is a very direct method. SLCT, LFA, and LogCluster use this method. All methods are offline and follow similar parsing steps: traverse the data multiple times, establish frequent pattern item sets each time, cluster the log messages, and extract event templates from each cluster. This method cannot parse the log online.

[0006] In terms of clustering-based methods, event templates form a natural pattern group of log messages. From this perspective, log parsing can be seen as a clustering problem of log messages. LKE, LogSig, and LogMine are offline algorithms that use clustering algorithms, while SHISO and LenMa are online algorithms. The LKE algorithm uses a hierarchical log distance clustering algorithm based on pairwise editing. LogSig is an algorithm based on message signatures that is used to cluster log messages into a predefined number of clusters. LogMine can generate event templates based on a hierarchical clustering algorithm and group log messages from the bottom up. SHISO and LenMa are both online methods that parse logs using similar stream processing methods. For each incoming log message, the parser first calculates the similarity with the existing log template. If a match is successful, the log is classified into the existing template; otherwise, a new message template is created. When using clustering methods, different parameters need to be set for different log data, making the method less versatile.

[0007] In terms of heuristic methods, log messages have some unique features that are different from ordinary text data. AEL, IPLoM, and Drain proposed log parsers based on heuristic methods. In particular, AEL classifies log messages by comparing the frequency of occurrence of constant tokens and variable tokens. IPLoM uses an iterative classification method based on the message length, the location of the tokens, and the corresponding relationship of the log. Drain uses a modified deep tree structure to represent logs and efficiently extracts templates from logs. Heuristic methods require the introduction of more prior knowledge and assumptions about log data, and are not very portable between different log data.

[0008] In the automated log parsing method based on machine learning that is closely related to the present invention, Tong Xiao et al. considered the frequency of words in the log data, used the classic Word2Vec preprocessing method in machine learning to vectorize the words, and then used the clustering method to classify the log templates of the same type into one category. Sasho Nedelkoski et al. proposed a method that uses the popular masked language modeling method in natural language processing to perform automated log parsing using self-supervised deep learning technology. Based on the existing log parser, Weibin Meng et al. used the parsed log data as a training set and trained a word classifier using the SVM method for subsequent update log parsing.

[0009] Currently, most automatic log parsing technologies require an existing log data set as a training set, and the training set and the log data to be parsed belong to the same system. In addition, existing log parsers still require some manually defined regular expressions, and their generalization between different log data sets is not strong. Summary of the invention

[0010] The present invention provides a log automatic parsing method based on RNN neural network and inverted index, which is used to solve or at least partially solve the technical problems of poor generalization and low parsing efficiency existing in the prior art.

[0011] The present invention provides a log automatic parsing method based on RNN neural network and inverted index, comprising:

[0012] S1: Obtain historical log data, preprocess and standardize the historical log data to obtain standardized data, and obtain a training data set based on the standardized data;

[0013] S2: Use the RNN neural network model training data set for learning, obtain multiple models based on the training data set, calculate the classification accuracy, recall rate and F index of each model, and obtain the best model based on the classification accuracy, recall rate and F index as the classifier;

[0014] S3: After pre-processing the log data to be parsed, the log data is input into the classifier for classification to obtain a classification result, which includes a log template class and a variable class; then, the data structure of the inverted index and the classification result obtained by the classifier are used to perform template extraction and template matching on the log data to be parsed to obtain a standardized log template, wherein the data structure of the inverted index is a Python dictionary; after the parsed log data is assigned a unique template ID, the template word in the log is used as the key and the template ID as the value, and is added to the Python dictionary in the form of a key-value pair, wherein the template word is a word classified as a log template class by the classifier.

[0015] In one embodiment, step S1 includes:

[0016] S1.1: Extract the words that make up the log messages from the historical log data;

[0017] S1.2: Classify the words constituting the log message extracted in step S1.1 into log template class and variable class, and manually annotate the two types of data, wherein the log template class is the part manually defined in the log generation process, and the variable class is the part generated by the program in the log generation process;

[0018] S1.3: Standardize the annotated words to obtain standardized data as the training data set.

[0019] In one embodiment, a test data set is obtained according to the standardized data, and step S2 includes:

[0020] S2.1: Vectorize the data in the training dataset using One-Hot encoding to obtain the vector representation of each word;

[0021] S2.2: Input the vector representation of the words in the training dataset into the pre-built RNN neural network for training;

[0022] S2.3: Use the test data set to test the trained RNN neural network model, and obtain the best model as the classifier based on the three criteria of classification accuracy, retrieval rate and F index.

[0023] In one embodiment, step S3 pre-processes the log data to be parsed and inputs it into the classifier for classification, including:

[0024] S3.1: Perform internal partial extraction on the log data to be parsed to extract the words that make up the log message;

[0025] S3.2: The extracted words are vectorized using One-Hot encoding and input into the classifier obtained in step S2 to obtain a classification result: log template class or variable class.

[0026] In one implementation, step S3 uses the data structure of the inverted index and the classification result obtained by the classifier to perform template extraction and template matching on the log data to be parsed to obtain a standardized log template, including:

[0027] S3.3: For a log message that has not been parsed, when a word in the log message is classified as a log template class by the classifier, the template ID number corresponding to the word is queried in the Python dictionary;

[0028] S3.4: Take the intersection of the template ID numbers corresponding to all template words to obtain a unique template ID as the matching template ID of the current log message.

[0029] The main features and innovations of this application compared to the prior art are:

[0030] 1. It is proposed to classify the words in the log into log template class and variable class, and transform the log parsing problem into a word classification problem, making it possible to use deep learning technology.

[0031] 2. It is proposed to use log data of different types from the log to be parsed as the training set, to mine the similarities and differences between template classes and variable classes in different data sets from the character level, and to use the RNN neural network to describe this feature.

[0032] 3. The method of the present invention does not use the regular expression required in most methods, making the method more portable. At the same time, the inverted index technology can match unmatched logs with higher accuracy, improving the accuracy of the method.

[0033] The log automatic parsing technology based on RNN neural network and inverted index created by the present invention has more powerful generalization ability than the existing log parsing technologies at home and abroad. The use of RNN neural network as a classifier eliminates the need for additional manual definition of regular expressions during the parsing process, and enables the method to be simply migrated between different log data sets. The use of inverted index technology can efficiently and quickly match unmatched logs, thereby improving the parsing accuracy of log parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 Schematic diagram of the RNN neural network training process in an embodiment of the present invention;

[0036] Figure 2 It is a flow chart of template extraction using RNN neural network and inverted index method in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention proposes a log template extraction method based on RNN neural network and inverted index, which is used to parse unstructured logs into structured template forms. Most of the traditional log parsing methods need to parse the log content based on manually defined regular expressions, and different parameters need to be set for different log data sets to ensure the parsing accuracy, which leads to the traditional methods having weak mobility and insufficient generalization ability on different log data. The present invention divides the words in the log into log template class and variable class, reveals the difference between the two at the character level, and uses the RNN neural network model as a classifier, which can improve the mobility of the method, and can obtain higher accuracy on different log data sets without manually modifying the defined regular expressions. Using the inverted index technology, the log parsing process can be accelerated and the parsing efficiency can be improved.

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The embodiment of the present invention provides a log automatic parsing method based on RNN neural network and inverted index, comprising:

[0040] S1: Obtain historical log data, preprocess and standardize the historical log data to obtain standardized data, and obtain a training data set based on the standardized data;

[0041] S2: Use the RNN neural network model training data set for learning, obtain multiple models based on the training data set, calculate the classification accuracy, recall rate and F index (F-measure) of each model, and obtain the best model based on the classification accuracy, recall rate and F index as the classifier;

[0042] S3: pre-processing the log data to be parsed and inputting it into the classifier for classification to obtain a classification result, which includes a log template class and a variable class;

[0043] S3: After pre-processing the log data to be parsed, the log data is input into the classifier for classification to obtain a classification result, which includes a log template class and a variable class; then, the data structure of the inverted index and the classification result obtained by the classifier are used to perform template extraction and template matching on the log data to be parsed to obtain a standardized log template, wherein the data structure of the inverted index is a Python dictionary; after the parsed log data is assigned a unique template ID, the template word in the log is used as the key and the template ID as the value, and is added to the Python dictionary in the form of a key-value pair, wherein the template word is a word classified as a log template class by the classifier.

[0044] In the specific implementation process, after the classification by the classifier in step S3, the words classified as log template class are retained as template words, and a standardized log template is obtained, and then the standardized log template is used to match subsequent log messages.

[0045] The present invention proposes to use the RNN neural network model as a classifier, and classify the words in the log into log template class and variable class by training the neural network. Figure 1 As shown, first select an existing log data set (historical log data). It should be noted that the existing log data can be log data of the same type as the log to be parsed, or it can be a different log data set. By manually labeling the log data, the words in the log data are divided into template class and variable class. Next, the words are vectorized using the One-Hot encoding method and sent to the RNN neural network for training. According to the model's accuracy, F value and other indicators, the optimal model is selected as the classifier for the subsequent steps.

[0046] The present invention proposes to combine the RNN neural network model with the inverted index to extract the template from the original log. The process is shown in Figure 2 .

[0047] ① Original log preprocessing: irrelevant information in the log message is processed in a predefined way, and only the log message content is retained, that is, the words that constitute the log message content are extracted.

[0048] ② Classification of words in logs: Use the trained RNN neural network to vectorize the words in the logs and input them into the RNN model to obtain a classification result.

[0049] ③ Template matching: Based on the results of word classification in a log, use the inverted index to find the matching log template; if the match is successful, a template ID is assigned; if the match fails, enter the template extraction stage to extract the template.

[0050] ④ Template extraction: If there is no template match in the current log, the template extraction phase is entered. According to the classification results of the RNN model, the words of the template class are retained and the variable class is replaced with '<*>'.

[0051] ⑤ Assign template ID: After template matching and template extraction, the log is finally assigned a template ID number as the final parsing result.

[0052] The present invention analyzes the log automatic parsing technology. The parsing process uses a pre-trained RNN neural network classifier to classify the log message words to be parsed, and uses the inverted index technology to match the unparsed log data. It should be noted that this method does not require the definition of regular expressions, and the log data used to train the RNN neural network can be unrelated to the log data to be parsed.

[0053] In one embodiment, step S1 includes:

[0054] S1.1: Extract the words that make up the log messages from the historical log data;

[0055] S1.2: Classify the words constituting the log message extracted in step S1.1 into log template class and variable class, and manually annotate the two types of data, wherein the log template class is the part manually defined in the log generation process, and the variable class is the part generated by the program in the log generation process;

[0056] S1.3: Standardize the annotated words to obtain standardized data as the training data set.

[0057] In the specific implementation process, the log data of the training data set can be logs belonging to the same system as the logs to be parsed, or log data belonging to different systems. After manual labeling, the words in the log data are divided into log template class and variable class.

[0058] In one embodiment, a test data set is obtained according to the standardized data, and step S2 includes:

[0059] S2.1: Vectorize the data in the training dataset using One-Hot encoding to obtain the vector representation of each word;

[0060] S2.2: Input the vector representation of the words in the training dataset into the pre-built RNN neural network for training;

[0061] S2.3: Use the test data set to test the trained RNN neural network model, and obtain the best model as the classifier based on the three criteria of classification accuracy, retrieval rate and F index.

[0062] In the specific implementation process, during the model tuning process, the parameter values ​​of different hyperparameters, such as the hidden layer size, the learning rate, etc., can be set, the model can be tested with a test data set, and the RNN neural network model with the best effect is selected according to the standard as the classifier for the subsequent steps. It should be noted that the method of the embodiment of the present invention focuses on the difference in character composition between the log template class and the variable class in the log data, and uses a neural network to represent this difference, and this difference is universal among different log data sets. Therefore, in step S1, different types of log data can be used as training sets for the neural network.

[0063] In one embodiment, step S3 pre-processes the log data to be parsed and inputs it into the classifier for classification, including:

[0064] S3.1: Perform internal partial extraction on the log data to be parsed to extract the words that make up the log message;

[0065] S3.2: The extracted words are vectorized using One-Hot encoding and input into the classifier obtained in step S2 to obtain a classification result: log template class or variable class.

[0066] In one implementation, step S3 uses the data structure of the inverted index and the classification result obtained by the classifier to perform template extraction and template matching on the log data to be parsed to obtain a standardized log template, including:

[0067] S3.3: For a log message that has not been parsed, when a word in the log message is classified as a log template class by the classifier, the template ID number corresponding to the word is queried in the Python dictionary;

[0068] S3.4: Take the intersection of the template ID numbers corresponding to all template words to obtain a unique template ID as the matching template ID of the current log message.

[0069] Specifically, the Python dictionary is pre-built. For each log message in the historical log data, if the word is judged as a variable class by the classifier, it is replaced by the wildcard '<*>'; if it is judged as a template class, it is retained in the log message. The log is parsed in the form of a template. A unique template ID number that increments from 0 is assigned to each retained template word in the log, and the template word is used as the key and the template ID as the value, and they are added to a Python dictionary in the form of a key-value pair. For the log messages that have not been parsed, after the words are judged as template classes, the corresponding template ID number in the Python dictionary is queried for each word judged as a template class, and the intersection of the template ID numbers corresponding to these template words is taken to obtain a unique template ID as the matching template ID of the current log message.

[0070] During the specific implementation process, template matching is first performed. If the current log matches an existing template, a template ID is directly assigned. If no existing template is matched, the words classified as template classes are retained in the log template according to the classification results of the classifier, and the variable words are replaced with wildcards '<*>'. Finally, a log template is obtained and a unique template ID is assigned. For the template words retained in the template, the template ID number and the template words in the log are added to the inverted index in the form of a key-value pair. When performing template matching, according to the classification results of the classifier, if a word is judged to be a template word, its corresponding template ID is first queried in the inverted index; this operation is performed for all words in a log that are judged to be template words, and the intersection of all the obtained template IDs is taken to finally obtain a unique template ID, which is the template ID number of the successful match and is assigned to the current log.

[0071] Compared with the prior art, the beneficial effects of the present invention are:

[0072] 1. It is proposed to classify the words in the log into template class and variable class, and transform the log parsing problem into a word classification problem, making it possible to use deep learning technology.

[0073] 2. It is proposed to use log data of different types from the log to be parsed as the training set, to mine the similarities and differences between template classes and variable classes in different data sets from the character level, and to use the RNN neural network to describe this feature.

[0074] 3. This method does not use the regular expression required by most methods, making the method more portable. At the same time, the inverted index technology can match unmatched logs with higher accuracy, improving the accuracy of the method.

[0075] The log automatic parsing technology based on RNN neural network and inverted index created by the present invention has more powerful generalization ability than the existing log parsing technologies at home and abroad. The use of RNN neural network as a classifier eliminates the need for additional manual definition of regular expressions during the parsing process, and enables the method to be simply migrated between different log data sets. The use of inverted index technology can efficiently and quickly match unmatched logs, thereby improving the parsing accuracy of log parsing.

[0076] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A log automatic parsing method based on RNN neural network and inverted index, characterized in that: include: S1: Obtain historical log data, preprocess and standardize the historical log data to obtain standardized data, and obtain a training data set based on the standardized data; S2: Use the RNN neural network model training data set for learning, obtain multiple models based on the training data set, calculate the classification accuracy, recall rate and F index of each model, and obtain the best model based on the classification accuracy, recall rate and F index as the classifier; S3: pre-process the log data to be parsed and input it into the classifier for classification to obtain classification results, which include log template classes and variable classes; then use the data structure of the inverted index and the classification results obtained by the classifier to perform template extraction and template matching on the log data to be parsed to obtain a standardized log template, wherein the data structure of the inverted index is a Python dictionary; after the parsed log data is assigned a unique template ID, the template word in the log is used as the key and the template ID as the value, and is added to the Python dictionary in the form of a key-value pair, wherein the template word is a word classified as a log template class by the classifier; Wherein, step S1 comprises: S1.1: Extract the words that make up the log messages from the historical log data; S1.2: Classify the words constituting the log message extracted in step S1.1 into log template class and variable class, and manually annotate the two types of data, wherein the log template class is the part manually defined in the log generation process, and the variable class is the part generated by the program in the log generation process; S1.3: Standardize the annotated words to obtain standardized data as training data set; Step S3 uses the data structure of the inverted index and the classification results obtained by the classifier to perform template extraction and template matching on the log data to be parsed to obtain a standardized log template, including: S3.3: For a log message that has not been parsed, when a word in the log message is classified as a log template class by the classifier, the template ID number corresponding to the word is queried in the Python dictionary; S3.4: Take the intersection of the template ID numbers corresponding to all template words to obtain a unique template ID as the matching template ID of the current log message.

2. The method according to claim 1, characterized in that The test data set is obtained according to the standardized data, and step S2 includes: S2.1: Vectorize the data in the training dataset using One-Hot encoding to obtain the vector representation of each word; S2.2: Input the vector representation of the words in the training dataset into the pre-built RNN neural network for training; S2.3: Use the test data set to test the trained RNN neural network model, and obtain the best model as the classifier based on the three criteria of classification accuracy, retrieval rate and F index.

3. The method according to claim 1, characterized in that Step S3 pre-processes the log data to be parsed and inputs it into the classifier for classification, including: S3.1: Perform internal partial extraction on the log data to be parsed to extract the words that make up the log message; S3.2: The extracted words are vectorized using One-Hot encoding and input into the classifier obtained in step S2 to obtain a classification result: log template class or variable class.