Robust-based log template extraction update method and system

Abstract

The application discloses a kind of based on robust log template extraction updating method and system, comprising: step 1, the log to be pattern recognition is obtained, and its pre-treatment is obtained log sequence;Step 2, the log sequence is extracted to obtain pseudo log template, search is carried out with the pseudo log template as index to obtain all templates under the index, and it is used as candidate template;Step 3, based on similarity and length difference, log sequence and candidate template are matched;Step 4, according to the result matched to the module in template library is updated.The application can quickly automatically pattern recognition to log, generates log template, and the identification precision is high, and pattern recognition result is not strongly dependent on word segmentation, and robustness is stronger.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of log pattern recognition, and particularly relates to a robust log template extraction and updating method and system. BACKGROUND

[0002] For the banking industry, due to strict regulation, most bank log data has the characteristics of detailed log data and high log quality. However, "too much information equals no information", if log information cannot be effectively managed, rich and standardized log data will not play its important role. Automatic implementation of pattern recognition through log pattern recognition algorithm is an important link to play the role of log data, and helps enterprises to realize "using data to speak, using data to manage, and using data to make decisions".

[0003] 1) Use data to speak: In the business system, the number of logs generated daily is often in the order of magnitude of billions, while the code for printing these logs is often only a few tens or a few hundred. Through log pattern recognition, and according to the log template, the log is compressed and merged, which can effectively reduce the information entropy of log data, realize the perception of business personnel on log data, and then count the number of logs or field proportion of each template, and convert log data into effective information.

[0004] 2) Use data to manage: Logs are a manifestation of system behavior, and logs of the same template represent the same event in the system behavior process. After log pattern recognition, the number of occurrences and occurrence rules of each event in the system can be understood, so that business personnel can have a number in mind, and then help business personnel to reasonably allocate resources, find blind spots in operation and maintenance management, and improve operation and maintenance management level.

[0005] 3) Use data to make decisions: Traditional log use often only occurs after a fault, and then traces through logs. However, through log pattern recognition, it is possible for business personnel to understand the behavior of the system according to the logs, and a comprehensive understanding of the system can assist enterprises to make more reasonable decisions.

[0006] The focus of log pattern recognition is to extract the log template, which is the invariant part in the log printing code. For example, if the code for printing the log is Log.warning("Node{}is unconnected".format(Node.get_name())), the template of the log printed by the code is Node<*>is unconnected, where <*> represents a placeholder for a parameter. Log pattern recognition can also be understood as the inverse printing process of the log. In recent years, the academic community has proposed a variety of automatic log pattern recognition algorithms. From the implementation method, these pattern recognition methods can be roughly divided into four types: clustering-based log pattern recognition algorithm, frequent item mining-based log pattern recognition algorithm, heuristic rule-based log pattern recognition algorithm, and deep learning-based log pattern recognition algorithm. At present, the deep learning-based log pattern recognition algorithm is still in the research stage due to the slow efficiency, the need for labeling for some algorithms, and the lack of wide applicability, and cannot be applied to industrial scenarios. For other algorithms, the best algorithms in the standard data set LogPai in the field of log pattern recognition are Drain[2], IPLOM[6], Spell[4], and AEL[8]. Among these algorithms, Drain[2], IPLOM[6], and AEL[8] all take the length of the log as an important standard for log grouping. However, log segmentation has a great impact on the length of the log. For example, the log Warning:123.12.22.12:22is unconnected and the log Warning:123.12.22.12isunconnected both have the template Warning:<*>is unconnencted, but if segmented according to the colon and space, the lengths of the two logs will be 9 and 7, respectively. In the case of a large number of logs and multiple log types, this situation where the lengths of logs with the same template differ greatly due to improper segmentation is likely to occur, resulting in tens of thousands of templates being recognized by the pattern recognition, and the pattern recognition result is not usable. SUMMARY

[0007] An object of the present application is to provide a robust log template extraction and update method to overcome the shortcomings of the prior art. The present application can quickly perform automatic log pattern recognition and generate log templates with high recognition accuracy. The pattern recognition result is not strongly dependent on segmentation, and the robustness is strong.

[0008] To solve the above technical problems, the present application adopts the following technical solutions:

[0009] A robust log template extraction and update method includes the following steps:

[0010] Step 1, obtaining the log to be pattern-recognized, and pre-processing the log to obtain a log sequence;

[0011] Step 2, refining the log sequence to obtain a pseudo log template, searching based on the pseudo log template to obtain all templates under the index, and taking the templates as candidate templates;

[0012] Step 3, matching the log sequence with the candidate templates based on similarity and length difference;

[0013] Step 4, updating the modules in the template library according to the matching result.

[0014] Further, step 1 specifically includes:

[0015] Step 1.1, extracting the content part of the obtained log;

[0016] Step 1.2, identifying the special characters in the log content part by regular or named entity recognition method and replacing them with specific symbols;

[0017] Step 1.3, segmenting the log content processed in step 1.2 to obtain a log sequence.

[0018] Further, the way of obtaining the pseudo log template in step 2 is:

[0019] Constructing a keyword dictionary, traversing each word in the log sequence, if the lowercase form of the word is in the keyword dictionary, extracting the word and connecting it in order by a connector to obtain a pseudo log template.

[0020] Further, the keyword dictionary includes key words describing system behavior, computer field proper nouns, verbs, and words representing system state.

[0021] Further, the matching method in step 3 is:

[0022] Calculating the length difference between the log sequence and the candidate template, if the length difference is greater than a threshold, the candidate template is discarded;

[0023] If the length difference is less than the threshold, calculate the Jaccard similarity of the two, and select the candidate template with the greatest similarity to the log sequence from all candidate templates with length difference less than the threshold, and compare the Jaccard similarity of the candidate template and the log sequence with the similarity threshold, if greater than the similarity threshold, it is considered that the log is successfully matched to a suitable template, i.e. the candidate template is the template of the log and is output; if less than the similarity threshold, it is considered that the log is not matched to a suitable template, then the log sequence of the log is added to the template library as a new template, and stored with the pseudo log template of the log as an index.

[0024] Further, the Jaccard similarity calculation formula is:

[0025]

[0026] In the formula, seq1, seq2 are log sequences, and set() is a function of converting log sequences into sets.

[0027] Further, the length difference calculation formula is:

[0028]

[0029] In the formula, len(seq1) represents the length of log sequence 1, len(seq2) represents the length of log sequence 2, and δ is a fixed value set.

[0030] Further, the step 4 update method is:

[0031] If the log is successfully matched to a suitable template in step 3, the matched template needs to be updated according to the log and stored in the template library;

[0032] Otherwise, the update is performed in the following way:

[0033] The set of same part words of the template with the largest Jaccard similarity and the log is found:

[0034] common_part = set(seq1) ∩ set(seq2);

[0035] In the formula, seq1, seq2 are log sequences, and set() is a function of converting log sequences into sets.

[0036] Then, the sequence of the template is traversed, if the word is in common_part, the word is retained, and if the word is not in common_part, the wildcard is replaced by the word.

[0037] Another object of the present application is to provide a system based on the above-mentioned robust log template extraction update method, comprising:

[0038] A log acquisition processing module is used to acquire logs to be pattern recognized, and to obtain log sequences by preprocessing.

[0039] A candidate template acquisition module is used to obtain pseudo log templates by refining the log sequences, and to search all templates under the index by taking the pseudo log templates as the index, and to take the templates as candidate templates.

[0040] A module matching module is used to match the logs to be pattern recognized with the candidate templates based on similarity and length difference.

[0041] A template library updating module is configured to update the modules in the template library according to the matched results.

[0042] Compared with the prior art, the present application has the following beneficial effects: the present application clusters logs based on the Jaccard similarity, and the algorithm has strong robustness and can achieve very good results in complex business scenario logs; the present application proposes a grouping strategy based on pseudo-log templates, and improves the accuracy and efficiency based on the length similarity; and it is verified through experiments that the traditional log pattern recognition algorithm based on strong assumptions can achieve good results in the LogPai public dataset, but these algorithms are not applicable in complex bank heterogeneous logs, while the present application can achieve the best accuracy in the public dataset and very good results in complex bank heterogeneous logs, and while giving consideration to the accuracy, the calculation efficiency of the present application is 4 times faster than the Drain and other classic algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of the robust log template extraction and updating method of the embodiments of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0045] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0046] The present application will be further described below with reference to specific embodiments, but is not limited to the embodiments.

[0047] As shown in Figure 1 The embodiments of the present application disclose a robust log template extraction and updating method, which comprises the following steps:

[0048] Step 1: obtaining logs to be pattern-recognized, and pre-processing the logs to obtain log sequences; this step specifically comprises the following sub-steps:

[0049] Step 1.1, extract the content part of the log to be identified; a classic unstructured log, the content of which includes a header part and a content part, as shown in the following examples, the first part of the log 1 and log 2 is the content automatically supplemented by the defined logger, which may contain the time, level, location and other information of the log, this part is the header part of the log, and the remaining part is the content part, which is determined by the code and corresponding variables of the printed log. Usually, the format of the content of the header part is fixed, so the header title such as logger.info representing the date can be removed by specific rules such as some regular expressions, leaving the content part of the log, which is followed by subsequent pattern recognition. The form of the log is as follows:

[0050] logger.info("Exceeded the maximum number of ssh connections:{}".format(connect_num)).

[0051] Log 1: 2022-10-20 19:23:32 [Info] ssh 10.1.1.122: Exceeded the maximum number of ssh connections: 500;

[0052] Log 2: 2022-10-20 19:24:32 [Info] ssh 10.1.1.123: Exceeded the maximum number of ssh connections: 1000.

[0053] Step 1.2, identify the special characters such as IP, URL, time, etc. in the extracted log content part by regular or named entity recognition method, and replace them with <ip> 、 <url> 、 <time>Special symbols, according to experience, these special characters in the log must be variable, therefore, in advance, the replacement can improve the accuracy of subsequent steps, and for the content of the url type, if not in advance, the replacement may be in the segmentation of the url content into multiple words, very affect the accuracy of pattern recognition.

[0054] Step 1.3, by some simple separator such as space, comma, colon, etc. to the log obtained in step 1.2 is segmented, get the log sequence; for Chinese log, can use Jieba to Chinese word segmentation, or for Chinese word segmentation, a Chinese character into a word.

[0055] Step 2, the log sequence is refined to obtain the pseudo log template, and the pseudo log template is used as the index to search for all templates under the index, and the templates are used as candidate templates;

[0056] Log is the embodiment of system behavior, in most cases, part of the words in the log can summarize the system behavior corresponding to the log, such as the log "Exceeded the maximum number of ssh connections:1000", only need "Exceeded ssh connections" three words, experienced operation and maintenance experts can understand that the log is talking about the problem of too many ssh connections. This simplified summary extracted from the log is called "pseudo log template". In order to extract the pseudo log template from the log, a keyword dictionary is first constructed, which contains some key words describing system behavior, including some computer field proper nouns, and some important verbs such as "exceed", "run", and some system state words such as "success", "failed". For the log to be pattern recognized, traverse each word in the segmented log sequence, if the lower case form of the word is in the keyword dictionary, extract the word and connect it in order by the connector "->" to form the "pseudo log template". For example:

[0057] Log sequence:

[0058] [Exceeded,,the,,maximum,,number,,of,,ssh,,connections,:,1000]

[0059] Keyword dictionary: exceed, exceeded, exceeded, ssh, connection, connections

[0060] Pseudo-log template: Exceeded->ssh->connections

[0061] After obtaining the pseudo-log template of the log, search in the template library with the pseudo-log template as the index, and all templates under the index obtained by the search are used as candidate templates for the next matching.

[0062] Step 3, based on the similarity and length difference, match the log sequence with the candidate templates;

[0063] This embodiment analyzes the distribution of length difference and Jaccard similarity in different log sources, and concludes that length difference and Jaccard similarity can be used to distinguish logs of different templates. Therefore, based on length difference and Jaccard similarity, we define the following similarity formula:

[0064] \begin{align*}\begin{split}sim(seq1,seq2)

[0065] =\left\{\begin{array}{ll}0,&\Delta len(seq1,seq2)

[0066] >s1\\sim_{Jaccard}(set(seq1),set(seq2)),&otherwise

[0067] \end{array}\right.\end{split}\end{align*}

[0068] Where seq1, seq2 are log sequences, s1 is the length difference threshold of the hyperparameter, and Δlen(seq1, seq2) is the length difference of the two logs, which is defined as

[0069]

[0070] Where len(seq1) is the length of log sequence 1, len(seq2) is the length of log sequence 2, and δ is a fixed value, which is 5 by default, which can reduce the sensitivity of the length difference formula to small log lengths;

[0071] sim Jaccard (set(seq1),set(seq2)) is the improved Jaccard similarity, which is defined as:

[0072]

[0073] Where set() is a function that converts a sequence to a set, which can remove duplicate elements in the sequence.

[0074] After a series of candidate templates are found in step 2, the similarity between the template and the log is compared according to the above-mentioned Jaccard similarity formula, and the template with the largest similarity is selected. In actual calculation, the length difference between the log sequence and the candidate template can be calculated first. If the length difference is greater than the threshold S1, the early abandonment strategy is adopted, and the Jaccard similarity between the two does not need to be calculated, which can greatly improve the calculation efficiency. If the length difference between the two is less than the threshold S1, the Jaccard similarity between the two is calculated, and the candidate template with the largest Jaccard similarity with the log sequence is selected from the series of candidate templates with the length difference less than the threshold S1. Then the maximum Jaccard similarity is compared with the Jaccard similarity threshold S2. If the maximum Jaccard similarity is greater than the Jaccard similarity threshold S2, it is considered that a suitable template is successfully matched for the log, that is, the candidate template with the largest Jaccard similarity with the log sequence is the template of the log, and it is output. If the Jaccard similarity between the candidate template with the largest Jaccard similarity and the log is less than the Jaccard similarity threshold S2, it is considered that a suitable template is not matched for the log, and the log sequence of the log is added to the template library as a new template, which is stored with the pseudo-log template of the log as an index.

[0075] Step 4, update the modules in the template library according to the matching result.

[0076] If a suitable template is successfully matched for the log in step 3, the template needs to be updated according to the log, stored in the template library, and the template library is enriched. Otherwise, the template with the largest Jaccard similarity will not be able to abstract and summarize the log, and the update method is as follows:

[0077] Find the set of common part words between the template with the largest Jaccard similarity and the log:

[0078] common_part = set(seq1) ∩ set(seq2)

[0079] Then, the sequence of the template is traversed. If the word is in common_part, the word is retained. If the word is not in common_part, the word is replaced with a wildcard < >, as shown in the following example:

[0080] Template: [Node, 001, is, unconnected]

[0081] Log sequence: [[Node, 002, is, unconnected]]

[0082] common_part: {Node, is, unconnected}

[0083] Updated template: [Node, <*>, is, unconnected]

[0084] This update method is a simplified update method, and there may be some variables that have not been changed to wildcards, but in most templates, this update method can get the correct template.

[0085] To illustrate the effect of the present embodiment, the classical Drain algorithm, Spell algorithm and IPLOM algorithm are selected as Baseline for comparison, experiments are performed on the LogPai public dataset and business log dataset, and evaluation indicators are used to evaluate various algorithms. Log pattern recognition is also a clustering process, and logs of the same template can be considered to belong to the same cluster. Therefore, the result of log pattern recognition can be evaluated by the clustering result. Common evaluation indicators for log pattern recognition include F1 value:

[0086]

[0087] Among them:

[0088]

[0089]

[0090] Among them: true positive (TP) indicates that two logs of the same template are assigned to the same cluster; true negative (TN) indicates that two logs of different templates are assigned to different clusters; false positive (FP) indicates that two logs of different templates are assigned to the same cluster; false negative (FN) indicates that two logs of the same template are assigned to different clusters.

[0091] Both the LogPai public dataset and the business log were tested on the logs, and were compared with the classical Drain algorithm, Spell algorithm and IPLOM algorithm. On the LogPai public dataset, the performances of the four algorithms were good, but the present embodiment still achieved the best result, and the result is as follows:

[0092]

[0093]

[0094] In the three data sets of the business log, the four algorithms perform well on the logs of platform 1 and platform 2, among which, the method of the embodiment has the best effect, and can obtain F1 of 0.9661 and 0.9849, while the effect of Spell is the worst, and can obtain F1 of 0.2546 and 0.8135 respectively, but on the log of platform 3, the performance of Drain algorithm, Spell algorithm and IPLOM algorithm is not ideal, and can only obtain F1 of about 0.5, but the method of the embodiment can still obtain F1 of 0.9014. This is because on the log of platform 3, there are more structured information, stack information, query information and the like, and the log length is much higher than that of the LogPai public data set and the log data set of platform 1 and 2. After the complexity of the log scene, the strong hypothesis of Drain and IPLOM algorithm does not work, and the simplified calculation of LCS of Spell also loses effect due to the complexity of the data, and obtains a poor result, and the result is as follows:

[0095]

[0096] The embodiment obtains very good results in the three data sets, and can guarantee that the F1 value of pattern recognition is above 0.9, which is 0.1-0.7 higher than that of Drain, IPLOM and Spell. The reason why the embodiment can obtain such good results is not only due to the selection of the robust similarity formula, but also closely related to the length difference application design and the design of the pre-classification of the pseudo log template.

[0097] The application also provides a system based on the robust log template extraction and updating method, comprising:

[0098] A log acquisition processing module is configured to acquire a log to be pattern-recognized, and to obtain a log sequence by preprocessing the log;

[0099] A candidate template acquisition module is configured to extract a pseudo log template from the log sequence, and to search all templates under the pseudo log template as an index to obtain the candidate templates;

[0100] A module matching module is configured to match the log to be pattern-recognized with the candidate templates based on similarity and length difference;

[0101] A template library updating module is configured to update the modules in the template library according to the matching result.

[0102] The above is only the preferred embodiment of the application, and does not limit the implementation and protection scope of the application. For those skilled in the art, it should be realized that any equivalent replacement and obvious changes made by using the content of the application should be included in the protection scope of the application.< / time> < / url> < / ip>

Claims

1. A robust-based log template extraction update method, characterized by, Comprising the following steps: Step 1, obtaining the log to be pattern recognized, pre-processing the log to obtain a log sequence; Step 2, refining the log sequence to obtain a pseudo log template, searching for all templates under the pseudo log template as an index to obtain the candidate templates; Step 3, based on the similarity and length difference, matching the log sequence with the candidate templates; the matching method of step 3 is: calculating the length difference between the log sequence and the candidate template, if the length difference is greater than the threshold, the candidate template is abandoned; If the length difference is less than the threshold, calculate the Jaccard similarity of the two, and select the candidate template with the greatest similarity to the log sequence from all candidate templates with a length difference less than the threshold, and compare the Jaccard similarity of the candidate template with the log sequence with the similarity threshold, if greater than the similarity threshold, it is considered that the log is successfully matched to the appropriate template, that is, the candidate template is the template of the log and is output; If less than the similarity threshold, it is considered that the log is not matched to the appropriate template, and the log sequence of the log is added to the template library as a new template and stored with the pseudo log template of the log as an index; The Jaccard similarity calculation formula is: wherein is a log sequence, set() is a function that turns a log sequence into a set; The length difference formula is: In the formula, len(seq 1 ) denotes the length of the log sequence 1, len(seq 2 ) denotes the length of the log sequence 2, and δ is a fixed value set. Step 4, updating the modules in the template library according to the matching result; the updating method of step 4 is: if the log is successfully matched to the appropriate template in step 3, the matched template needs to be updated according to the log and stored in the template library; otherwise, the template library is updated in the following way: Find the set of common parts of the template with the greatest Jaccard similarity and the log: common_part = set(seq1) set(seq2) wherein is a log sequence, set() is a function that turns a log sequence into a set; Then, traverse the sequence of the template, if the word is in common_part, keep the word, if the word is not in common_part, replace it with a wildcard.

2. The robust-based log template extraction update method of claim 1, wherein, Step 1 specifically includes: Step 1.1, extracting the content part of the obtained log; Step 1.2, identifying the special characters in the log content part by regular or named entity recognition method and replacing them with a preset form of symbol; Step 1.3, segmenting the log content processed in step 1.2 to obtain a log sequence. 3.The robust-based log template extraction update method of claim 1, wherein, The way to obtain the pseudo log template in step 2 is: Construct a keyword dictionary, traverse each word in the log sequence, if the lowercase form of the word is in the keyword dictionary, extract the word, and connect it in order by the connector to obtain the pseudo log template.

4. The robust-based log template extraction update method of claim 3, wherein, The keyword dictionary includes key words describing system behavior, computer domain specific terms, verbs, and words representing system state.

5. A system for robust log template extraction update method according to any one of claims 1-4, characterized in that, It includes: A log acquisition processing module for obtaining a log to be pattern recognized, pre-processing the log to obtain a log sequence; A candidate template acquisition module for refining the log sequence to obtain a pseudo log template, and searching for all templates under the pseudo log template as an index to obtain the candidate templates; A module matching module for matching the log to be pattern recognized with the candidate templates based on similarity and length difference; The template library updating module is configured to update the modules in the template library according to the matched results. The template library updating module is configured to update the modules in the template library according to the matched results.

Images