Log template mining method and system based on multi-index

Abstract

The present invention relates to the technical field of log information processing, and more particularly to a multi-index-based log template mining method and system. The method preprocesses collected raw log data to extract a set of log items from the log, wherein the log item set includes key events and key variables. Sampled log items are obtained by sampling the extracted log item set, and a number of initial log templates are formed based on the sampled log items. The discriminability of the initial log templates is evaluated using index data, and target indexes are selected and formed based on the discriminability and performance overhead. The log items are classified based on the index combination, divided into multiple clusters, and intermediate log templates are generated. The discriminability of the intermediate log templates is evaluated using the index combination, and the index combination is adaptively adjusted to achieve log template mining. The method can adapt to mining tasks for various log templates while reducing manual intervention, and has strong universality.

Description

Technical Field

[0001] The present invention relates to the technical field of log information processing, and in particular to a log template mining method and system based on multiple indexes. Background Art

[0002] Log template mining is the process of extracting valuable information, such as events or variables, from raw log files, generating structured information for subsequent business processing. Essentially, log template mining involves classifying logs into different categories (i.e., log templates) based on different indexes. Log template mining can be broadly divided into two categories. One is based on frequent item methods, such as SLCT and LogCluster. These methods scan log entries multiple times to identify frequently occurring words, then group the log entries into clusters based on these words, generating log templates. This is primarily due to the fact that not all events occur frequently, and some occasional events are noteworthy, but these algorithms will miss them. The other is based on heuristic rule methods, such as Drain and Logpunk. These methods often implement log template mining based on heuristic rules, such as log length. The Drain algorithm accelerates preprocessing by combining multiple indexes using a classification tree. It first uses length as an index to group all log entries into different groups, and then further indexes the groups using the first word as an index. If log entries meet these two characteristics, classification can be completed very efficiently. However, logs cannot always meet this condition. For example, variable-length variables in logs may cause length-based classification to fail. Summary of the Invention

[0003] To this end, the present invention provides a log template mining method and system based on multiple indexes, which solve the problems of the existing log template mining, such as lack of universality and need to improve mining efficiency.

[0004] According to the design scheme provided by the present invention, on the one hand, a log template mining method based on multiple indexes is provided, comprising:

[0005] Preprocessing the collected raw log data to extract log item sets from the logs, wherein the log item sets include: key events and key variables;

[0006] The extracted log item set is sampled to obtain sampled log items, and a number of initial log templates are formed based on the sampled log items;

[0007] Obtain index data under specified indicators in the initial log template, use the index data to evaluate the discrimination of the initial log template, select target indexes based on the discrimination and performance overhead, and form an index combination. The index data under the specified indicators include: log item length index data, log item first word index data, log item punctuation index data, and log item keyword index data. The log template discrimination is used to indicate the degree of distinction between log templates.

[0008] Log items are classified according to the index combination, divided into multiple clusters and an intermediate log template is generated. The discrimination of the intermediate log template is evaluated using the index combination. If the discrimination meets the expectations, the intermediate log template is output as the target log template. If the discrimination does not meet the expectations, the algorithm returns and readjusts the index combination to reclassify the log items using the adjusted index combination until the discrimination of the generated intermediate log template meets the expectations.

[0009] As the log template mining method based on multiple indexes of the present invention, the collected original log data is further preprocessed, including:

[0010] Regular expressions are used to extract key events and variables from raw log data and remove redundant information from the raw log data.

[0011] As the log template mining method based on multiple indexes of the present invention, further, several initial log templates are formed based on the sampled log items, including:

[0012] For sampled log items, a log mining algorithm is used to mine log templates to obtain several initial log templates. The sampled log items are obtained by sampling a log item set based on a specified number of log items at an interval. The log mining algorithm is used to convert unstructured logs into structured logs.

[0013] As the log template mining method based on multiple indexes of the present invention, the discrimination degree of the initial log template is further evaluated using the index data, including:

[0014] Classify the initial log template log items using the index data under each specified indicator;

[0015] Based on the log item classification results and the number of initial log templates, the discrimination degree of the initial log template under the corresponding specified indicator index data is obtained, and the discrimination degree of the initial log template under the corresponding specified indicator index is evaluated by the discrimination degree.

[0016] As the log template mining method based on multiple indexes of the present invention, further, target indexes are selected based on the discrimination and performance overhead to form an index combination, including:

[0017] Based on the index style, the log item length index and log item punctuation index are classified into the format style index type, and the log item first word index and log item keyword index are classified into the content style index type;

[0018] The recommended index combination is determined based on the discrimination size and index performance overhead and using index combination rules. The index combination rules include: selecting candidate indexes from different style index types, determining the comprehensive capabilities of each candidate index based on the discrimination size and index performance overhead, determining the number of candidate indexes based on the comprehensive index performance overhead and log item classification efficiency, and selecting candidate indexes based on the priority of index performance overhead.

[0019] As the log template mining method based on multiple indexes of the present invention, further, the log items are classified according to the index combination, including:

[0020] Instantiate a multi-level classification tree for classifying log items according to the index combination, and determine the indexes used at different levels of the classification tree;

[0021] Classify the log items in turn according to the indexes of each level in the multi-level classification tree, calculate the similarity of the log items in each category in the classification results of each log item, and group the log items with similarity greater than the threshold into a cluster;

[0022] A log item in the cluster is randomly selected as the cluster head node, and the similarity between cluster head nodes is used to judge the similarity between classes. Log items are then merged based on the similarity between classes to obtain log item clusters.

[0023] As the log template mining method based on multiple indexes of the present invention, further, log items are classified in sequence according to the indexes of each level in the multi-level classification tree, including:

[0024] Starting from the node to be classified in the multi-level classification tree, the current level index is classified. During the classification process, the current level index type is first determined.

[0025] If the index type is a log item length index, the length of the log item to be classified is calculated. If the child node of the node to be classified contains the length, the log item to be classified is transferred to the child node for further classification. If the child node of the node to be classified does not contain the length, a new child node is created for the node to be classified and the new child node is labeled according to the length.

[0026] If the index type is the first word index of the log item, the first word of the log item to be classified is read. If the child node of the node to be classified contains the first word, the log item to be classified is transferred to the child node for further classification. If the child node of the node to be classified does not contain the first word, a new child node is created for the node to be classified and the new child node is labeled according to the first word.

[0027] If the index type is a log-type punctuation index, the log item to be classified is read and a punctuation sequence of the log item to be classified is generated. If the child node of the node to be classified contains the punctuation sequence, the log item to be classified is transferred to the child node for further classification. If the child node of the node to be classified does not contain the sequence, a new child node is created for the node to be classified and marked according to the punctuation sequence.

[0028] If the index type is a log item keyword index, read the log item to be classified and check whether the keyword in the keyword set corresponding to the keyword index exists in the log item to be classified. If the keyword in the keyword set exists in the log item to be classified, transfer the log item to be classified to the child node for further classification. If the keyword in the keyword set does not exist in the log item to be classified, create a new child node for the node to be classified and mark the new child node with a special identifier.

[0029] Determine whether the current level index is the bottom index of the multi-level classification tree. If it is the bottom index, obtain the log item classification cluster based on the child node of the current index. If it is not the bottom index, use the child node as the middle node of the classification tree to perform the next level index classification.

[0030] The log template mining method based on multiple indexes of the present invention further classifies log items in sequence according to the indexes at each level in the multi-level classification tree, and further includes:

[0031] For a multi-level classification tree in which the index type is a log item keyword index, all log item classification clusters in a subtree corresponding to a root node marked with a special identifier are counted within a specified time period, so as to update a keyword set corresponding to the keyword index based on the log item classification cluster statistical results and a preset trigger condition, wherein the log item classification process is divided into initial classification and subsequent classification based on the number of log item categories, so that in the initial classification, the preset trigger condition is set to that the number of log item categories contained in the log item classification cluster exceeds a specified proportion of the total number of log items, and in the subsequent classification, the preset trigger condition is set to that the number of log item categories accounts for a specified proportion of the total number of log items;

[0032] Read the log items in the log item classification cluster in the subtree corresponding to the root node marked with a special identifier in sequence. For each log item in the classification cluster, determine whether the log item in the classification cluster contains the keywords in the keyword set. If not, process the next log item. If it does, determine whether the node marked with the special identifier has a sibling node marked with the keywords in the keyword set. If so, transfer the log item in the classification cluster to the sibling node. If not, create a new sibling node for the node marked with the special identifier, and mark the new sibling node as the keyword in the keyword set, until all log items in the classification cluster are processed.

[0033] On the other hand, the present invention also provides a log template mining system based on multiple indexes, comprising: a preprocessing module, a sampling module, an evaluation module and a classification module, wherein:

[0034] A preprocessing module is used to preprocess the collected raw log data and extract log item sets from the logs, wherein the log item sets include: key events and key variables;

[0035] A sampling module, configured to obtain sampled log items by sampling the extracted log item set, and form a number of initial log templates based on the sampled log items;

[0036] An evaluation module is configured to obtain index data under specified indicators in the initial log template, evaluate the discrimination of the initial log template using the index data, select target indexes based on the discrimination and performance overhead, and form an index combination. The index data under the specified indicators include: log item length index data, log item first word index data, log item punctuation index data, and log item keyword index data. The log template discrimination is used to indicate the degree of distinction between log templates.

[0037] The classification module is used to classify log items according to the index combination, divide the log items into multiple clusters and generate intermediate log templates. The index combination is used to evaluate the discrimination of the intermediate log template. If the discrimination meets the expectations, the intermediate log template is output as the target log template. If the discrimination does not meet the expectations, it returns and readjusts the index combination to reclassify the log items using the adjusted index combination until the discrimination of the generated intermediate log template meets the expectations.

[0038] Beneficial effects of the present invention:

[0039] The present invention obtains the distinguishing ability of different indexes for the current log through sampling mining, thereby adaptively recommending suitable index combinations and using index combinations to realize log template mining. On the basis of reducing manual intervention, it can adapt to the mining tasks of various log templates and adaptively select index combinations according to actual conditions, so that the log template has a higher degree of distinction and the accuracy of log template mining is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the log template mining process based on multiple indexes in the embodiment;

[0041] Figure 2 This is an illustration of multi-index mining based on a classification tree in an embodiment;

[0042] Figure 3 This is a schematic diagram of the multi-level classification tree generation process in the embodiment;

[0043] Figure 4Schematic diagram of the multi-index mining process in the embodiment. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below with reference to the accompanying drawings and technical solutions.

[0045] The diversity of logs makes log template mining difficult. It is difficult to have a log template mining algorithm that can process most logs while achieving good accuracy and recall. To this end, the present invention provides a log template mining method based on multiple indexes, such as Figure 1 As shown, including:

[0046] S101 : Pre-processing the collected original log data to extract a log item set from the log, wherein the log item set includes: key events and key variables.

[0047] Log preprocessing removes non-critical information from the original log text, preserving valuable data for subsequent processing. Specifically, in this embodiment, regular expressions can be used to extract key events and variables from the original log data and remove redundant information from the original log data. For example, the IP address in the original log text is often not valuable for subsequent log anomaly detection. Therefore, after log preprocessing, the IP address in the original log text is filtered out or represented only by a fixed identifier without paying attention to the content of the specific IP address.

[0048] S102 : Obtain sampled log items by sampling the extracted log item set, and form several initial log templates based on the sampled log items.

[0049] Specifically, for sampled log items, a log mining algorithm is used to mine log templates to obtain several initial log templates. The sampled log items are obtained by sampling a log item set based on a specified number of log items at an interval. The log mining algorithm is used to convert unstructured logs into structured logs.

[0050] Sampling mining is to extract several log templates from the log, and evaluate them according to the log templates and the mining process. For unknown logs, it is impossible to determine which log template mining algorithm is most suitable. In order to evaluate which index is suitable for the current log, in the embodiment of this case, a certain number of log items are extracted from the processed log to form a set of log items to be mined, represented by log_m, and a log mining algorithm such as the Drain algorithm is used for trial mining to obtain a set of log templates after trial mining, represented by t_m. In order to obtain log features as comprehensively as possible, a log item can be extracted at intervals of a certain number of log items, and these extracted log item sets are combined to form log_m. Among them, Drain is an online log parsing algorithm based on a fixed-depth tree, which can have good comprehensive performance in most log data sets. Therefore, in the embodiment of this case, the Drain algorithm can be used to pre-process the log items. The pseudo code of the template trial mining algorithm can be shown as follows:

[0051]

[0052]

[0053] S103. Obtain index data under specified indicators in the initial log template, use the index data to evaluate the discrimination of the initial log template, select the target index and form an index combination based on the discrimination size and performance overhead, the index data under the specified indicators include: log item length index data, log item first word index data, log item punctuation index data and log item keyword index data, the log template discrimination is used to indicate the degree of distinction between log templates.

[0054] Specifically, the index data is used to evaluate the discrimination of the initial log template, which can be designed to include:

[0055] Classify the initial log template log items using the index data under each specified indicator;

[0056] Based on the log item classification results and the number of initial log templates, the discrimination degree of the initial log template under the corresponding specified indicator index data is obtained, and the discrimination degree of the initial log template under the corresponding specified indicator index is evaluated by the discrimination degree.

[0057] Discrimination is used to evaluate the degree to which an index distinguishes log templates. Assuming a certain index has n enumeration values ​​and m corresponding log templates, the discrimination corresponding to this index is m / n. For example, if log item length is used as an index, and the possible values ​​for log item length are {4, 5, 6, 7}, and there are seven log templates, then the discrimination corresponding to the length index is 4 / 7. This situation may be caused by having log items with the same length corresponding to multiple log templates. Ideally, the discrimination is 1, meaning that all log items can be completely classified into different log templates using this index.

[0058] In this embodiment, the discrimination evaluation mainly includes length discrimination evaluation, first word discrimination evaluation, punctuation discrimination evaluation, and keyword discrimination evaluation. New indexes can be introduced, thereby also introducing new discrimination evaluation methods.

[0059] Length discrimination evaluation assesses the algorithm's ability to classify based on length indexes. Length-indexed classification involves categorizing log entries based on their length, grouping log entries of the same length into the same category. Length refers to the number of words in a log entry. Length is not concerned with the log's content, but rather its physical form, allowing it to be calculated with minimal computational effort. However, because log entries are generally of similar length, classification based on length indexes often lacks discriminability, making it difficult to rely solely on length indexes for classification. Length indexes are typically used in combination with other indices. The pseudocode for the length discrimination evaluation algorithm is shown below:

[0060]

[0061]

[0062] First word discrimination evaluation is an assessment of the algorithm's ability to classify based on the first word index. Classification based on the first word index means classifying log items according to the first word of the log item, and grouping log items with the same first word into the same category. Generally speaking, the first words of different log templates may be different, but it cannot be assumed that all log items meet this condition. Some software output logs may place the event completer as a variable in the first position of the log item, which will result in messages belonging to the same log template being divided into different categories. On the other hand, it is also possible to classify log items that do not belong to the same log template into the same category. The pseudo code of the algorithm for first word discrimination evaluation is as follows:

[0063]

[0064] Punctuation discrimination evaluation assesses the algorithm's ability to classify log entries based on punctuation indexes. Punctuation index classification is based on the punctuation marks in log entries. Generally speaking, while log entries may contain variables that result in different log entries, these variables rarely contain punctuation marks. Therefore, log entries belonging to the same log template typically have consistent punctuation marks. However, the punctuation index does not analyze the log entry content, so it often lacks good discrimination for shorter log entries with fewer punctuation marks. In an extreme case, a large number of log entries may contain "xxxxxxx.", making it difficult to distinguish them using the ".". Longer log entries with more punctuation marks exhibit better discrimination.

[0065] Punctuation indexes are generated using a sequence + punctuation method. Given a log entry, in order to obtain a punctuation index, in this embodiment, all non-punctuation characters are removed, leaving only punctuation marks. For example, given a log entry "A, B: C.", its punctuation index is: ",:.". Retaining all punctuation marks may result in inconsistent punctuation index lengths, which increases the difficulty of subsequent index matching. The pseudocode for the punctuation discrimination evaluation algorithm is shown below:

[0066]

[0067] Keyword discrimination evaluation is an assessment of the algorithm's ability to classify based on keyword indexing. Keyword indexing classification is based on the keywords in the log entries. The problem with first-word indexing is that the first word does not always have good discrimination. Therefore, in this embodiment, keywords with better discrimination are selected for index mining. Keywords are the most representative words in a log entry that can effectively distinguish it from other log entries. Keywords may be the first word or a word at the end of the log entry.

[0068] To find keywords, you can use the TF-IDF algorithm, using the already obtained log template as the dataset (i.e., t_m), to find the keywords that best represent the log template. The log template at this point may differ from the final log template, but it still reflects the final log template to a certain extent. Therefore, although the keywords are not analyzed from the final log template, the obtained keywords are representative. TF-IDF is a statistical method used to assess the importance of a word to a document set or a document in a corpus. Its main idea is: if a word or phrase appears frequently in one article and rarely appears in other articles, it is considered to have good category differentiation ability and is suitable for classification.

[0069] The pseudo code for generating a keyword index is as follows:

[0070]

[0071] After obtaining the keyword set, we can evaluate the discriminability of the log item keywords. The pseudo code for the evaluation is as follows:

[0072]

[0073] Determine the appropriate index combination and use it for mining. The index combination can have only one index, two indexes, or three indexes. Too many indexes can improve accuracy, but will introduce too much performance overhead. Therefore, the classification tree used in multi-index mining can have only one level classification, two levels classification, Figure 2 The classification tree used in the multi-index mining shown has three levels of classification. After several classifications, the log items are divided into several clusters, and finally the similarity of each log item in the cluster is determined.

[0074] The index combination is determined by classification decision-making. Classification decision-making determines the index combination based on the comprehensive capabilities of each index. The comprehensive capabilities of each index are determined by index performance metrics and discrimination. Index performance metrics are derived from the performance cost of index classification. For example, log item length is relatively easy to obtain, while obtaining keywords requires more performance cost.

[0075] Among them, the target index is selected based on the discrimination size and performance cost and an index combination is formed, which can be designed to include:

[0076] Based on the index style, the log item length index and log item punctuation index are classified into the format style index type, and the log item first word index and log item keyword index are classified into the content style index type;

[0077] The recommended index combination is determined based on the discrimination size and index performance overhead and using index combination rules. The index combination rules include: selecting candidate indexes from different style index types, determining the comprehensive capabilities of each candidate index based on the discrimination size and index performance overhead, determining the number of candidate indexes based on the comprehensive index performance overhead and log item classification efficiency, and selecting candidate indexes based on the priority of index performance overhead.

[0078] The performance cost index can be shown in Table 1. The performance cost index can be adjusted according to the development software and the library functions it relies on, so as to make the index performance cost index more accurate. The parameters set in this embodiment are obtained by evaluating the software written in Python.

[0079] Table 1 Index performance cost

[0080] index Performance cost Length Index 1 Punctuation Index 0.9 First word index 0.9 Keyword Index 0.8

[0081] The rules for determining index combinations can be described as follows:

[0082] Rule 1: An index within an index combination cannot be just a format style index or just a content style index.

[0083] Length and punctuation indexes are format-style indexes. These indexes focus solely on the format of the log entries, regardless of the content. First-word and keyword indexes are content-style indexes, focusing on the content of the log entries. To maximize the effectiveness of the index combination, both index styles should be included to improve the index's classification capabilities.

[0084] Rule 2: Only indexes with comprehensive capabilities greater than the threshold can be selected into the index combination.

[0085] Different logs may be better suited for mining with different indexes, so no single index combination is ideal for all logs. After evaluating discrimination, we can determine which index is most suitable for the log. Therefore, we can comprehensively evaluate both accuracy and performance to select the most appropriate index combination for the log.

[0086] Rule 3: It is recommended to use 2-3 types of indexes to form an index combination.

[0087] Excessive indexing complicates the classification process and increases performance overhead. Furthermore, because indexing doesn't guarantee complete classification accuracy, incorrect classifications can lead to subsequent errors in classification and similarity determination. The proliferation of these errors reduces mining efficiency.

[0088] Rule 4: Prioritize indexes with low performance overhead as the indexes used for the first group of classifications.

[0089] Prioritizing performance can significantly reduce the number of classification comparisons, thereby reducing performance overhead. However, if the discriminability of this index is low, it may introduce errors and increase subsequent operations. For example, if the log item length is used as the index, and the current log item contains variable-length variables, log items belonging to the same log template will be divided into different sets. However, based on existing experimental data, prioritizing performance remains the more appropriate strategy.

[0090] Classification decision means giving a recommended index combination based on the rules. Users can also adjust the content of the index combination according to their needs, that is, the index set used and the order of use.

[0091] S104. Classify the log items according to the index combination, divide the log items into multiple clusters and generate an intermediate log template. Use the index combination to evaluate the discrimination of the intermediate log template. If the discrimination meets the expectations, output the intermediate log template as the target log template. If the discrimination does not meet the expectations, return and readjust the index combination to reclassify the log items using the adjusted index combination until the discrimination of the generated intermediate log template meets the expectations.

[0092] Specifically, log items are classified according to index combinations, which can be designed to include:

[0093] Instantiate a multi-level classification tree for classifying log items according to the index combination, and determine the indexes used at different levels of the classification tree;

[0094] Classify the log items in turn according to the indexes of each level in the multi-level classification tree, calculate the similarity of the log items in each category in the classification results of each log item, and group the log items with similarity greater than the threshold into a cluster;

[0095] A log item in the cluster is randomly selected as the cluster head node, and the similarity between cluster head nodes is used to judge the similarity between classes. Log items are then merged based on the similarity between classes to obtain log item clusters.

[0096] The classification tree is instantiated based on the index combination and gradually improved in the process of hierarchical classification, such as Figure 3 As shown in the figure. Initially, the classification tree only has the root node. Based on the index combination generated by the classification decision, the classification tree is instantiated, the classification index used at different levels of the classification tree is determined, and the classification tree is in a classification-ready state. After the classification begins, due to the small number of nodes in the classification tree at the beginning, failures to match child nodes will frequently occur. For example, when classifying according to the length index, the current log item to be classified has a length of 6, but the node's subsequent child nodes do not have a length of 6. In this case, it is necessary to construct descendant nodes with a length of 6. At this time, new nodes will be generated, gradually forming the classification tree. After a period of classification, the classification tree will basically remain unchanged.

[0097] There will be some differences in classification according to different indexes. In the embodiment of this case, we focus on 4 types of indexes such as length index, and the classification standards of different indexes will be different. Because the multi-level classification method adopted by the present invention, the classification result may be a cluster, that is, a leaf node, which is also the final result of the classification, or the input of the next level of classification, that is, the intermediate result of the classification. During the classification process, there may be a situation where the corresponding descendant node does not exist. At this time, a new node will be generated for the classification tree. The pseudo code for classification by different indexes can be shown as follows:

[0098]

[0099] Among the above classifications, the keyword index is relatively special. In the embodiment of this case, the earliest KT is obtained based on the sampling mining results. Obviously, the number of keywords contained in KT at this time is relatively small and cannot reflect the true situation of all log items. Therefore, if the classification tree adopts a keyword index, then since the early KT has fewer keywords, a large number of log items may not be correctly classified, and KT needs to be expanded as necessary. For this reason, in the embodiment of this case, all clusters in the subtree with NoKT as the root node are counted at a certain time, and this is used as a data set to update KT. The pseudo code of the algorithm for updating KT can be shown as follows:

[0100]

[0101] The conditions for triggering keyword set updates are divided into the following two situations:

[0102] Case 1: Under the initial classification, the number of log entries contained in all clusters of the subtree with NoKT as the root node exceeds 1 / 4 of the processed log entries.

[0103] Case 2: In subsequent classification, the number of log entries contained in all clusters of the subtree with NoKT as the root node exceeds 1% of the total number of log entries, and two consecutive keyword set update operations should be separated by a certain time (the present invention uses the number of classification log entries as the timer, which is agreed to be 1% of the total number of classification log entries)

[0104] The reasons for the above settings are as follows. When the classification tree just starts to classify, more log items will be classified into the NoKT subtree. At this time, the number of log items contained in the NoKT subtree will exceed 1 / 4 of the processed log items at a faster rate, and the keyword set update process will be triggered. When the classification has been carried out for a period of time, since the keyword set already contains more keywords and can classify most log items, the keyword set update operation will no longer be triggered for a long time. On the other hand, due to the fact that the keyword set has not been triggered for a long time, some keywords may not be updated to KT, which will result in a small number of log items being unable to be classified according to the keywords. Therefore, when the keyword set is rarely updated, the trigger condition will be updated to: the number of log items contained in all clusters of the subtree with NoKT as the root node exceeds 1% of the total number of log items.

[0105] Of course, the above thresholds can be adjusted appropriately according to actual needs. For example, in subsequent classification, the number of log items contained in all clusters of the subtree with NoKT as the root node exceeds 1% of the total number of log items. The 1% here can be adjusted appropriately as needed. Because there may be a certain number of log items in the current log that do not have keywords that meet the conditions, it is not very meaningful to repeatedly and frequently perform keyword collection. This is also why the present invention requires that during subsequent classification, the keyword collection update interval be at least a certain time (the present invention uses the number of classified log items as the timer, which is agreed to be 1% of the total number of classified log items).

[0106] The criterion for determining initial and subsequent classification in the triggering keyword set update condition is: 100,000 log item classifications have been completed. Before 100,000 log items have been classified, it's considered initial classification; after that, it's considered subsequent classification. Based on experience, most log templates contain no more than 1,000 items, so by the time 100,000 log items have been classified, theoretically, most keywords should have been discovered. Of course, some logs may be unique, and the log items corresponding to the log templates may be unevenly distributed, resulting in some log templates appearing only in subsequent log sets. This necessitates gradual adjustments to the keyword set during subsequent classification.

[0107] After updating the keyword set, all nodes in the NoKT subtree will be reclassified to reduce the number of nodes in the NoKT subtree. The process is roughly as follows: read the log items in the cluster of the NoKT subtree in turn, and check whether these log items contain keywords (due to the addition of new keywords, some log items that originally did not contain keywords may contain the newly added keywords). If they do, check whether the sibling node of the NoKT node contains a node marked with the keyword. If so, transfer the node to the sibling node and complete subsequent classification under the node. The pseudo code of the classification tree adjustment algorithm based on specific keyword adjustment is as follows:

[0108]

[0109]

[0110] To sum up, the pseudo code of the algorithm classified by level can be described as follows:

[0111]

[0112] The final similarity determination, based on classification, determines the template mining step. Ideally, after classification, log items belonging to the same group should belong to the same log template. In reality, log items belonging to the same group may not belong to the same log template, and log items distributed in different groups may still belong to the same log template. Similarity determination is the final guarantee for the accuracy of log template mining.

[0113] Since the index combination must include the first word index or the keyword index, the similarity determination used in this embodiment will take into account the situation of these two indexes. When the similarity determination process of the first word index is used, the algorithm pseudo code is as follows:

[0114]

[0115] When the similarity determination process of keyword index is adopted, the algorithm pseudo code is as follows:

[0116]

[0117]

[0118] One possible scenario is to use both the first-word index and the keyword index. In this case, the similarity determination process used when the keyword index is present is used. This is because this method is less susceptible to the effects of variable-length variables. Log entries that meet the similarity threshold (which can default to 0.9) are grouped into a cluster, and a random log entry within the cluster is selected as the cluster head node. For subsequent cluster similarity determinations, the cluster head node is used as the representative node in the similarity calculation.

[0119] Similarity determination is divided into two steps: intra-cluster determination and inter-cluster determination. In intra-cluster determination, the similarity is determined for log items within the group. After intra-cluster determination, a small number of clusters will be formed within the group (each cluster should belong to the same log template). Because classification accuracy cannot be guaranteed, clusters from different groups may belong to the same log template, so inter-cluster determination is also performed. Since the number of clusters obtained after intra-cluster determination is significantly smaller than the original number of log items, the number of inter-cluster determinations is relatively small, which does not affect overall performance overhead.

[0120] In summary, the process of multi-index determination can be used Figure 4 As shown in the figure, classification decisions determine the current index combination based on the discrimination evaluation results and decision rules or user settings. Classification by level performs classification operations based on indexes, dividing log items into different groups. Similarity determination is performed on groups, performing both intra-group and inter-group similarity determinations, ultimately dividing log items into different clusters.

[0121] The purpose of mining accuracy assessment is to evaluate whether the current index combination meets expectations. The initial index combination is derived from sampling mining, and its classification results are evaluated to see if they differ from the actual results. Mining accuracy assessment further confirms the classification performance of the index combination after a certain period of mining. If deviations are found in the current index combination, the index combination is regenerated and the classification tree is adjusted.

[0122] In this embodiment, the mining accuracy assessment can be set to occur after the classification mining of 100,000 log items has been completed. It can be assumed that after the classification mining of 100,000 log items, a relatively complete picture of the log items has been obtained. At this point, the index combinations obtained from the sampling mining can be evaluated. Of course, some log template mining tasks may be relatively simple, with a small number of log items to be mined, in which case the mining accuracy assessment step may not be performed. It can be set to require the mining accuracy assessment when the number of log items to be mined exceeds one million.

[0123] The mining accuracy evaluation is similar to the discrimination evaluation. Both involve statistical calculations of the discrimination of four indexes, but the difference lies in the evaluation data. The data source for the discrimination evaluation is the log template set obtained through sampling mining, while the data source for the mining accuracy evaluation is the log template set obtained through mining the first 100,000 log items in the log set.

[0124] The decision to rebuild the classification tree will be based on the mining accuracy evaluation results. If the mining accuracy evaluation results do not deviate significantly from the discrimination evaluation results, the classification tree will not be rebuilt. The classification tree will only be rebuilt when there is a significant deviation between the two results. This is because rebuilding the classification tree will incur a significant performance overhead. The condition for rebuilding the classification tree is: based on the evaluation results of the two methods and the decision combination generated according to the classification decision, if there is a difference in one index, then the classification tree needs to be rebuilt. The pseudo code of the mining accuracy evaluation algorithm is as follows:

[0125]

[0126] The classification tree reconstruction process is consistent with the hierarchical classification process. Rebuilding the classification tree is equivalent to reclassifying the log items. To reduce the overhead of tree reconstruction, the mining accuracy assessment can be triggered earlier, for example, after mining 50,000 or 10,000 log items. This allows the tree to be rebuilt without significant loss. However, there is a trade-off: mining too few log items may not fully reflect the characteristics of the mined logs, thus still risking the selection of the wrong index combination. This threshold can be set by the user.

[0127] Furthermore, based on the above method, an embodiment of the present invention also provides a log template mining system based on multiple indexes, comprising: a preprocessing module, a sampling module, an evaluation module and a classification module, wherein:

[0128] A preprocessing module is used to preprocess the collected raw log data and extract log item sets from the logs, wherein the log item sets include: key events and key variables;

[0129] A sampling module, configured to obtain sampled log items by sampling the extracted log item set, and form a number of initial log templates based on the sampled log items;

[0130] An evaluation module is configured to obtain index data under specified indicators in the initial log template, evaluate the discrimination of the initial log template using the index data, select target indexes based on the discrimination and performance overhead, and form an index combination. The index data under the specified indicators include: log item length index data, log item first word index data, log item punctuation index data, and log item keyword index data. The log template discrimination is used to indicate the degree of distinction between log templates.

[0131] The classification module is used to classify log items according to the index combination, divide the log items into multiple clusters and generate intermediate log templates. The index combination is used to evaluate the discrimination of the intermediate log template. If the discrimination meets the expectations, the intermediate log template is output as the target log template. If the discrimination does not meet the expectations, it returns and readjusts the index combination to reclassify the log items using the adjusted index combination until the discrimination of the generated intermediate log template meets the expectations.

[0132] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0133] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0134] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0135] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or software functional modules. The present invention is not limited to any specific combination of hardware and software.

[0136] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A log template mining method based on multiple indexes, characterized in that: Include: Preprocessing the collected raw log data to extract log item sets from the logs, wherein the log item sets include: key events and key variables; The extracted log item set is sampled to obtain sampled log items, and a number of initial log templates are formed based on the sampled log items; Obtain index data under specified indicators in the initial log template, use the index data to evaluate the discrimination of the initial log template, select target indexes based on the discrimination and performance overhead, and form an index combination. The index data under the specified indicators include: log item length index data, log item first word index data, log item punctuation index data, and log item keyword index data. The log template discrimination is used to indicate the degree of distinction between log templates. Log items are classified according to the index combination, divided into multiple clusters and an intermediate log template is generated. The discrimination of the intermediate log template is evaluated using the index combination. If the discrimination meets the expectations, the intermediate log template is output as the target log template. If the discrimination does not meet the expectations, the algorithm returns and readjusts the index combination to reclassify the log items using the adjusted index combination until the discrimination of the generated intermediate log template meets the expectations.

2. The log template mining method based on multiple indexes according to claim 1 is characterized in that: Preprocess the collected raw log data, including: Regular expressions are used to extract key events and variables from raw log data and remove redundant information from the raw log data.

3. The log template mining method based on multiple indexes according to claim 1 is characterized in that: Based on the sampled log items, several initial log templates are formed, including: For sampled log items, a log mining algorithm is used to mine log templates to obtain several initial log templates. The sampled log items are obtained by sampling a log item set based on a specified number of log items at an interval. The log mining algorithm is used to convert unstructured logs into structured logs.

4. The log template mining method based on multiple indexes according to claim 1 is characterized in that: Use index data to evaluate the discrimination of the initial log template, including: Classify the initial log template log items using the index data under each specified indicator; Based on the log item classification results and the number of initial log templates, the discrimination degree of the initial log template under the corresponding specified indicator index data is obtained, and the discrimination degree of the initial log template under the corresponding specified indicator index is evaluated by the discrimination degree.

5. The log template mining method based on multiple indexes according to claim 1 is characterized in that: Select the target index based on the discrimination and performance cost and form an index combination, including: Based on the index style, the log item length index and log item punctuation index are classified into the format style index type, and the log item first word index and log item keyword index are classified into the content style index type; The recommended index combination is determined based on the discrimination size and index performance overhead and using index combination rules. The index combination rules include: selecting candidate indexes from different style index types, determining the comprehensive capabilities of each candidate index based on the discrimination size and index performance overhead, determining the number of candidate indexes based on the comprehensive index performance overhead and log item classification efficiency, and selecting candidate indexes based on the priority of index performance overhead.

6. The log template mining method based on multiple indexes according to claim 1 is characterized in that: Classify log items based on index combinations, including: Instantiate a multi-level classification tree for classifying log items according to the index combination, and determine the indexes used at different levels of the classification tree; Classify the log items in turn according to the indexes of each level in the multi-level classification tree, calculate the similarity of the log items in each category in the classification results of each log item, and group the log items with similarity greater than the threshold into a cluster; A log item in the cluster is randomly selected as the cluster head node, and the similarity between cluster head nodes is used to judge the similarity between classes. Log items are then merged based on the similarity between classes to obtain log item clusters.

7. The log template mining method based on multiple indexes according to claim 6 is characterized in that: Log items are classified in sequence according to the indexes at each level in the multi-level classification tree, including: Starting from the node to be classified in the multi-level classification tree, the current level index is classified. During the classification process, the current level index type is first determined. If the index type is a log item length index, the length of the log item to be classified is calculated. If the child node of the node to be classified contains the length, the log item to be classified is transferred to the child node for further classification. If the child node of the node to be classified does not contain the length, a new child node is created for the node to be classified and the new child node is labeled according to the length. If the index type is the first word index of the log item, the first word of the log item to be classified is read. If the child node of the node to be classified contains the first word, the log item to be classified is transferred to the child node for further classification. If the child node of the node to be classified does not contain the first word, a new child node is created for the node to be classified and the new child node is labeled according to the first word. If the index type is a log-type punctuation index, the log item to be classified is read and a punctuation sequence of the log item to be classified is generated. If the child node of the node to be classified contains the punctuation sequence, the log item to be classified is transferred to the child node for further classification. If the child node of the node to be classified does not contain the sequence, a new child node is created for the node to be classified and marked according to the punctuation sequence. If the index type is a log item keyword index, read the log item to be classified and check whether the keyword in the keyword set corresponding to the keyword index exists in the log item to be classified. If the keyword in the keyword set exists in the log item to be classified, transfer the log item to be classified to the child node for further classification. If the keyword in the keyword set does not exist in the log item to be classified, create a new child node for the node to be classified and mark the new child node with a special identifier. Determine whether the current level index is the bottom index of the multi-level classification tree. If it is the bottom index, obtain the log item classification cluster based on the child node of the current index. If it is not the bottom index, use the child node as the middle node of the classification tree to perform the next level index classification.

8. The log template mining method based on multiple indexes according to claim 7 is characterized in that: Classify log items in sequence according to the indexes at each level in the multi-level classification tree, and also include: For a multi-level classification tree in which the index type is a log item keyword index, all log item classification clusters in a subtree corresponding to a root node marked with a special identifier are counted within a specified time period, so as to update a keyword set corresponding to the keyword index based on the log item classification cluster statistical results and a preset trigger condition, wherein the log item classification process is divided into initial classification and subsequent classification based on the number of log item categories, so that in the initial classification, the preset trigger condition is set to that the number of log item categories contained in the log item classification cluster exceeds a specified proportion of the total number of log items, and in the subsequent classification, the preset trigger condition is set to that the number of log item categories accounts for a specified proportion of the total number of log items; Read the log items in the log item classification cluster in the subtree corresponding to the root node marked with a special identifier in sequence. For each log item in the classification cluster, determine whether the log item in the classification cluster contains the keywords in the keyword set. If not, process the next log item. If it does, determine whether the node marked with the special identifier has a sibling node marked with the keywords in the keyword set. If so, transfer the log item in the classification cluster to the sibling node. If not, create a new sibling node for the node marked with the special identifier, and mark the new sibling node as the keyword in the keyword set, until all log items in the classification cluster are processed.

9. A log template mining system based on multiple indexes, characterized in that: Contains: preprocessing module, sampling module, evaluation module and classification module, among which, A preprocessing module is used to preprocess the collected raw log data and extract log item sets from the logs, wherein the log item sets include: key events and key variables; A sampling module, configured to obtain sampled log items by sampling the extracted log item set, and form a number of initial log templates based on the sampled log items; An evaluation module is configured to obtain index data under specified indicators in the initial log template, evaluate the discrimination of the initial log template using the index data, select target indexes based on the discrimination and performance overhead, and form an index combination. The index data under the specified indicators include: log item length index data, log item first word index data, log item punctuation index data, and log item keyword index data. The log template discrimination is used to indicate the degree of distinction between log templates. The classification module is used to classify log items according to the index combination, divide the log items into multiple clusters and generate intermediate log templates. The index combination is used to evaluate the discrimination of the intermediate log template. If the discrimination meets the expectations, the intermediate log template is output as the target log template. If the discrimination does not meet the expectations, it returns and readjusts the index combination to reclassify the log items using the adjusted index combination until the discrimination of the generated intermediate log template meets the expectations.

10. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 8.

Images