Keyword-based log analysis method and system
By extracting log keyword combinations and determining positional similarity using the TF-IDF algorithm, the problem of low log parsing accuracy in existing technologies is solved, achieving more efficient and accurate log template mining.
Patent Information
- Application Number
- CN202511372421.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-24
- Publication Date
- 2025-12-30
AI Technical Summary
Existing log parsing methods based on heuristic rules cannot accurately reflect the essential characteristics of logs, resulting in low parsing accuracy and difficulty in adapting to various log types. This is mainly due to their reliance on simple features such as log length and the first word.
The TF-IDF algorithm is used to extract the keyword combination that best represents the logs. A two-dimensional log fingerprint is constructed by checking the log word frequency and part-of-speech tags. Similarity is determined by combining the keyword position. A two-level index table is established to group log items and mine templates.
It improves the accuracy of log template mining, reduces false positives, and enhances the adaptability and accuracy of log parsing, especially for logs with arbitrary formats.
Smart Images
Figure CN121234907A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data analysis, in particular to a log analysis method and system based on keywords. BACKGROUND
[0002] Log analysis is a preliminary work of log analysis, and its task is to convert log original text into structured information for subsequent log analysis. The result of log analysis is mainly log template (sometimes also called log event), and sometimes also contains log variables and other contents. Log original text contains a large amount of information, which is not always useful for log analysis. In order to reduce the difficulty and workload of log analysis, it is necessary to preprocess the log original text before log analysis, and only keep the useful components for subsequent log analysis. Generally speaking, what is useful for log analysis is the log template, so the main task of log analysis is to extract the log template from the log original text.
[0003] The traditional log analysis method based on heuristic rules does not fully reflect the essential characteristics of various logs, so these log analysis methods cannot adapt to various logs. This is because different logs have different characteristics, and the log analysis method based on heuristic rules is usually designed based on a certain assumption, which leads to the log analysis method designed according to a certain assumption can only adapt to specific logs.
[0004] The log analysis method based on heuristic rules needs to establish a number of heuristic rules, and whether the setting of these heuristic rules is reasonable directly relates to the accuracy of log analysis. For example, Drain first classifies logs by log length, then classifies log items according to the first word of the log, forms a number of log item groups, and finally determines similar log items using a similarity determination algorithm to obtain log templates from the similar log item set. Drain does this because it assumes that the length of the log should not be completely the same, and the first word of each log template should be different. However, reality is not completely like this. For some logs, the first word may be the same, so the accuracy of Drain's analysis of such logs is not high. SUMMARY
[0005] The present application is directed to the problem that the log analysis method based on heuristic rules cannot accurately reflect the essence of the log due to its dependence on simple characteristics such as log length and first word, resulting in low analysis accuracy and difficulty in adapting to various logs. A log analysis method and system based on keywords is proposed, which uses the TF-IDF algorithm to extract the keyword combination that best represents the log, which is more representative than the first word and the last word, and the accuracy of log analysis is also higher.
[0006] To achieve the above purpose, the technical scheme adopted is:
[0007] The application provides a keyword-based log analysis method, comprising the following steps:
[0008] Step 1: Taking the log items of the identified logs as a corpus, extracting the keyword combination of each log by a TF-IDF algorithm to construct a log fingerprint, sequentially performing log word frequency checking and log word checking on the to-be-identified log item, matching the log fingerprint to determine the log type, and executing a new log storage process if the matching is unsuccessful;
[0009] Step 2: Constructing an index table according to the keyword combination of the log template, for the new log item, determining the log type based on step 1, and querying in the index table of the corresponding type to be classified into a specific log template under the type;
[0010] Step 3: Based on the keyword position, determining the similarity of single keywords or double keywords for the grouped log item, and extracting the log template after determining the similar log item.
[0011] According to the keyword-based log analysis method, further, the construction of the log fingerprint in step 1 specifically comprises:
[0012] Taking the log items of all identified logs as a corpus, and regarding all the contents of each log as a single document;
[0013] Calculating the TF value and the IDF value of each word in the document by a TF-IDF algorithm, and further calculating the TF-IDF value of each word;
[0014] Selecting the N words with the maximum TF value and the corresponding TF value to form a TF set, selecting the first N words with the maximum TF-IDF value and the corresponding TF-IDF value to form an IDFTF set, and taking the combination of the TF set and the IDFTF set as the fingerprint of the type of log.
[0015] According to the keyword-based log analysis method, further, the specific process of the log word frequency checking in step 1 is:
[0016] For a single to-be-identified log item, calculating the TF value of each word, and statistically forming the first N words with the maximum TF value to form a to-be-checked TF set, and calculating the similarity between the to-be-checked TF set and the TF set in the known log fingerprint, if the similarity value is lower than a first threshold value, then it is preliminarily determined as the log type;
[0017] The similarity calculation process is: initializing the similarity value value as 0; taking each word in the TF set to be checked in turn, checking whether the word exists in the TF set of the known log fingerprint; if the word exists, adding the absolute value of the difference between the TF values of the word in the set to be checked and the known set to the value; if the word does not exist, adding the TF value of the word in the set to be checked to the value; after traversing all N words, dividing the value by N to obtain the average similarity value, and if the average similarity value is less than the first threshold value, outputting the corresponding log category.
[0018] According to the log analysis method based on keywords, further, the specific process of the log word checking in step 1 is:
[0019] A corpus containing the to-be-recognized log item and excluding the suspected log type sample is constructed, the IDF value and the TF-IDF value of each word in the to-be-recognized log item are calculated, the first N words with the largest TF-IDF value are counted to form a to-be-checked IDFTF set, the similarity calculation is performed between the to-be-checked IDFTF set and the IDFTF set in the known fingerprint, and if the similarity value is lower than the second threshold value, the log type is finally determined.
[0020] The similarity calculation process is: initializing the similarity value value as 0; taking each word in the to-be-checked IDFTF set in turn, checking whether the word exists in the IDFTF set of the known log fingerprint; if the word exists, adding the absolute value of the difference between the TF-IDF values of the word in the set to be checked and the known set to the value; if the word does not exist, adding the TF-IDF value of the word in the set to be checked to the value; after traversing all N words, dividing the value by N to obtain the average similarity value, and if the average similarity value is less than the second threshold value, outputting the corresponding log category.
[0021] According to the log analysis method based on keywords, further, the new log storage process includes:
[0022] Extracting a plurality of log items of the new log to form a log sample and adding the log sample to a system log sample set;
[0023] According to the log fingerprint construction process, the log fingerprints of all logs are recalculated and the old fingerprints are replaced.
[0024] According to the log analysis method based on keywords, further, step 2 includes an initial template generation substep and a keyword group generation substep:
[0025] The initial template generation substep: for an unknown log, a Drain algorithm is used for partial log template mining to obtain an initial log template set;
[0026] Keyword group generation sub-step: Using the initial log template set or the known log template set as the corpus, each log template is treated as a document. The TF-IDF algorithm is used to calculate the TF value, IDF value, and TF-IDF value of each word. The word with the largest TF-IDF value in each log template is extracted as its primary keyword. If the TF-IDF value of the primary keyword is lower than the set threshold, the word with the second largest TF-IDF value is extracted as the secondary keyword. Together, they constitute the keyword group of the log template.
[0027] According to the keyword-based log parsing method of the present invention, step 2 further includes an index table construction sub-step: based on the keyword groups of all log templates, a two-level index table is constructed, where the first-level nodes are the main keywords of each log template and the second-level nodes are the secondary keywords of the corresponding log template.
[0028] According to the keyword-based log parsing method of the present invention, step 2 further includes a log item grouping sub-step: for log items whose log type is determined by step 1, a query is performed in the index table corresponding to their log type, and it is checked in turn whether the log item contains the keyword in the first-level node of the index table. If it contains the keyword, it is further checked whether it contains the corresponding second-level node keyword. Based on the check result, it is assigned to the corresponding log template group. If it does not contain the keyword, it is assigned to the unknown group.
[0029] According to the keyword-based log parsing method of the present invention, step 3, which involves determining the similarity of a single keyword or two keywords based on keyword position, further includes:
[0030] The single keyword similarity determination process includes: determining the position of the keyword in the two log entries; scanning forward and backward based on the keyword position, comparing whether the words are the same position by position; and obtaining a comprehensive similarity value after normalization by combining the total number of word differences, the position offset of the two keywords, and the length difference of the suffix of the two log entries.
[0031] The dual-keyword similarity determination process includes: determining the positions of the two keywords in two log entries; dividing the log entries into three segments—prefix, infix, and suffix—with the two keywords as the boundary; scanning and comparing the three segments respectively, comparing whether the words are the same position by position; and obtaining a comprehensive similarity value after normalization by combining the total number of word differences, the positional offset of the two first keywords, and the length differences of the infix and suffix segments of the two log entries.
[0032] Furthermore, the present invention also provides a keyword-based log parsing system for implementing the above-described keyword-based log parsing method. The system includes:
[0033] The log type identification module is used to extract the keyword combinations of each type of log to construct a log fingerprint using the log items of the identified logs as a corpus and the TF-IDF algorithm. The log items to be identified are then checked for log word frequency and log part-of-speech tagging in turn. The log type is determined by matching the log fingerprint. If no match is found, the new log is added to the database.
[0034] The log item grouping module is used to build an index table based on the keyword combination of log templates. For new log items, the log type is determined by the log type identification module, and the corresponding type index table is queried to assign them to a specific log template under that type.
[0035] The similarity determination module is used to determine the similarity of grouped log items based on the keyword position using single or double keywords, and extract log templates after determining similar log items.
[0036] The beneficial effects achieved by adopting the above technical solution are:
[0037] Existing log parsing methods based on heuristic rules mostly employ direct and simple features such as log length, first word, and last word. These methods are prone to misjudgment due to the arbitrariness of log format (e.g., variables as the first word). This solution extracts the most class-discriminating keyword combinations using the TF-IDF algorithm, constructing a two-dimensional log fingerprint containing both TF and TFIDF values. Combined with dual verification of word frequency and part-of-speech tagging, it can accurately identify log types. In the log item grouping stage, a two-level index table is constructed based on keyword combinations to avoid interference from irrelevant logs. In the similarity determination stage, scanning is performed around keyword positions to reduce errors in comparing non-critical information. The entire process focuses on the essential features of the log, thereby improving the accuracy of log template mining. Attached Figure Description
[0038] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. The drawings are merely illustrative of some embodiments of the present invention and are not intended to limit the scope of the present invention to all embodiments.
[0039] Figure 1 This is a flowchart illustrating the keyword-based log parsing method according to an embodiment of the present invention;
[0040] Figure 2 This is a schematic diagram of the log type identification process according to an embodiment of the present invention;
[0041] Figure 3 This is a schematic diagram of the log item grouping process according to an embodiment of the present invention;
[0042] Figure 4 This is a log item grouping process based on an index table, as described in an embodiment of the present invention. Detailed Implementation
[0043] The exemplary solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art.
[0044] First, let's explain the term TF-IDF as follows: TF-IDF stands for Term Frequency-Inverse Document Frequency, a weighted technique widely used in information retrieval and text mining. It assesses the importance of a word or phrase within a document to uncover keywords, which can then be used for text analysis, search engine optimization, and other scenarios. Its core idea is: if a word or phrase appears frequently in one article (high TF) and rarely appears in other articles (high IDF), then this word or phrase is considered to have strong category-discriminating ability and is suitable for representing the content of that article. Specifically, TF-IDF consists of two parts: Term Frequency (TF) and Inverse Document Frequency (IDF). Term Frequency (TF) refers to the frequency with which a given word appears in the document. This value is usually normalized by dividing the term frequency by the total number of words in the document to prevent it from being biased towards longer documents. Inverse document frequency (IDF) reflects the general importance of a term. The fewer documents that contain a term, the higher the IDF value, indicating that the term has a good ability to distinguish categories.
[0045] To improve log parsing accuracy, this invention starts with the essential characteristics of logs and proposes using the TF-IDF algorithm to find representative keywords for a certain type of log. Based on log type identification, it further establishes a mapping between the original log text and log templates using these keywords, ultimately determining log item similarity around the keywords. More specifically, this invention first finds keyword combinations that represent a certain type of log, thus determining the log item category based on these keywords; then it finds keyword combinations that represent a certain type of log template, thus determining which log template a new log item belongs to. To avoid the influence of log noise, words and their immediately following punctuation marks are treated as a whole. Therefore, the keywords in this invention may be single words or combinations of words and punctuation marks. For example, through statistical analysis, we find that log item A contains a large number of words W1, while other log items contain almost no words W1; therefore, if we find that the current log item has a large number of words W1, then this log item is very likely log item A. Similarly, through statistics, we found that the X log template contains the word Y1, while other log templates almost never contain the word Y1. Therefore, if we find that the current log item contains the word Y1, then this log item is very likely to contain the X log template.
[0046] To find features that better reflect the essence of logs, this invention employs the TF-IDF algorithm to find the keyword combinations that best represent logs, and uses different methods to use these keyword combinations to achieve log type identification and log template mining. To achieve the above objectives, the keyword-based log parsing method disclosed in this invention mainly includes three steps: log type identification, log item grouping, and similarity determination. In log type identification, log type identification is completed based on the log fingerprint composed of log keywords, thus providing a foundation for subsequent log template mining to select and use corresponding keyword groups. In log item grouping, the keyword groups that best represent the log templates are determined using existing log templates, and an index table is constructed based on these keyword groups to achieve log item grouping. In similarity determination, this invention proposes a keyword-based intra-group similarity determination method to determine log templates based on log item grouping. The specific implementation process is as follows: Figure 1 As shown.
[0047] Step S101: Using the log items of the identified logs as the corpus, extract the keyword combinations of each log type to construct the log fingerprint. Perform log word frequency check and log part-of-speech check on the log items to be identified in sequence, match the log fingerprint to determine the log type, and if no match is found, execute the new log entry process.
[0048] The purpose of log type identification is to provide a foundation for selecting and using keyword combinations corresponding to the log type in the subsequent log template mining step. Different log types have different log templates, and different log templates also have their own unique and representative keyword combinations. If log types are not distinguished, all keyword combinations can only be matched against the target log item, which undoubtedly increases the workload of matching and may also lead to a decrease in parsing accuracy due to conflicts between different logs. If log types can be identified, the number of keyword combinations to be matched will be reduced, and the probability of conflicts with keyword combinations used by other logs will also be reduced.
[0049] This invention determines log categories based on keyword combinations. Different log types have different representative words; if these words can be found, the log category can be determined. This invention uses the TF-IDF algorithm to obtain the keyword combinations that best represent each type of log, and then determines the log category based on whether the words in the logs to be categorized match the keyword combination characteristics. Since the TF-IDF algorithm relies on a pre-defined corpus, if the scope of logs to be identified by the system changes, it means that the corpus has changed, which necessitates the re-determination of representative words. For example, originally, word W1 was a representative keyword for log type X, but now log type Y has been added, and log type Y also contains a large number of words W1, then word W1 is no longer the most representative keyword.
[0050] like Figure 2 As shown, to identify log types, a log fingerprint is first constructed based on the feature information collected from several logs. For unknown log items to be identified, log identification is completed through log word frequency checking and log part-of-speech tagging, in conjunction with the log fingerprint. First, necessary analysis is performed on the already identified logs to extract representative word combinations, forming the "fingerprint" of that log type. For new log items to be parsed, the existence of these word combinations is checked, and the word frequency is verified to meet the threshold requirement (i.e., log word frequency check). If the word frequency verification is passed, the log type is preliminarily determined. The TF-IDF values of these word combinations are then checked against other log samples to see if they meet the threshold requirement. If they do, the log type is confirmed (i.e., log part-of-speech tagging). If two or more logs meet the requirements, the log type with the highest similarity is taken as the final result.
[0051] (1.1) Constructing Log Fingerprints
[0052] This invention extracts the relevant information of the 20 most representative words from a log as its fingerprint. More words can be selected, but from an efficiency standpoint, 20 words balance recognition accuracy and performance overhead. Users can adjust this as needed; for example, to improve recognition accuracy, 20 can be changed to 40 or a larger number. This invention uses the TFIDF algorithm to extract the most representative keywords, using 10,000 log entries from all identified logs as the TFIDF corpus. The selection of 10,000 log entries also balances recognition accuracy and performance overhead. Users can also adjust this as needed.
[0053] Step 1: Constructing Log "Fingerprints"
[0054] 1. Use 10,000 log entries from all identified logs as the corpus for TFIDF;
[0055] 2. When using the TF-IDF algorithm, all content of each log type is treated as one document;
[0056] 3. Calculate the TF value for each word using the TF-IDF algorithm, denoted as tf. ij , where i represents the i-th log and j represents the j-th word;
[0057] 4. Calculate the IDF value for each word using the TF-IDF algorithm, denoted as idf. ij , where i represents the i-th log and j represents the j-th word;
[0058] 5. Calculate TF-IDF for each word. ij =tf ij *idf ij ;
[0059] 6. Statistics on TensorFlow ij The 10 largest words are recorded along with their corresponding TensorFlow (TF) values. i ={(w i0 ,tf i0 ),...,(w i9 ,tf i9 )};
[0060] 7. Calculate TF-IDF ij The top 10 words are recorded along with their corresponding TF-IDFTF values, and this is represented as an IDF. i ={(x i0 idftf i0 ),...,(x i9 idftf i9 )};
[0061] 8. The fingerprint of log i is {TF} i IDFTF i}
[0062] Repeat step 1 to generate fingerprints for all types of logs.
[0063] (1.2) Log word frequency check
[0064] Log term frequency checking preliminarily determines the log category by examining the frequency of words appearing in the log entries to be identified. This approach is based on the significant performance overhead of log part-of-speech tagging, and since term frequency is one of the computational components of part-of-speech tagging, it can be used to initially filter out some obviously mismatched log categories, thereby reducing the number of log entries requiring part-of-speech tagging.
[0065] Process 2: Log word frequency check
[0066] 1. Use the log items to be tested as the corpus for TFIDF, and treat all content as a single document;
[0067] 2. Calculate the TF value for each word using the TF-IDF algorithm, denoted as tf. j , where j represents the j-th word;
[0068] 3. Statistics on TensorFlow j The 10 largest words are recorded along with their corresponding TensorFlow (TF) values. x ={(w0,tf0),...,(w9,tf9)};
[0069] 4. Match TF x TF in the known log fingerprint i To obtain log categories that are below the threshold F_TF, the calculation method is as follows (assuming TF is calculated). i and TF x ):
[0070] a) Initialize the similarity value value = 0;
[0071] b) From TF x Take w0, and check if w0 is in TF. i If it exists (without loss of generality, assume w0 = w) ia Then value = value + |tf0 - tf ia |, which is value plus the frequency difference between the two words. If it does not exist, then value = value + tf0;
[0072] c) Similarly, sequentially from TF x Take w1, ..., w9, and complete the process with TF. iCheck and update value according to step b).
[0073] d) value = value / 10;
[0074] e) If value < F_TF, output log category i.
[0075] If no log category is output, the recognition fails, meaning this is a completely unknown log. Proceed to the new log warehousing process (see step (1.4)). If only 1 log category is output, continue with the log word nature check to confirm the log type. If multiple log categories are output, the log word nature check is even more necessary to confirm the log type.
[0076] (1.3) Log word nature check
[0077] The log word nature check is to check the word natures of 10 words representing the log fingerprint. The content of the check is the tfidf value of each word. If the tfidf value of the word in the log item to be recognized is similar to that of a recognized log, then the log type can be determined. The log word nature check needs to calculate the tfidf value of each word in the log item to be recognized, so it needs to form a corpus in combination with the log samples. Since the log word frequency check has initially determined the log category, the corpus needs to include the log item to be recognized and the log samples (excluding the suspected log type samples). For example, if the log type to be recognized is Apache, then remove the Apache log items from the log samples and add the log item to be recognized to form the corpus for this TFIDF calculation.
[0078] Process 3: Log word nature check
[0079] 1. Obtain the log samples and the log item to be checked (assume the log type to be determined is i);
[0080] 2. Remove log i from the log samples and add the log item to be checked to form the corpus used in the current calculation. When calculating, consider all the content of each log item as a document;<*
[0081] 3. According to the TFIDF algorithm, calculate the IDF value of each word in the log item to be checked, denoted as idf j ;
[0082] 4. Calculate tfidf for each word in the log item to be checked j = tf j * idf j ; (tf j has been calculated in Process 2);
[0083] 5. Statistic idftf jThe 10 largest words, recording the words and their corresponding IDFTF values, denoted as IDFTF x ={(w x0 , idftf x0 ),...,(w x9 , idftf x9 )};
[0084] 6. Match the log fingerprint IDFTF i With IDFTF x For the similarity degree, determine whether the log category is lower than the threshold F_IDFTF. The calculation method is as follows:
[0085] a) Initialize the similarity value value = 0;
[0086] b) Take w0 from IDFTF x , and check whether w0 exists in IDFTF i . If it exists (assuming w0 = w ia ), then value = value + |idftf0 - idftf ia |, that is, value plus the difference in word frequencies of the two words. If it does not exist, then value = value + idftf0;
[0087] c) Similarly, take w1,..., w9 from IDFTF x in turn, complete the check with IDFTF i , and update value;
[0088] d) value = value / 10;
[0089] e) If value < F_IDFTF, then output log i and the corresponding value (marked as value i ); otherwise, it is determined that the current log item to be recognized is not log i.
[0090] The process [Log Word - property Check] is to check the similarity of two logs. So if the log word - frequency check outputs multiple suspected similar logs, then multiple log word - property checks are required. If the process [Log Word - property Check] gives multiple similar conclusions, then the final result selects the log type with the smallest value of value. If no log satisfies the word - property check, then turn to the new log storage process (see step (1.4)).
[0091] (1.4) New log storage
[0092] If the current system cannot recognize the logs, a new log fingerprint needs to be added to the system. A prerequisite for adding new logs to the database is that the user can confirm the log type and initiate the new log addition process. If the user is not interested in the current logs and skips this step, the system will not be able to recognize that type of log. The new logs added to the database include: a log sample and the log fingerprint.
[0093] Step 4: New logs are written to the database
[0094] 1. Extract several log entries from the new log to form a log sample corresponding to the log, and add it to the system's log sample set;
[0095] 2. Following procedure 1, calculate the log fingerprint of all logs and replace the old log fingerprint.
[0096] Step S102: Construct an index table based on the keyword combination of the log template. For a new log item, determine the log type based on step S101, query the index table of the corresponding type, and assign it to a specific log template under that type.
[0097] like Figure 3 As shown, the Drain algorithm is first used to mine some log templates, obtaining a subset of log templates. After obtaining the log templates, keywords are extracted based on the TFIDF algorithm, and finally, all log items are grouped based on the keywords. For known logs, this invention directly groups log items based on keywords. Therefore, if the log to be processed is a known log, the system already has its keyword group information, and initial template generation and keyword group generation are optional steps. Thus, this invention is more suitable for online log template mining; it cannot fully utilize the potential of the method designed in this invention for unknown logs. If the log item grouping results are unsatisfactory, manifested in a large number of log items not being able to rely on keyword grouping, then keyword group updates are triggered to optimize the keyword groups. Note: Keyword group updates are only completed after the current round of log template mining, relying on a more complete set of log templates.
[0098] (2.1) Initial template generation
[0099] For unknown logs, this invention employs the Drain algorithm to mine partial log templates, obtaining a subset of log templates, and then extracting keywords from these templates. Since only a subset of log templates is obtained, this incompleteness leads to inaccurate keyword analysis in subsequent steps. To improve accuracy, the number of log items parsed beforehand can be increased. The workflow of the Drain algorithm is existing technology and will not be elaborated upon here.
[0100] This step is optional. Its purpose is to obtain the log template. If the system has already processed similar logs, it will retain the corresponding log template, so this step can be skipped.
[0101] (2.2) Keyword Phrase Generation
[0102] This invention finds the keywords that best represent a log template. This step still uses the TFIDF algorithm, but the corpus is a set of log templates corresponding to a certain type of log. Each log template is treated as a single document, and the TFIDF algorithm is used to find the words that best represent that log template. To improve efficiency, punctuation marks immediately following a word are also considered part of the word. Thus, even the same word will be considered different depending on whether it is followed by a space or a comma. Including punctuation marks in the calculation fully utilizes their positional information, partially compensating for the TFIDF algorithm's lack of concern for position.
[0103] Step 5: Keyword Phrase Generation
[0104] 1. Using all log templates as a corpus for TFIDF, each log template is treated as a document. The TFIDF algorithm is used to find the keyword group that best represents the log template.
[0105] 2. Extract words from the log template. To improve the representativeness of log words, treat each word and the punctuation mark immediately following it as a single word. If there is a space after the word, no special processing is performed.
[0106] 3. Calculate the TF value for each word using the TF-IDF algorithm, denoted as tf. ij , where i represents the i-th log template and j represents the j-th word;
[0107] 4. Calculate the IDF value for each word using the TF-IDF algorithm, denoted as idf. ij ;
[0108] 5. Calculate TF-IDF for each word. ij =tf ij *idf ij ;
[0109] 6. Generate keywords for each log template and record them as {t i ,w ij}, where w ij It is a log template t i The word with the largest TF-IDF value; if w ij If the threshold F_MA is not exceeded, it is recorded as {t}. i ,wij ,w ib}, where w ib The second largest word in the log template ti (tfidf).
[0110] Generally, each log template uses only one most representative word, i.e., the one with the highest TF-IDF value. However, in some special cases, some log templates are quite similar, so relying solely on one most representative word is insufficient to completely distinguish between different log templates. Therefore, it is permissible to use two words as keyword phrases simultaneously.
[0111] (2.3) Log Item Grouping
[0112] Log item grouping is accomplished using an index table, such as... Figure 4 As shown, the system generates an index table based on keyword groups. Because some log templates contain two keywords in their keyword groups, the index table has a two-level structure. The first-level structure of the index table is the keyword with the highest TF-IDF value for each log template. If a log template corresponds to two keyword groups, the corresponding index table also has a second-level structure, as shown by keyword 11 and keyword 12 in the figure. Multiple second-level nodes are possible because multiple log templates may correspond to the same keyword with the highest TF-IDF value, so they share a single first-level node. If a log item does not contain all the keywords in the index table, it is categorized as * and considered an unknown log item, pending further processing.
[0113] Step 6: Keyword Classification
[0114] 1. The system generates an index table based on keyword phrases;
[0115] 2. When a new log entry is received, check whether the log entry contains the keywords listed in the first-level node of the index table. If it does, skip to step 3; otherwise, classify the log entry as "*", treat it as an unknown log entry, and skip to step 5.
[0116] 3. Check if a second-level node exists for this first-level node. If it does, proceed to step 4; otherwise, group the node into the log entry group corresponding to the current index table node and proceed to step 5.
[0117] 4. Continue searching whether the log item contains the keyword corresponding to the second-level node. If it does, the node is grouped into the log item group corresponding to the current index table node, and the process jumps to step 5. If not, the log item is marked with "*" as an unknown log item, and the process jumps to step 5.
[0118] Repeat steps 2-4 until all log entries have been processed.
[0119] (2.4) Keyword Phrase Update
[0120] If a large number of nodes are marked with "*", indicating they are unknown log entries, it means the currently used keyword group is inappropriate. Therefore, this invention requires keyword group calculation. This invention stipulates that if 30% of log entries are marked with "*", a keyword group update operation needs to be initiated. The specific keyword group update operation will be delayed until the current round of log processing is completed, because only then will new log templates be mined. In other words, the keyword group update will be used for the next round of log template mining.
[0121] The keyword group update process is similar to step (2.3), except that when updating the keyword group, the log template is often more comprehensive (because the log template is not comprehensive, the keyword group does not have enough representativeness).
[0122] Step S103: For the grouped log items, determine the similarity of single or double keywords based on the keyword position, and extract the log template after determining similar log items.
[0123] Since there are two types of classification results—one is normal classification based on the index table, and the other is grouping into a specific group—this invention uses keyword position-based similarity determination for the former. For the latter, this invention still uses existing similarity determination methods.
[0124] (3.1) Intra-group similarity determination
[0125] Based on the log item grouping principle, log items in the same group must contain the same keywords. Keywords are not necessarily the first or last word. Considering that existing similarity assessments often compare each item one by one from beginning to end, which fails to leverage the advantages of keywords, this invention compares keywords both forward and backward. Considering the possibility of multiple keywords, this invention scans each keyword in turn; that is, keyword A is scanned one word forward, and the word preceding keyword B is also scanned. This process is repeated until all log items have been scanned.
[0126] Log entries with only one keyword are similar to each other according to process 7. Log entries with two keywords are similar to each other according to process 8.
[0127] Step 7: Single Keyword Similarity Determination
[0128] 1. Determine the keyword position of log item A (i.e., which word in the log item the keyword is, let's call it p), and the keyword position of log item B (let's call it q);
[0129] 2. Let variable x = 1, sim = 0;
[0130] 3. If both p - x and q - x are greater than or equal to 1, check whether the (p - x)-th word of log entry A is the same as the (q - x)-th word of log entry B. If they are the same, set sim = sim + 0; otherwise, set sim = sim + 1;
[0131] 4. x = x + 1, and repeat step 3 until either p - x or q - x equals 0;
[0132] 5. Set sim = sim + |p - q|;
[0133] 6. x = 1;
[0134] 7. Let len_a be the length of log entry A and len_b be the length of log entry B
[0135] 8. If p + x is less than len_a and q + x is less than len_b, check whether the (p + x)-th word of log entry A is the same as the (q + x)-th word of log entry B. If they are the same, set sim = sim + 0; otherwise, set sim = sim + 1;
[0136] 9. x = x + 1, and repeat step 8 until either p + x equals len_a or q + x equals len_b;
[0137] 10. Set sim = sim + |len_a - p - len_b + q|
[0138] 11. sim = sim / min(len_a, len_b)
[0139] 12. If sim < TH_S, consider log entry A and log entry B to be similar.
[0140] Process 8: Determination of double-keyword similarity
[0141] 1. Determine the positions of two keywords in log entry A (i.e., which word of the log entry the keyword is, say p and s, and s > p), and the positions of keywords in log entry B (say q and t, and t > q);
[0142] 2. Let the variable x = 1 and sim = 0;
[0143] 3. If both p - x and q - x are greater than or equal to 1, check whether the (p - x)-th word of log entry A is the same as the (q - x)-th word of log entry B. If they are the same, set sim = sim + 0; otherwise, set sim = sim + 1;
[0144] 4. x = x + 1, and repeat step 3 until either p - x or q - x equals 0;
[0145] 5. Set sim = sim + |p - q|;
[0146] 6. x = 1;
[0147] 7. If p + x is less than s, and q + x is less than t, then check whether the p + x words of log entry A are the same as the q + x words of log entry B. If they are the same, record sim = sim + 0; otherwise, record sim = sim + 1.
[0148] 8. x = x + 1, repeat step 8 until p + x equals s or q + x equals t.
[0149] 9. Record sim = sim + |s - p - t + q|
[0150] 10. x = x + 1
[0151] 11. Let len_a be the length of log entry A and len_b be the length of log entry B
[0152] 12. If s + x is less than len_a, and t + x is less than len_b, then check whether the s + x words of log entry A are the same as the t + x words of log entry B. If they are the same, record sim = sim + 0; otherwise, record sim = sim + 1.
[0153] 13. x = x + 1, repeat step 8 until s + x equals len_a or t + x equals len_b.
[0154] 14. Record sim = sim + |len_a - s - len_b + t|
[0155] 15. sim = sim / min(len_a, len_b)
[0156] 16. If sim < TH_S, then consider log entry A and log entry B to be similar.
[0157] Log entries determined to be similar by similarity are grouped into the same cluster. All log entries within the same cluster contain the same log template. Count all log entries within the same cluster, consider the different parts as constants and the same parts as variables, retain the constants, and set the variables to *, thereby obtaining a log template from one cluster.
[0158] (3.2) *Group similarity determination
[0159] For the log entries grouped into the * group, the system still uses the existing log entry similarity determination method. Briefly speaking, it starts comparing from the first word of the log entry until the last word. If the number of words at the same position that are the same reaches a certain degree, the two log entries are considered similar. The present invention still uses the existing method, so it will not be elaborated here. [[ID=ID=39]]
[0160] Correspondingly, the present invention also discloses a keyword-based log parsing system, which includes:
[0161] The log type identification module is used to extract the keyword combinations of each type of log to construct a log fingerprint using the log items of the identified logs as a corpus and the TF-IDF algorithm. The log items to be identified are then checked for log word frequency and log part-of-speech tagging in turn. The log type is determined by matching the log fingerprint. If no match is found, the new log is added to the database.
[0162] The log item grouping module is used to build an index table based on the keyword combination of log templates. For new log items, the log type is determined by the log type identification module, and the corresponding type index table is queried to assign them to a specific log template under that type.
[0163] The similarity determination module is used to determine the similarity of grouped log items based on the keyword position using single or double keywords, and extract log templates after determining similar log items.
[0164] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features within the technical scope disclosed in the present invention; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A keyword-based log parsing method, characterized by, The method comprises the following steps: Step 1: Taking the log items of the identified logs as a corpus, extracting the keyword combination of each log by the TF-IDF algorithm to construct a log fingerprint, performing log word frequency checking and log word checking on the log items to be identified in turn, matching the log fingerprint to determine the log type, and executing a new log storage process if the matching is unsuccessful; Step 2: Constructing an index table according to the keyword combination of the log template, determining the log type of the new log item based on step 1, and querying the corresponding type index table to be classified into a specific log template under the type; Step 3: Based on the keyword position, the similarity of single keyword or double keywords is determined for the grouped log items, and the log template is extracted after the similar log items are determined. 2.The keyword-based log parsing method according to claim 1, characterized in that, The construction of the log fingerprint in step 1 specifically comprises: Taking all the log items of the identified logs as a corpus, and regarding all the contents of each log as a single document; Calculating the TF value and IDF value of each word in the document by the TF-IDF algorithm, and then calculating the TF-IDF value of each word; Selecting the N largest TF values and corresponding TF values to form a TF set, and selecting the N largest TF-IDF values and corresponding TF-IDF values to form an IDFTF set, and taking the combination of the TF set and the IDFTF set as the fingerprint of the type of log. 3.The keyword-based log parsing method according to claim 2, characterized in that, The specific process of the log word frequency checking in step 1 is: For a single log item to be identified, the TF value of each word is calculated, the N largest TF values are counted to form a checking TF set, the checking TF set is compared with the TF set in the known log fingerprint, and if the similarity value is lower than the first threshold value, it is preliminarily determined as the log type; The similarity calculation process is: initializing the similarity value value as 0; taking each word in the checking TF set in turn, and checking whether it exists in the TF set of the known log fingerprint; If it exists, the value is added to the absolute value of the difference between the TF values of the word in the checking set and the known set; if it does not exist, the value is added to the TF value of the word in the checking set; After traversing all N words, the average similarity value is obtained by dividing the value by N, and if the average similarity value is less than the first threshold value, the corresponding log category is output. 4.The keyword-based log parsing method according to claim 2, wherein, The specific process of the log word checking in step 1 is: A corpus containing the log item to be identified and excluding the suspected log type sample is constructed, the IDF value and TF-IDF value of each word in the log item to be identified are calculated, the N largest TF-IDF values are counted to form a checking IDFTF set, the checking IDFTF set is compared with the IDFTF set in the known fingerprint, and if the similarity value is lower than the second threshold value, it is finally determined as the log type; The similarity calculation process is: initializing the similarity value value as 0; taking each word in the checking IDFTF set in turn, and checking whether it exists in the IDFTF set of the known log fingerprint; If the word exists, value is added to the absolute value of the difference between the TF-IDF value of the word in the set to be checked and the known set; if not, value is added to the TF-IDF value of the word in the set to be checked; After traversing all N words, the value is divided by N to obtain the average similarity value, and if the average similarity value is less than the second threshold value, the corresponding log category is output. 5.The keyword-based log parsing method according to claim 1, wherein, The new log storage process includes: Extracting several log items of the new log to form a log sample and adding the log sample to a system log sample set; According to a log fingerprint construction process, the log fingerprints of all logs are recalculated and the old fingerprints are replaced. 6.The keyword-based log parsing method according to claim 1, wherein, Step 2 includes an initial template generation sub-step and a keyword group generation sub-step: The initial template generation sub-step: for unknown logs, a Drain algorithm is used for partial log template mining to obtain an initial log template set; The keyword group generation sub-step: taking the initial log template set or the known log template set as a corpus, each log template is regarded as a document, and the TF value, IDF value and TF-IDF value of each word are calculated using the TF-IDF algorithm, the word with the maximum TF-IDF value in each log template is extracted as its main keyword, and if the TF-IDF value of the main keyword is lower than a set threshold value, the second largest TF-IDF value is additionally extracted as a secondary keyword, which together constitute the keyword group of the log template. 7.The keyword-based log parsing method according to claim 6, characterized in that, Step 2 further includes an index table construction sub-step: based on the keyword groups of all log templates, a two-level index table is constructed, the first level node is the main keyword of each log template, and the second level node is the secondary keyword of the corresponding log template. 8.The keyword-based log parsing method according to claim 7, characterized in that, Step 2 further includes a log item grouping sub-step: for the log items to be grouped determined by step 1, the corresponding index table of the log type to which the log item belongs is queried, whether the log item contains the keyword in the first level node of the index table is checked in turn, if it contains, whether it contains the corresponding second level node keyword is further checked, and according to the checking result, it is classified into the corresponding log template group, and if it does not contain, it is classified into an unknown group. 9.The keyword-based log parsing method according to claim 1, wherein, The single-keyword similarity determination process includes: determining the positions of the keywords in the two log items; based on the keyword positions, scanning forward and backward respectively, and comparing the words position by position; and obtaining a comprehensive similarity value after normalization by comprehensively considering the total number of word differences, the position offset of the two keywords, and the length difference of the suffix parts of the two log items. The double-keyword similarity determination process includes: determining the positions of the two keywords in the two log items; dividing the log items into three segments, prefix, infix and suffix, based on the two keywords; scanning and comparing the three segments respectively, and comparing the words position by position; and obtaining a comprehensive similarity value after normalization by comprehensively considering the total number of word differences, the position offset of the two first keywords, and the length difference of the infix segment and the suffix segment of the two log items. The system for implementing the keyword-based log analysis method according to any one of claims 1-9 includes:
10. A keyword-based log analysis system, characterized by, The log type identification module is configured to take the log items of the identified logs as a corpus, extract keyword combinations of each log by using a TF-IDF algorithm to construct a log fingerprint, perform log word frequency checking and log word type checking on the log items to be identified in sequence, match the log fingerprint to determine the log type, and perform a new log storage process if the matching is unsuccessful; The log item grouping module is configured to construct an index table according to the keyword combinations of the log templates, query the index table corresponding to the type of the new log item based on the log type determined by the log type identification module, and group the new log item into a specific log template of the type. The similarity determination module is configured to determine the similarity of the grouped log items based on single-keyword or double-keyword similarity determination of the keyword positions, and extract the log template after determining the similar log items.