Real-time log analysis method and system fusing position information and fixed depth tree

Abstract

The invention relates to a real-time log analysis method and system fusing position information and a fixed depth tree, and the method comprises the steps: firstly, constructing a feature constant buffer pool and constant tokens as a fixed depth prefix tree of a two-layer node, and solving the problem that the number of branches of the prefix tree explodes due to the fact that a first token is a variable in a Drain algorithm; secondly, the efficiency of the Drain algorithm is improved by performing regular matching on the token and then setting the regular matching to a template updating stage; furthermore, by updating a lightweight context counter in real time, the algorithm can perform constant identification based on the stability degree and context information of each token position, and the problem of excessive generalization of the Drain algorithm in a template updating stage is effectively improved; and finally, the template Hash value is cached in the leaf node, so that the algorithm can solve the template matching problem of the variable-length log.

Description

Technical Field

[0001] This invention relates to the field of log parsing technology, and in particular to a real-time log parsing method and system that integrates location information and a fixed-depth tree. Background Technology

[0002] Log parsing is a crucial step in transforming unstructured logs into structured data, and it is widely used in areas such as system observability, fault detection, and security auditing. Transforming unstructured logs into structured data refers to extracting key information from a string of plain text, including IP addresses, timestamps, log templates, and other relevant information.

[0003] Currently, in public and enterprise private cloud scenarios, software generates TB and PB levels of log data daily, requiring efficient log parsing that balances accuracy and speed. Log types are diverse, including system component logs and application component logs, with completely different templates and no discernible pattern. Variable identification is particularly challenging in log parsing. While common variables like IP addresses and timestamps can be replaced using regular expressions, a large number of variables requiring proprietary business knowledge or domain-specific information cannot be identified using conventional methods.

[0004] Currently, Drain is one of the most widely used and effective algorithms in industry. It can parse logs into a fixed-depth prefix tree. When a new log arrives, it searches the prefix tree. If a match is found, the log template stored in the leaf nodes is used; otherwise, a new branch is created. However, it relies on the log pattern and requires parsing specific software logging systems. For logs that violate its pattern, this type of algorithm performs poorly.

[0005] In summary, the existing methods have the following drawbacks: (1) The Drain algorithm natively only supports segmentation based on delimiters. For natural languages ​​such as English, it can achieve good log token segmentation and subsequent parsing. However, for natural languages ​​such as Chinese, which are segmented based on semantics and punctuation, the Drain algorithm cannot handle them well.

[0006] (2) In the log parsed by default, the earlier the position is, the more likely it is to be a constant, and the later the position is, the more likely it is to be a variable. However, this is acceptable in academic research, but for logs in cloud applications where the beginning position is a variable, it will lead to the problem of branch explosion of the prefix tree.

[0007] (3) Since Drain is based on a prefix tree, apart from the first-level root node and the second-level token length node and leaf node, the remaining intermediate nodes are actually a partitioned token. If the log token partition is not accurate, it is easy to cause logs of the same pattern to be distributed on different tree branch paths. At present, there is no completely accurate algorithm for token partitioning. Therefore, the problem of logs of the same pattern but different lengths being distributed on different paths is unavoidable in Drain.

[0008] (4) In most industrial scenarios, Drain and other non-deep learning algorithms rely heavily on regular expressions to preprocess variables. However, if users use too many regular expressions, it will have a significant impact on the performance of the overall log parsing system.

[0009] In summary, there is currently a lack of a real-time log parsing method and system to solve or partially solve the aforementioned problems. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a real-time log parsing method and system that integrates location information and a fixed-depth tree, so as to solve or partially solve the problems of inaccurate semantic segmentation, the risk of branch explosion in the prefix tree, and logs of different lengths distributed on different paths.

[0011] The objective of this invention can be achieved through the following technical solutions: One aspect of the present invention provides a real-time log parsing method that integrates location information and a fixed-depth tree, comprising the following steps: Retrieve the raw log to be parsed; The original log is segmented using English delimiters, Chinese character recognition, and word segmentation to obtain a token list; Traverse the token list, search the pre-built feature constant cache pool, and attempt to match the first feature constant token of the original log. Determine whether the first feature constant token is successfully matched. If not, the token list is assigned to the None cluster. Construct each level of the tree in sequence and return the log after regularization as a template. If yes, proceed to different levels for node matching according to constant token, token list length and token list prefix. If a node match is successful, the similarity between the templates that eventually enter the leaf node and the corresponding multiple log groups is calculated, and the log template with the highest similarity is returned. The positions of the log template with the highest similarity are traversed, and the positions that meet the regular expression or the current position saturation condition are updated to wildcards. The branches are merged based on the current position saturation to obtain the final log template. If a match fails, build a tree branch downwards along the current level and return the logs processed by the regular expression as a template.

[0012] As a preferred technical solution, the process of updating positions that meet the regular expression or whose current position saturation meets the conditions to wildcards includes the following steps: Iterate through the token list and the tokens at each position of the log template, and update the counter information in the corresponding leaf node, including the cumulative number of token samples at each position, the token that appears most frequently at each position and its quantity, and retain the top-K token pairs that appear most frequently at each position; During the traversal, it is determined whether the current position of the log template is a wildcard. If it is, the wildcard at the current position is retained; otherwise, the regular expression matching process is initiated. During the regular expression matching process, it is determined whether the token at the current position matches a regular expression of numbers or hexadecimal. If yes, the token at the current position is updated to a wildcard. If no, based on the counter information, the saturation of the current position is calculated according to the stability of the current position and the stability of the positions before and after it. Whether the token at the current position is updated to a wildcard is determined according to whether the saturation reaches a threshold.

[0013] As a preferred technical solution, the process of merging branches based on the current position saturation includes the following steps: Based on the saturation of the current position, if the preset conditions are met, the token of the current position is added to the feature constant cache pool. If the current position is under the None branch, the branch merging is triggered, and the current branch is moved to the regular branch.

[0014] As a preferred technical solution, the position saturation is calculated using the following formula: in, Indicates the first Position saturation of tokens at each location, , They represent the first The stability of each location, and the stability of its neighboring locations. For position The token that appears most frequently Total number of times the token has appeared For counting functions, This means that for each , in the The number of times each position appears , This represents the total number of positions.

[0015] As a preferred technical solution, the process of constructing tree branches downwards along the current level includes the following steps: If the current search is in a node that matches the length of the token list, create the node corresponding to the length of the token list, and then iterate through the remaining tokens in the token list, creating nodes sequentially towards the leaf nodes. If a leaf node has been reached, construct a log set at the leaf node, add the ID of the current original log to the ID list in the log set, and set the token list of the current original log as the template for the new log set. Build a token counter in the leaf node to count the distribution of unique values ​​and contexts at each position.

[0016] As a preferred technical solution, the process of matching the target log template with the highest similarity to the original log includes the following steps: Determine whether a node matching the length of the token list is found. If so, match the target log template with the highest similarity to the original log from multiple log templates in the leaf node. If not, search under the node with the length adjacent to the length of the token list. If a leaf node is found, perform a minHash operation on multiple log templates of the leaf node and the current log, and calculate their Jaccard similarity. If the similarity exceeds the threshold, the matching target log template is obtained.

[0017] As a preferred technical solution, the process of obtaining the first feature constant token of the original log through matching includes the following steps: Iterate through the token list and determine if the corresponding token exists in the feature constant cache pool. If it does, the match is successful, and the current token is marked as the first feature constant token. If the entire token list has been traversed and none of the tokens are in the feature constant cache pool, the match fails, and the first feature constant token is marked as None. The process of selecting the corresponding branch based on the first feature constant token includes: Determine whether the first feature constant token is None. If it is, proceed to the branch with the value of None in the second level of the tree for searching. If not, proceed to the regular branch, that is, search for the node with the value of the first feature constant token.

[0018] As a preferred technical solution, the process of token segmentation through English delimiters, Chinese character recognition, and word segmentation includes the following steps: The original log was split according to the English delimiter; Iterate through each segmented token, check if there are Chinese characters, and if so, extract consecutive Chinese characters and perform Chinese word segmentation. The tokens after segmentation and word segmentation are merged into a list to obtain the token list.

[0019] As a preferred technical solution, after obtaining the raw log to be parsed, the process also includes extracting the log string from the raw log in JSON format, cleaning up extra spaces, and filtering information using regular expressions defined by the algorithm user.

[0020] Another aspect of the present invention provides a real-time log parsing system that integrates location information and a fixed-depth tree, for implementing the aforementioned real-time log parsing method, the system comprising: The log acquisition and preprocessing module is used to acquire and parse the raw logs to be parsed.

[0021] The token segmentation module is used to segment the original log using English delimiters, Chinese character recognition, and word segmentation to obtain a token list.

[0022] The first feature constant token extraction module is used to obtain the first feature constant token by traversing the token list and the pre-built constant cache pool.

[0023] The target log template matching module is used to first select the corresponding branch based on the first feature constant token in a pre-built fixed-depth tree, and then match the corresponding node based on the length of the token list. If the match fails, a new path branch is created. If the match succeeds, the token list is traversed and the search is performed along the direction of the leaf node. If the search fails, a new path branch is created. If the search succeeds, the target log template with the highest similarity to the original log is matched from multiple log templates in the leaf node.

[0024] The template update and branch merging module is used to traverse each position in the target log template, update the positions that meet the regular expression or the current position saturation condition to wildcards, and obtain the final log template to realize log parsing.

[0025] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) Achieve accurate segmentation of tokens in mixed Chinese and English: This invention obtains the original log to be parsed, segments the tokens by English delimiters, recognizes Chinese characters and word segmentation, and obtains a token list. It improves the segmentation mechanism, effectively supports the parsing of logs with mixed Chinese and English, and expands the segmentation capability.

[0026] (2) Overcoming the branch explosion problem of the prefix tree: On the one hand, the present invention constructs a feature constant cache pool. Based on the token list and the pre-constructed feature constant cache pool, the first feature constant token is obtained by traversal. On the other hand, the fixed-depth tree of the present invention adds a constant layer above the length layer. In the pre-constructed fixed-depth tree, the corresponding branch is first selected based on the first feature constant token, and then the corresponding node is matched based on the length of the token list. With the help of the subsequent search algorithm process, the branch explosion problem caused by the first token being a variable is overcome.

[0027] (3) Overcoming the problem of matching failure caused by logs of different lengths being distributed on different paths: When matching logs belonging to the same pattern but with different lengths, the problem arises because the token segmentation step cannot be completely accurate. It may segment log word units with separators such as spaces into different tokens, thus causing logs belonging to the same template to be distributed on different paths in the parse tree. This invention determines whether a node matching the length of the token list is found. If so, it matches the target log template with the highest similarity to the original log from multiple log templates in the leaf node. If not, it searches under the node with the length adjacent to the length of the token list. If a leaf node is found, it performs a minHash operation on multiple log templates of the leaf node and the current log, and calculates their Jaccard similarity. If it exceeds the threshold, the matching target log template is obtained, thus overcoming this problem.

[0028] (4) Overcoming the problem of overgeneralization in the Drain method: This invention traverses each position in the target log template, updates positions that meet the regular expression or the current position saturation condition to wildcards, merges branches based on the current position saturation, obtains the final log template, realizes log parsing, calculates the stability and context prediction ability of each position by updating a lightweight counter in real time, and thus determines whether the position is a variable. Compared with the Drain algorithm, which completely relies on the token of a single position to independently determine whether it is a variable, this algorithm considers the co-occurrence information of the preceding and following neighborhoods, and uses the top-1 frequency to replace the probability distribution entropy. While retaining the calculation accuracy, it greatly reduces the time complexity of the algorithm, and effectively improves the problems of overgeneralization and noise interference in the Drain algorithm. Attached Figure Description

[0029] Figure 1 This is a flowchart of the real-time log parsing method that integrates location information and a fixed-depth tree in the embodiment; Figure 2 This is a schematic diagram of the parse tree structure in the embodiment; Figure 3 This is a flowchart of token segmentation in the embodiment; Figure 4 This is a flowchart illustrating the process of obtaining the first distinctive constant token in this embodiment. Figure 5 This is a flowchart of a fixed-depth tree search in the embodiment; Figure 6 This is a flowchart illustrating the branch selection based on characteristic constants in the embodiment. Figure 7 This is a flowchart illustrating the creation of a new path branch in the embodiment; Figure 8This is a flowchart illustrating the similarity calculation in the embodiment; Figure 9 This is a flowchart illustrating the process of updating the template and triggering inter-cluster merging in the embodiment. Figure 10 This is a flowchart illustrating the matching of variable-length logs with the same pattern in this embodiment. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] Example 1 To address the problems existing in the aforementioned prior art, this embodiment provides a real-time log parsing method that integrates location information and a fixed-depth tree, aiming to achieve log parsing through log template matching. For details, see... Figure 1 For a schematic diagram of the overall matching process, please refer to [link / reference]. Figure 1 When a new log arrives, it will sequentially go through four main steps: preprocessing, token partitioning, selecting the first feature constant in the log, and fixed-depth tree matching. (1) Preprocessing is mainly used to clean the logs in a custom way. Typical operations include: extracting log strings from JSON format logs, cleaning up extra spaces, and filtering information using regular expressions defined by the algorithm user.

[0032] (2) The token segmentation module mainly splits the log string into individual words or symbol units to prepare for the subsequent entry into the fixed-depth prefix tree.

[0033] (3) The main function of the module for obtaining the first feature constant is to statistically identify tokens that may be constants based on a large number of logs. Its purpose is to cooperate with the None cluster branch of the parse tree to avoid the situation where the Drain algorithm encounters a variable first token, resulting in branch explosion and low parsing accuracy.

[0034] (4) The fixed-depth tree matching module is the core part of the algorithm. This module mainly uses a fixed-depth prefix tree and the unique value statistics of each position to find the template of the log.

[0035] The following sections will explain each step in detail.

[0036] See Figure 2The fixed-depth prefix tree constructed in this embodiment can be configured according to parameters, but must be at least 3. The layers are as follows: the root node is `root`, the second-level node is the first constant token appearing in the cache pool in the log, and the third-level node is the length of the token list after log splitting. If the tree depth is configured to be greater than 3, subsequent levels traverse the log splitting token list from left to right, selecting log tokens from the non-constant pool. The leaf nodes store multiple log sets, each set having an extracted log template and a list of log IDs matching that template. In addition, each log set in the leaf nodes holds a position token counter to statistically analyze the token distribution at each position. Its purpose is to determine whether a position can become a variable position when updating the template later, thus preventing the native drain algorithm from replacing tokens at the same position in the template with wildcards when position tokens are different. In addition, each log set in the leaf node also includes a minHash cache, which records the minHash value of each log template. Its purpose is to quickly calculate the similarity between the log and the template when performing variable-length log matching, thereby reducing time complexity.

[0037] There is also a special path in the tree, namely the None cluster path. This path is prepared to deal with the case where the constant pool is empty in the initial stage of the algorithm or the log does not contain the tokens in the constant pool. As the amount of logs increases, fixed constant bits will appear in the log set of the leaf nodes under the None cluster, thereby migrating the constant tokens to the constant pool and merging the relevant paths into the normal token cluster.

[0038] In addition to the tree, a constant pool is provided to store tokens that have been identified as constants in the log. The purpose of preparing this constant pool is mainly to avoid the situation where the first token in the drain algorithm is a variable, which would lead to an explosion in the number of tree branches.

[0039] See Figure 3 The flowchart for token segmentation illustrates the logic of the current algorithm's log segmentation. Compared to the Drain algorithm, which can only segment logs based on symbols, the current algorithm incorporates some Chinese semantic word segmentation capabilities, enabling it to segment Chinese logs. The specific logic is as follows: First, split the log into tokens according to English delimiters, such as spaces, commas, hyphens, slashes, punctuation marks, and @ symbols, to divide the log into more dispersed tokens as much as possible.

[0040] The second step involves the algorithm iterating through the token list segmented in the first step and determining whether each token can be further segmented into Chinese characters, i.e., determining whether Chinese characters exist in each token list.

[0041] The third step is to extract consecutive Chinese characters from the token list during the process of traversing the token list and then segment them using a Chinese word segmentation algorithm.

[0042] Finally, once the iteration is complete, the split tokens are merged into a list and passed to the next module for processing.

[0043] This step improves the segmentation mechanism, effectively supporting the parsing of logs containing Chinese and mixed Chinese and English text, and expands its segmentation capabilities compared to the Drain algorithm.

[0044] See Figure 4 To obtain the first constant token with distinct characteristics, the algorithm, after obtaining the partitioned token list, will attempt to extract the first constant token in the list. The logic is as follows: (1) Iterate through the token list and check if the corresponding token exists in the constant cache pool.

[0045] (2) If the current token exists in the cache pool, mark the token as the first feature constant.

[0046] (3) Otherwise, mark it as None.

[0047] This step, by extracting constant information and combining it with the subsequent search algorithm, can effectively solve the branch explosion problem caused by the first token being a variable in the Drain algorithm.

[0048] See Figure 5 For the fixed-depth tree search process, after the first variable token is extracted, the log will enter the fixed-depth prefix tree constructed by the algorithm for template matching (search). The specific logic is as follows: (1) Enter the root node in the tree and perform a depth-down traversal.

[0049] (2) Based on the first feature constant token obtained in the previous step, check whether there is a node with the value token in the second layer node, that is, select the branch based on the first feature constant token.

[0050] (3) After entering a specific branch, continue to check the third-level nodes according to the length of the token list after log splitting, and determine whether there is a node in the path whose corresponding value is the length of the token list. If the node exists, proceed to step (4) to continue searching downwards; otherwise, create a new path branch. The downward search means traversing the token list from left to right, but it is necessary to skip the tokens that were previously selected as feature constants.

[0051] (4) If there is a node with the length of the token list, continue searching towards the leaf node of the tree. If a node with the current token as its value is not found in a certain layer during the search, a new path branch is created; otherwise, continue traversing to the leaf node.

[0052] (5) After the leaf node is found, the algorithm will calculate the similarity between the current log and the templates in multiple log groups in the leaf node, and find the log template that is most similar to the current log.

[0053] (6) Further, when a matching log template is found, the template content is updated according to specific conditions, and the inter-cluster (inter-path) merging logic is triggered.

[0054] (7) Finally, the matching template will be returned.

[0055] See Figure 6 To select branches based on characteristic constants, this process corresponds to... Figure 5 The logic for step (2) of the corresponding content is as follows: (1) If the first characteristic constant is None.

[0056] (2) Then enter the cluster (path) with the value of None in the second level of the tree for searching.

[0057] (3) Otherwise, the search proceeds to the regular branch, that is, the search is performed at the node where the value is the first feature constant token.

[0058] See Figure 7 To illustrate the branch selection process based on characteristic constants, the process of creating new path branches is explained. Its logic is as follows: (1) First, determine the depth of the current search. If the current level is 3, that is, there is no node with a value of the length of the token list, then create a node with that length.

[0059] (2) Then, iterate through the remaining tokens and create nodes in the direction of the leaf nodes.

[0060] (3) If a leaf node has been reached, a new log set is constructed in the leaf node, and the ID of the current log is added to the ID list in the log set. At the same time, the current log is set as the template of the new log set.

[0061] (4) Finally, a new token counter is constructed to count the unique values ​​and context of each position. This counter can effectively avoid template over-generalization.

[0062] See Figure 8 This describes the similarity calculation process.

[0063] (1) First, it will get all the log sets under the current leaf node and extract all the log templates.

[0064] (2) Iterate through all the log templates, calculate the similarity between the template and the current log token list based on Jaccard similarity, and finally find the most similar log template 1.

[0065] (3) If the similarity between the most similar template and the current log is greater than the configured similarity threshold, then try to update the template.

[0066] (4) Otherwise, it means that the current log and the searched log template are not particularly similar and need to be classified separately. Therefore, a new log collection is created in the current leaf node, and the ID of the current log is added to the ID list in the log collection. At the same time, it is set as the template of the new log collection.

[0067] See Figure 9 This is to update the template and trigger the inter-cluster merge algorithm.

[0068] (1) First, it will iterate through the templateToken of the matched log template and find the token at the same position of the new log.

[0069] (2) Update the counter information. The updated information is as follows: 1) Current location Cumulative number of token samples .

[0070] 2) Current location The most frequently occurring word and its quantity .

[0071] By setting this information, the algorithm effectively saves the statistical information of the token at the current position and integrates the positional context information of the tokens before and after it. This effectively avoids the inaccuracy of determining whether the current position is a variable solely based on the entropy of the current position. In addition, only the Top-k information is saved (which can be set by the user) to prevent memory explosions in actual engineering projects.

[0072] (3) Determine if templateToken is a wildcard "*" or <number>or <hex>If yes, templateToken does not need to be changed and remains a wildcard; otherwise, it needs to be checked whether the token matches a regular expression for numbers and hexadecimal values. If both regular expressions are satisfied, templateToken is changed to... <number>or <hex>Since the algorithm has already segmented the logs as dispersedly as possible based on the delimiter, this step only requires two regular expressions to cover most industrial scenarios. In addition, the regular expression matching is performed only in the template update step. Compared with the Drain algorithm, placing the regular expression matching logic later greatly improves the algorithm's throughput and parsing efficiency.

[0073] (4) If the token cannot match the regular expression in the above steps, then based on the record of the position counter, calculate whether the current position has reached the saturation threshold according to the saturation algorithm. The threshold calculation formula is: in The stability of the current position, i.e., the percentage of the most common tokens at the current position: This represents the stability of the positions of the preceding and following neighbors: The proposed threshold calculation formula calculates the stability and context prediction capability of each position by updating a lightweight counter in real time, thereby determining whether the position is a variable. Compared with the Drain algorithm, which relies entirely on the token of a single position to independently determine whether it is a variable, this algorithm considers the co-occurrence information of the preceding and following neighborhoods and uses the top-1 frequency to replace the probability distribution entropy. While maintaining the calculation accuracy, it significantly reduces the time complexity of the algorithm and effectively improves the problems of overgeneralization and noise interference in the Drain algorithm.

[0074] (5) If the threshold is reached, it means that this position is a variable, so set templateToken to wildcard *.

[0075] (6) If the threshold is not reached, check if VarScore is approximately equal to 0. If the condition is met, add the token to the constant cache pool.

[0076] (7) If the current token is added to the constant cache pool, further determine whether the current branch is under the None cluster. If it is under the None cluster, trigger inter-cluster merging and migrate the branch to the normal cluster. Only the second-level node needs to be replaced with the first constant token that was just determined.

[0077] (8) Return to template.

[0078] See Figure 10 This is a diagram illustrating the process of matching variable-length logs with the same pattern. Figure 5 The process of searching for the most similar log template in the parse tree involves matching logs belonging to the same pattern but with different lengths. The token segmentation step cannot be completely accurate and may split log units containing delimiters such as spaces into different tokens. This results in logs belonging to the same template being distributed across different paths in the parse tree. To address this issue... Figure 10 Further optimizations have been made to the process of searching for the most similar log templates: (1) First, determine whether the current token list length layer can be found in the search process. If it can be found, then use Figure 5 The logic of the corresponding process matches the template correctly.

[0079] (2) If the length layer cannot be found, then search the adjacent length node layer. The adjacent length means that if the current log token list length is 4, then search the adjacent 3 and 5 paths again.

[0080] (3) When searching with adjacent paths, if a leaf node is found, the minHash operation is performed on the template of the log set in the leaf node and the current log, and the Jaccard similarity between the two is calculated. If it exceeds the threshold, the matching template is selected. The minHash of the template is calculated in advance when the path is created and the template is updated, and is cached in the leaf node.

[0081] (4) Return the matching template.

[0082] Example 2 Based on Embodiment 1, this embodiment provides a real-time log parsing system that integrates location information and a fixed-depth tree, used to implement the real-time log parsing method of Embodiment 1. The system includes: (1) Log acquisition and preprocessing module, used to acquire the raw logs to be parsed and parse them.

[0083] (2) Token partitioning module, used to partition the original log by English delimiter, Chinese character recognition and word segmentation to obtain a token list.

[0084] (3) The first feature constant token extraction module is used to obtain the first feature constant token by traversing the token list and the pre-built constant cache pool.

[0085] (4) Target log template matching module, which is used to first select the corresponding branch based on the first feature constant token in the pre-constructed fixed depth tree, and then match the corresponding node based on the length of the token list. If the matching fails, a new path branch is created. If it is, the token list is traversed and the search is performed along the direction of the leaf node. If the search fails, a new path branch is created. If the search is successful, the target log template with the highest similarity to the original log is matched from multiple log templates in the leaf node.

[0086] (5) Template update and branch merging module, used to traverse each position in the target log template, update the positions that meet the regular expression or the current position saturation condition to wildcards, merge branches based on the current position saturation, obtain the final log template, and realize log parsing.

[0087] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.< / hex> < / number> < / hex> < / number>

Claims

1. A real-time log parsing method that fuses position information and fixed-depth trees, characterized by, The method comprises the following steps: obtaining original logs to be parsed; performing token division on the original logs by English delimiter segmentation, Chinese character recognition and word segmentation to obtain a token list; traversing the token list, searching a pre-constructed feature constant cache pool, and attempting to match a first feature constant token of the original logs; judging whether the matching of the first feature constant token is successful, if not, attributing the token list to a None cluster, sequentially constructing each layer node of a tree and returning a log processed by a regular expression as a template, if yes, entering different levels according to the constant token, the length of the token list and the prefix of the token list to match nodes; if the node matching is successful, calculating the similarity between the final leaf node and the template of the corresponding multiple log groups, returning a log template with the highest similarity, traversing each position of the log template with the highest similarity, updating positions meeting the conditions of the regular expression or the current position saturation to wildcards, merging branches based on the current position saturation, and obtaining a final log template; if the matching is unsuccessful, constructing tree branches downward along the current level and returning a log processed by a regular expression as a template.

2. The real-time log parsing method of claim 1, wherein The process of updating the positions meeting the conditions of the regular expression or the current position saturation to wildcards comprises the following steps: traversing the token list and the tokens of each position of the log template, updating the counter information in the corresponding leaf node, including the number of token samples accumulated at each position, the token with the highest frequency at each position and the number of the token, and retaining top-K token pairs with the highest frequency at each position; in the traversal process, judging whether the current position of the log template is a wildcard, if yes, retaining the wildcard of the current position, if no, entering a regular matching process; in the regular matching process, judging whether the token of the current position matches a regular expression of a number or a hexadecimal number, if yes, updating the token of the current position to a wildcard, if no, calculating the saturation of the current position based on the counter information according to the stability of the current position and the stability of the neighboring positions before and after the current position, and judging whether to update the token of the current position to a wildcard according to whether the saturation reaches a threshold.

3. The real-time log parsing method of claim 2, wherein, The process of merging branches based on the saturation of the current position comprises the following steps: based on the saturation of the current position, if a preset condition is met, adding the token of the current position to the feature constant cache pool, if the current position is under a None branch, triggering inter-branch merging and migrating the current branch to a regular branch.

4. The real-time log parsing method of claim 1, wherein, The position saturation is calculated by the following formula: ， ， ， ， ， wherein, denotes the position saturation of the token at the th position, , denotes the stability at the th position, the stability of the preceding and following neighboring positions, respectively, is the token that appears most often at the position , is the total number of times a token has appeared, is the counting function, denotes, for each , the number of times it has appeared at the th position, , is the total number of positions.

5. The real-time log parsing method of claim 1, wherein, The process of constructing tree branches downward along the current level comprises the following steps: if the node in the current search path is located at the token list length layer, creating a node corresponding to the length of the token list, and then sequentially traversing the remaining tokens of the token list to sequentially create nodes in the direction of the leaf node; If the leaf node has been reached, a log set is constructed at the leaf node, and the id of the current raw log is added to the id list in the log set, while the token list of the current raw log is set as the template of the new log set; A token counter is constructed at the leaf node to count the distribution of each position unique value and context.

6. The real-time log parsing method of claim 1, wherein, The process of matching the first constant token of the raw log includes the following steps: The token list is traversed to determine whether the corresponding token exists in the constant token cache pool. If yes, the matching is successful, and the current token is marked as the first constant token. If the token list is traversed and each token is not in the constant token cache pool, the first constant token is marked as None, and the matching fails. The process of selecting the corresponding branch based on the first constant token includes: It is determined whether the first constant token is None. If yes, the second layer node value None branch in the tree is entered for searching. If no, the regular branch, i.e., the node whose matching value is the first constant token, is entered for searching.

7. The real-time log parsing method of claim 1, wherein, The process of matching the target log template with the highest similarity to the raw log includes the following steps: It is determined whether a node with a length matching that of the token list is matched. If yes, the target log template with the highest similarity to the raw log is matched from the multiple log templates in the leaf node. If no, the node corresponding to the length adjacent to that of the token list is searched. If the leaf node is searched, the minHash operation is performed on the multiple log templates of the leaf node and the current log, and the jaccard similarity is calculated. If the threshold is exceeded, the matching target log template is obtained.

8. The real-time log parsing method of claim 1, wherein, The process of token division by English delimiter segmentation, Chinese character recognition and word segmentation includes the following steps: The raw log is segmented according to the English delimiter; It is determined whether there is a Chinese character by traversing each token after segmentation. If yes, the continuous Chinese is extracted and Chinese word segmentation is performed. The segmented and word segmented tokens are combined into a list to obtain the token list.

9. The real-time log parsing method of claim 1, wherein, After obtaining the raw log to be parsed, the log string is extracted from the JSON format raw log, the redundant spaces are cleaned, and the algorithm uses the regular expression defined by the user to filter the information.

10. A real-time log parsing system that fuses location information and fixed-depth trees, characterized by, The system for implementing the real-time log parsing method according to any one of claims 1-9 includes: A log acquisition and preprocessing module for acquiring the raw log to be parsed and performing parsing; A token division module for token division of the raw log by English delimiter segmentation, Chinese character recognition and word segmentation to obtain a token list; A first constant token extraction module for obtaining the first constant token based on the token list and a pre-constructed constant token cache pool by traversal; A target log template matching module for matching the target log template with the highest similarity to the raw log based on the first constant token. The target log template matching module is configured to, in a pre-constructed fixed-depth tree, first select a corresponding branch based on the first feature constant token, then match a corresponding node based on the length of the token list, if the matching fails, create a new path branch, if not, traverse the token list, search in the direction of the leaf node, if the search fails, create a new path branch, if the search succeeds, from multiple log templates in the leaf node, match a target log template with the highest similarity to the original log; The template updating and branch merging module is configured to traverse each position in the target log template, update positions meeting a regular expression or a current position saturation condition as wildcards, obtain a final log template, and realize log parsing.

Images