Heuristic online log parsing method and system based on adaptive deep parse tree
By adopting an adaptive deep parsing tree structure and a heuristic field selection strategy, combined with domain knowledge preprocessing, the efficiency and accuracy issues of log parsing algorithms in large-scale and heterogeneous environments are solved, achieving efficient and accurate log parsing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2024-12-31
- Publication Date
- 2026-05-01
AI Technical Summary
Existing log parsing algorithms are inefficient and inaccurate when processing large-scale, highly dynamic, and heterogeneous log datasets, and are difficult to adapt to complex and diverse log data structure changes.
An adaptive depth-based parsing tree structure is adopted, combined with heuristic field selection and domain-general knowledge preprocessing. The depth and branches of the tree are dynamically adjusted, and the log template is quickly identified through an alternating first and last matching strategy, which reduces computational complexity and improves parsing efficiency.
It achieves efficient and accurate parsing on large-scale and complex log datasets, can respond to fault events in real time, adapt to diverse and dynamically changing log formats, and improves parsing speed and accuracy.
Smart Images
Figure CN119835142B_ABST
Abstract
Description
A Heuristic Online Log Parsing Method and System Based on Adaptive Depth-Level Parsing Trees Technical Field
[0001] This invention relates to the field of log data parsing technology, and in particular to a heuristic online log parsing method and system based on adaptive deep parse trees. Background Technology
[0002] With the rapid development of information technology, log data has become crucial input data in various application scenarios such as system operation, fault diagnosis, and performance monitoring. Especially in complex computing environments such as distributed systems, supercomputers, and mobile devices, the volume and diversity of log data generation are increasing dramatically, and the structure of log data is becoming increasingly complex. This makes log parsing increasingly difficult, and traditional parsing methods can no longer meet the needs of modern large-scale, dynamically changing, and diverse log datasets.
[0003] 1. Challenges of Log Parsing. Log parsing aims to convert unstructured or semi-structured log messages into structured data for further analysis. However, log data has several key characteristics that present challenges for log parsing: Large volume and high real-time requirements: Especially in distributed systems, the sheer volume of log data and the high real-time demands necessitate efficient and scalable parsing algorithms. Diversity and heterogeneity of log data: Different operating systems, applications, and platforms generate log data with significantly different formats, making it difficult for a single parsing method to adapt to various data formats. Complex and dynamically changing log structures: Many modern application systems dynamically adjust their log formats and structures based on changes in the operating environment, requiring parsing methods to possess dynamic adaptability.
[0004] 2. Shortcomings of Existing Technologies. Currently, existing log parsing algorithms largely rely on frequent pattern mining, cluster analysis, and heuristic rules. However, these methods often suffer from the following shortcomings when dealing with large-scale, highly dynamic, and heterogeneous log datasets: Frequent pattern mining algorithms (such as LFA and FT-Tree) rely on frequent patterns in log messages to extract log templates, but their computational complexity is high and parsing efficiency is poor when the log data volume is huge and the field distribution is uneven. Cluster analysis algorithms (such as LogSig) cluster log messages by calculating similarity, but the clustering process requires calculating a large amount of similarity, which is particularly computationally expensive on large-scale datasets. Heuristic rule-based algorithms (such as Drain and Spell) typically rely on manually set rules or fixed deep tree structures, lacking sufficient flexibility and unable to effectively adapt to dynamically changing log data formats. These methods all face problems of low processing efficiency, poor accuracy, and poor adaptability to data changes to varying degrees, especially when dealing with large-scale, dynamically changing log data, where their performance significantly deteriorates.
[0005] 3. The need for efficient and accurate log parsing. To address these issues, modern log parsing technology requires a solution that balances high efficiency and high accuracy, especially in large-scale datasets and heterogeneous environments, capable of automatically adapting to changes in log data structure and responding to fault events in real time. Therefore, developing an algorithm with high adaptability, low time complexity, and the ability to parse logs from different sources and formats in real time has become an urgent need. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides a heuristic online log parsing method and system based on adaptive deep parse trees; it aims to solve the problems of low efficiency and poor accuracy in log parsing in existing technologies, and can significantly improve parsing speed and accuracy, especially when dealing with large-scale and complex log datasets.
[0007] On the one hand, a heuristic online log parsing method based on adaptive depth parse trees is provided, including:
[0008] Obtain the set of log messages to be parsed and preprocess the set of log messages to be parsed;
[0009] Construct a parse tree, which includes a root node and is connected to several log length matching nodes; each log length matching node is the root node of a subtree.
[0010] The number of fields in each log message is input into different log length matching nodes. Based on the matching results, the preprocessed log messages are matched into the relevant subtrees. The depth of the current subtree is determined based on the length of the preprocessed log messages and the preset fitness. Using an alternating first and last method, several fields of the preprocessed log messages are matched into the nodes of the current subtree. After traversing multiple levels of nodes, the leaf node of the current subtree is reached. Each leaf node includes several log groups. Each log group includes a log template and a corresponding list of log IDs.
[0011] In the leaf nodes of the current subtree, find the log group that best matches the preprocessed log message; update the log template and log ID list of the best matching log group, which are the structured data of the log message to be parsed; after all log messages have been processed, the structured data of the log message set to be parsed is obtained.
[0012] On the other hand, a heuristic online log parsing system based on adaptive depth parse trees is provided, including:
[0013] The acquisition module is configured to: acquire a set of log messages to be parsed and preprocess the set of log messages to be parsed;
[0014] The building module is configured to: build a parse tree, the parse tree including a main root node, the main root node being connected to several log length matching nodes; each log length matching node is the root node of a subtree;
[0015] The matching module is configured to: input the number of fields of each log message into different log length matching nodes; match the preprocessed log messages to the relevant subtrees based on the matching results; determine the depth of the current subtree based on the length of the preprocessed log messages and the preset fitness; match several fields of the preprocessed log messages to the nodes of the current subtree using an alternating first-to-last method; and reach the leaf node of the current subtree after traversing multiple levels of nodes. Each leaf node includes several log groups, and each log group includes a log template and a corresponding list of log IDs.
[0016] The output module is configured to: find the log group that best matches the preprocessed log message in the leaf node of the current subtree; update the log template and log ID list of the best matching log group, wherein the log template and log ID list of the best matching log group are structured data of the log message to be parsed; and obtain the structured data of the set of log messages to be parsed after all log messages have been processed.
[0017] Furthermore, an electronic device is also provided, including:
[0018] Memory, used for non-transitory storage of computer-readable instructions; and
[0019] Processor, for executing the computer-readable instructions,
[0020] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.
[0021] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method described in the first aspect is performed.
[0022] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.
[0023] The above technical solution has the following advantages or beneficial effects:
[0024] Based on an adaptive deep tree structure, combined with heuristic algorithms and domain-general knowledge preprocessing, the log parsing process is optimized, which can effectively adapt to the diversity and dynamic changes of log data, and improve the accuracy and efficiency of parsing.
[0025] 1. Adaptive Depth Tree Structure: One of the core innovations of this invention is the construction of an adaptive depth tree structure. By adaptively adjusting the depth and branches of the tree structure, the tree's shape is dynamically optimized based on the characteristics of the log messages (such as the number of fields, field types, and their variation patterns). Specifically, during parsing, the tree's depth and number of branches are automatically adjusted according to the complexity of the log, thereby reducing redundant calculations and improving parsing speed while ensuring parsing accuracy. This adaptive mechanism enables the algorithm to flexibly adapt to log data from different sources, in different formats, and with varying frequencies of change.
[0026] 2. Heuristic Field Selection Strategy: To further improve parsing efficiency, this invention introduces a heuristic field selection strategy. During log parsing, this invention constructs log groups by alternately selecting the first and last fields of the log. This strategy, based on the structural characteristics of log data, can effectively identify key fields in log messages, quickly locate log templates, thereby reducing unnecessary field interference and shortening parsing time. Compared with the traditional field-by-field matching method, this method not only improves parsing speed but also enhances the algorithm's adaptability to complex and irregular log data.
[0027] 3. Domain-General Knowledge-Driven Preprocessing: To improve parsing accuracy and reduce computational complexity, this invention incorporates domain-general knowledge for preprocessing log data. Domain knowledge preprocessing primarily utilizes rule engines and pattern recognition technology to perform preliminary analysis of the log data, simplifying and labeling log messages according to their potential structure. This allows the algorithm to avoid interference from irrelevant data during parsing, identify the structural characteristics of log templates in advance, and facilitate faster parsing. This method not only improves parsing accuracy but also effectively reduces computational overhead, particularly noticeable on large-scale log datasets.
[0028] In summary, the heuristic online log parsing algorithm based on adaptive deep trees of this invention achieves efficient and accurate log parsing through flexible tree structure adjustment, heuristic field selection, and domain knowledge preprocessing. This innovative method has significant advantages, especially in its application to large-scale and complex log datasets. Through the adaptive deep tree structure, the algorithm can dynamically adjust the depth and branches of the parsing tree according to the characteristics of the log data, flexibly adapting to diverse and complex log formats, thereby improving parsing efficiency and accuracy. The heuristic field selection strategy helps to quickly identify core fields and reduce interference from irrelevant fields, further improving parsing speed. Simultaneously, combining domain-general knowledge preprocessing reduces the computational complexity of parsing, improving the algorithm's accuracy and overall performance.
[0029] This method effectively overcomes the shortcomings of traditional log parsing algorithms when handling large volumes of diverse and dynamically changing logs. It demonstrates superior parsing capabilities, particularly in heterogeneous environments such as large-scale distributed systems, operating systems, and mobile systems, exhibiting both real-time performance and high efficiency. During log parsing, this algorithm can flexibly extract key information from log data of different formats and sources, and achieves efficient processing on large datasets, meeting the demands for high accuracy and real-time performance in practical applications. This represents a significant breakthrough in large-scale log parsing tasks. Attached Figure Description
[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0031] Figure 1 is an example diagram summarizing the log parsing task.
[0032] Figure 2 is a flowchart of a heuristic online log parsing algorithm based on adaptive deep trees.
[0033] Figure 3 shows an example of an adaptive deep tree structure.
[0034] Figures 4(a) and 4(b) are examples of parse tree structure updates.
[0035] Figures 5(a) to 5(c) show the comparison of experimental results on log parsing efficiency. Detailed Implementation
[0036] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0037] Example 1
[0038] This embodiment provides a heuristic online log parsing method based on an adaptive depth parsing tree;
[0039] As shown in Figure 2, the heuristic online log parsing method based on adaptive depth parse tree includes:
[0040] S101: Obtain the set of log messages to be parsed and preprocess the set of log messages to be parsed;
[0041] S102: Construct a parse tree, the parse tree including a root node, the root node being connected to several log length matching nodes; each log length matching node is the root node of a subtree;
[0042] S103: Input the number of fields of each log message into different log length matching nodes. Based on the matching results, match the preprocessed log messages into the relevant subtrees. Determine the depth of the current subtree based on the length of the preprocessed log messages and the preset fitness. Using an alternating first and last method, match several fields of the preprocessed log messages into the nodes of the current subtree. After traversing multiple levels of nodes, reach the leaf node of the current subtree. Each leaf node includes several log groups. Each log group includes a log template and a corresponding list of log IDs.
[0043] S104: In the leaf nodes of the current subtree, find the log group that best matches the preprocessed log message; update the log template and log ID list of the best matching log group, where the log template and log ID list of the best matching log group are the structured data of the log message to be parsed; after all log messages have been processed, the structured data of the log message set to be parsed is obtained.
[0044] Furthermore, after inputting the number of fields of each log message into different log length matching nodes, the process includes: if the number of fields of the current log message is not the same as the log length of any log length matching node, then a new log length matching node is added, and the log length of the newly added log length matching node is the number of fields of the current log message.
[0045] Furthermore, after determining the depth of the current subtree based on the length of the preprocessed log message and the preset fitness, the process includes: if the actual depth of the current subtree is h and the determined depth of the current subtree is H, then supplementing the current subtree with... layer.
[0046] Furthermore, the method of using alternating beginning and end to match several fields of the preprocessed log message to the nodes of the current subtree includes: if there is no matching node in the current subtree, then a new node is added to the current subtree, and the content of the new node is set to the corresponding field.
[0047] Adding a new node to the current subtree includes two forms:
[0048] Form 1: Generate a sibling node B next to an existing node A. Sibling node B shares the parent node with node A.
[0049] Form 2: Generate a child node a below the existing node A, which is connected to node A. The parent node of child node a is node A.
[0050] If the content of the newly added node and node A belongs to the same log message, then form two is used;
[0051] If the content of the newly added node and node A does not belong to the same log message, then form one is used.
[0052] Furthermore, after traversing multiple levels of nodes and reaching the leaf node of the current subtree, the process includes:
[0053] If there is no log group in the current leaf node that matches the similarity of the current log message, then a new log group is created, and the log template and the corresponding log ID list of the current log message are filled into the log group.
[0054] If a log group with a similarity to the current log message exists in the current leaf node, then it is determined whether the content of the similarity-matching log group is consistent with the content of the current log message. If they are consistent, no update is made; otherwise, the content of the similarity-matching log group is updated.
[0055] Further, S101: obtaining the log message to be parsed includes: obtaining the log message to be parsed by web crawling.
[0056] The log message to be parsed includes: a fixed part (log template) and a variable part (log parameters).
[0057] Furthermore, the preprocessing of the log messages to be parsed includes:
[0058] (1-1) Process the setting symbols in the log message by adding spaces before and after the setting symbols to separate the setting symbols from the log content and make them independent fields;
[0059] (1-2) Replace the delimiter in the log message with a space, and divide the log message into several fields based on the space;
[0060] (1-3) Replace dynamic fields in log messages with wildcards.
[0061] For example, (1-1) for certain symbols in the log message (such as '=', '(', ')', '[',' ]', '{', '}'), spaces are added before and after the symbols to separate these symbols from the log content, making them independent fields. For example, the symbol =10 will become "space=space10".
[0062] For example, (1-2) replaces delimiters (such as ',', ';', '|') in the log with spaces, and splits the log message into multiple fields based on the space delimiter. This process ensures that each field is an independent part of the log message, facilitating subsequent parsing operations.
[0063] For example, (1-3) uses regular expressions common in the design domain to match dynamic parts of the log (such as IP addresses, timestamps, numbers, paths, etc.) and replaces these dynamic fields with wildcards "<*>". For example, the matched IP address "10.250.19.16" will be replaced with "<*>", and the date "2024.10.01" will also be replaced with "<*>".
[0064] It should be understood that upon arrival of log messages, they are first preprocessed to be converted into a standard format that is easy to parse.
[0065] As shown in Figure 1, the log parsing task aims to convert unstructured or semi-structured log messages into structured data. This invention provides an online log parsing method, AD-Tree, based on an Adaptive Depth Tree. By dynamically adjusting the tree depth and employing flexible field matching strategies, it achieves efficient and accurate log parsing. The structure of the parsing tree is shown in Figure 3.
[0066] Furthermore, the number of fields for each log message is input into different log length matching nodes. Based on the matching results, the preprocessed log messages are matched into relevant subtrees, including:
[0067] Analyze the number of fields M in the log message, input the number of fields M into the log length matching node. If the log length of the log length matching node is also M, it means that the current log message has successfully matched the current subtree, and the current log message is allowed to match the corresponding subtree.
[0068] If the log length of the matching node is not equal to M, it means that the current log message fails to match the current subtree, and the current log message is not allowed to match the current subtree.
[0069] It should be understood that after preprocessing is complete, the parsing of log messages begins. First, an appropriate subtree is selected at the root node of the tree based on the number of fields (i.e., length) of the log message, and then a matching log group is searched within that subtree.
[0070] For example, at the first level of the parse tree, nodes are divided based on the number of fields in the log message. The parse tree selects the appropriate subtree for matching based on the number of fields in the log message (i.e., message length). For instance, if the log message contains 6 fields, the parse tree will go to the subtree associated with "Length=6". This effectively reduces the number of nodes that need to be compared, thereby improving matching efficiency.
[0071] Further, determining the depth of the current subtree based on the length of the preprocessed log message and a preset fitness includes:
[0072] Calculate the length of the preprocessed log message and the score of the preset fitness. Round the score to an integer to get the depth of the current subtree. The depth of the subtree is also the total number of levels of the nodes in the subtree.
[0073] For example, after selecting a suitable subtree, the depth of the subtree is calculated based on the length of the log message and a predefined fitness degree. For instance, when degree = 0.4 and the log length is 8, the depth of the subtree will be round(8*0.4) = 3, meaning that matching will be performed on 3 levels of nodes within the subtree. This depth value is adaptively adjusted according to the log length to ensure optimal matching processing for logs of different lengths.
[0074] Furthermore, the method of alternating between the first and last fields of the preprocessed log message is used to match several fields to nodes in the current subtree, including:
[0075] The preprocessed log messages are divided into several fields, and the fields of the log messages are sequentially numbered from 1 to N.
[0076] First, take the field with number 1, and match the field with number 1 with the nodes of the current subtree;
[0077] If the content of field number 1 is different from the content of the first-level node of the current subtree, it means that the match failed. Then, a new node is added in the first level of the current subtree, and the content of the new node is set to the field number 1.
[0078] If the content of field number 1 is the same as the content of the first-level node of the current subtree, then the match is successful; if the match is successful, then in an alternating manner, field number N is taken and matched with the second-level node of the current subtree.
[0079] If the content of field number N is the same as the content of the second-level node of the current subtree, it means that the match is successful. Then, using the alternating first and last method, the node numbered 2 is matched with the third-level node of the current subtree.
[0080] If the content of field number N is different from the content of the second-level node of the current subtree, it means that the match failed. Then, a new node is added in the second level of the current subtree, and the content of the new node is set to the field number N.
[0081] This process continues until all fields of the preprocessed log message match the nodes of the current subtree, or the depth and breadth of the current subtree reach the set thresholds, at which point the process ends.
[0082] It should be understood that the alternation of the first and last fields means taking the first field, then the Nth field, then the second field, then the (N-1)th field, then the third field, then the (N-2)th field, then the fourth field, then the (N-3)th field, and so on.
[0083] It should be understood that, within the depth of the parse tree, matching is performed according to the field order of the log message. Field matching uses an alternating approach, starting with the first and ending each time, to optimize the search path. For example, for the log message "PacketResponder<*>forblock blk_3328 terminating", the parse tree will first match the field "PacketResponder" with nodes in the tree, and then continue matching "terminating". During the matching process, if a field contains numbers (such as blk_3328), the parse tree will use special "<->" nodes for matching to avoid generating too many child nodes.
[0084] To avoid branch explosion, the maximum number of child nodes for each node is controlled during the parse tree search. When the number of child nodes of a node reaches the maximum value, any mismatched marker will be directly matched to the special node "<->" under that node, ensuring that the depth and breadth of the tree do not expand excessively.
[0085] Furthermore, after traversing multiple levels of nodes, the process reaches the leaf node of the current subtree. Each leaf node includes several log groups, and each log group includes a log template and a corresponding list of log IDs.
[0086] A log template is an abstract form generated during log parsing. It contains the core information of system operation and is used to represent a type of log message with similar structure. It retains the fixed / constant parts of the log message and replaces the variable parameter parts with wildcards.
[0087] The log list represents the collection of log messages belonging to the current log template, and records the IDs of these log messages;
[0088] The log template is extracted from a fixed part of the log message and is used to describe the characteristics of the log group; the log ID list records the unique identifiers of the log messages in the group. Once a leaf node matching a log message is found, it is necessary to further determine the degree of matching between the log group under that node and the current log message.
[0089] Further, S103: In the leaf nodes of the current subtree, find the log group that best matches the preprocessed log message, including:
[0090] (2-1): For each log group, calculate the similarity between its template and the current log message. The similarity calculation is performed by comparing the matching degree of the two fields, as shown in the following formula:
[0091]
[0092] in, and The first and second log messages and log templates are respectively One field, The function is used to determine whether two fields match. If the two fields are the same, they are considered a match; if the two fields are different, they are considered a mismatch. Number of fields.
[0093] (2-2): Selecting matching log groups: When the similarity between a log message and a certain log group is greater than a preset threshold, the log message is considered to be a successful match. If multiple log groups have the same similarity, the log group containing the most wildcards "<*>" will be selected as the final matching result.
[0094] If a log group matching the log message is found in the parse tree, update the log ID list for that log group and update the template. If no matching log group is found, a new log group needs to be created, and the parse tree structure may need to be updated.
[0095] Further, S104: updating the log template and log ID list of the best matching log group includes:
[0096] If a matching log group is found, compare the fields in the log message and the log template one by one;
[0097] If the fields are the same, keep the template fields unchanged;
[0098] If the fields are inconsistent, the template field will be replaced with the wildcard "<*>".
[0099] If no matching log group is found, a new log group is created; the template of the new log group is exactly the same as the current log message, and the ID list contains only the unique identifier of the current log message.
[0100] After creating a new log group, the structure of the parse tree is updated based on the characteristics of the logs.
[0101] At this point, the new log group will be inserted into the corresponding position in the tree, and new nodes will be created or existing nodes will be adjusted. Examples of parsing tree structure updates are shown in Figures 4(a) and 4(b). Figures 5(a) to 5(c) are comparison charts of log parsing efficiency experimental results.
[0102] Table 1. Comparison of Experimental Results on Log Parsing Accuracy
[0103]
[0104] Example 2
[0105] This embodiment provides a heuristic online log parsing system based on an adaptive depth-based parse tree, including:
[0106] The acquisition module is configured to: acquire a set of log messages to be parsed and preprocess the set of log messages to be parsed;
[0107] The building module is configured to: build a parse tree, the parse tree including a main root node, the main root node being connected to several log length matching nodes; each log length matching node is the root node of a subtree;
[0108] The matching module is configured to: input the number of fields of each log message into different log length matching nodes; match the preprocessed log messages to the relevant subtrees based on the matching results; determine the depth of the current subtree based on the length of the preprocessed log messages and the preset fitness; match several fields of the preprocessed log messages to the nodes of the current subtree using an alternating first-to-last method; and reach the leaf node of the current subtree after traversing multiple levels of nodes. Each leaf node includes several log groups, and each log group includes a log template and a corresponding list of log IDs.
[0109] The output module is configured to: find the log group that best matches the preprocessed log message in the leaf node of the current subtree; update the log template and log ID list of the best matching log group, wherein the log template and log ID list of the best matching log group are structured data of the log message to be parsed; and obtain the structured data of the set of log messages to be parsed after all log messages have been processed.
[0110] It should be noted that the acquisition module, construction module, matching module, and output module described above correspond to steps S101 to S104 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.
[0111] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0112] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.
[0113] Example 3
[0114] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.
[0115] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.
[0116] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.
[0117] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.
[0118] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.
[0119] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.
[0120] Example 4
[0121] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.
[0122] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A heuristic online log parsing method based on adaptive depth parse tree, characterized by: include: Obtain the set of log messages to be parsed and preprocess the set of log messages to be parsed; Construct a parse tree, which includes a root node and is connected to several log length matching nodes; each log length matching node is the root node of a subtree; input the number of fields of each log message into different log length matching nodes, and match the preprocessed log message to the relevant subtree according to the matching result; The depth of the current subtree is determined based on the length of the preprocessed log message and the preset fitness. The step of determining the depth of the current subtree based on the length of the preprocessed log message and the preset fitness includes: calculating the score of the length of the preprocessed log message and the preset fitness, rounding the score to an integer to obtain the depth of the current subtree, where the depth of the subtree is also the total number of levels of nodes in the subtree; using an alternating first and last method, matching several fields of the preprocessed log message to nodes in the current subtree, first taking the first field, then the Nth field, then the second field, then the (N-1)th field, and so on, where N is the last field of the log message; if there is no matching node in the current subtree, then a new node is added to the current subtree, and the content of the new node is set to the corresponding field; Adding a new node to the current subtree includes two forms: Form 1: Create a sibling node B next to an existing node A, with sibling node B sharing a parent node with node A; Form 2: Create a child node a connected to node A below an existing node A, with the parent node of child node a being node A. If the content of the new node and node A belongs to the same log message, then Form 2 is used; if the content of the new node and node A does not belong to the same log message, then Form 1 is used. After traversing multiple levels of nodes, the current subtree's leaf node is reached. Each leaf node includes several log groups, and each log group includes a log template and a corresponding list of log IDs. In the leaf nodes of the current subtree, the log group that best matches the preprocessed log message is found. The log template and log ID list of the best-matching log group are updated, and the log template and log ID list of the best-matching log group are the structured data of the log message to be parsed. After all log messages have been processed, the structured data of the set of log messages to be parsed is obtained.
2. The heuristic online log parsing method based on adaptive deep parse tree as described in claim 1, characterized in that, After inputting the number of fields for each log message into different log length matching nodes, the process includes: if the number of fields for the current log message is not the same as the log length of any log length matching node, then a new log length matching node is added, and the log length of the new log length matching node is the number of fields for the current log message; after determining the depth of the current subtree based on the length of the preprocessed log message and the preset fitness, the process includes: if the actual depth of the current subtree is h, and the determined depth of the current subtree is H, then the current subtree will be supplemented. layer.
3. The heuristic online log parsing method based on adaptive deep parse tree as described in claim 1, characterized in that, After traversing multiple levels of nodes and reaching the leaf node of the current subtree, the process includes: if there is no log group in the current leaf node that matches the similarity of the current log message, a new log group is added, and the log template and the corresponding log ID list for the current log message are filled into the log group; if there is a log group in the current leaf node that matches the similarity of the current log message, it is determined whether the content of the similarity-matching log group is consistent with the content of the current log message. If they are consistent, no update is made; if they are inconsistent, the content of the similarity-matching log group is updated. The preprocessing of the log message to be parsed includes: processing the setting symbols in the log message by adding spaces before and after the setting symbols to separate the setting symbols from the log content, making them independent fields; replacing the separators in the log message with spaces, and dividing the log message into several fields based on the spaces; and replacing the dynamic fields in the log message with wildcards.
4. The heuristic online log parsing method based on adaptive deep parse tree as described in claim 1, characterized in that, The number of fields for each log message is input into different log length matching nodes. Based on the matching results, the preprocessed log message is matched into the relevant subtree. This includes: analyzing the number of fields M of the log message, inputting the number of fields M into the log length matching node; if the log length of the log length matching node is also M, it means that the current log message has successfully matched the current subtree and is allowed to be matched into the corresponding subtree; if the log length of the log length matching node is not equal to M, it means that the current log message has failed to match the current subtree and is not allowed to be matched into the current subtree.
5. The heuristic online log parsing method based on adaptive deep parse tree as described in claim 1, characterized in that, The method of alternating first and last fields matching several fields of the preprocessed log message to nodes in the current subtree includes: dividing the preprocessed log message into several fields, and sequentially numbering the fields of the log message according to the order 1 to N; first, taking the field numbered 1, and matching the field numbered 1 with the nodes of the current subtree; if the content of the field numbered 1 is different from the content of the first-level node of the current subtree, it indicates a match failure, then adding a new node in the first level of the current subtree, and setting the content of the new node to the field numbered 1; if the content of the field numbered 1 is the same as the content of the first-level node of the current subtree, it indicates a match success; if the match is successful, The process then proceeds in an alternating manner, taking the field numbered N and matching it against the second-level nodes of the current subtree. If the content of the field numbered N is the same as the content of the second-level node of the current subtree, the match is successful. The process then continues in an alternating manner, taking the node numbered 2 and matching it against the third-level nodes of the current subtree. If the content of the field numbered N is different from the content of the second-level node of the current subtree, the match fails. A new node is then added to the second level of the current subtree, and its content is set to the field numbered N. This process continues until all fields of the preprocessed log message match the nodes of the current subtree, or the depth and breadth of the current subtree reach the set thresholds, at which point the process ends.
6. The heuristic online log parsing method based on adaptive deep parse tree as described in claim 1, characterized in that, In the leaf nodes of the current subtree, find the log group that best matches the preprocessed log message, including: for each log group, calculate the similarity between its template and the current log message; the similarity calculation is done by comparing the matching degree of two fields, and the specific formula is as follows: in, and The first and second log messages and log templates are respectively One field, The function is used to determine whether two fields match. If the two fields are the same, they are considered a match; if the two fields are different, they are considered a mismatch. The number of fields is specified; the matching log group is selected: when the similarity between a log message and a certain log group is greater than the preset threshold, the log message is considered to be a successful match; if multiple log groups have the same similarity, the log group containing the most wildcards "<*>" will be selected as the final matching result; if a log group matching the log message is found in the parse tree, the log ID list of that log group is updated and the template is updated; if no matching log group is found, a new log group is created and the parse tree structure is updated.
7. The heuristic online log parsing method based on adaptive deep parse tree as described in claim 1, characterized in that, The log template and log ID list for the best matching log group are updated, including: if a matching log group is found, the fields in the log message and log template are compared one by one; if the fields match, the template fields remain unchanged; if the fields do not match, the template fields are replaced with wildcards "<*>"; if no matching log group is found, a new log group is created; the template of the new log group is exactly the same as the current log message, and the ID list contains only the unique identifier of the current log message; after creating the new log group, the structure of the parse tree is updated according to the characteristics of the log.
8. A heuristic online log parsing system based on adaptive depth parse tree, employing the heuristic online log parsing method based on adaptive depth parse tree as described in any one of claims 1-7, characterized in that, include: The acquisition module is configured to: acquire a set of log messages to be parsed and preprocess the set of log messages to be parsed; The building module is configured to: build a parse tree, the parse tree including a main root node, the main root node being connected to several log length matching nodes; each log length matching node is the root node of a subtree; The matching module is configured to: input the number of fields of each log message into different log length matching nodes, and match the preprocessed log messages into the relevant subtrees based on the matching results; The depth of the current subtree is determined based on the length of the preprocessed log message and the preset fitness; several fields of the preprocessed log message are matched to the nodes of the current subtree using an alternating first and last method. After traversing multiple levels of nodes, we reach the leaf node of the current subtree. Each leaf node includes several log groups. Each log group includes a log template and a corresponding list of log IDs. The output module is configured to: find the log group that best matches the preprocessed log message in the leaf node of the current subtree; update the log template and log ID list of the best matching log group, wherein the log template and log ID list of the best matching log group are structured data of the log message to be parsed; and obtain the structured data of the set of log messages to be parsed after all log messages have been processed.
9. An electronic device, characterized in that, include: A memory for non-transitory storage of computer-readable instructions; and a processor for executing the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, perform the method described in any one of claims 1-7.
10. A storage medium, characterized in that, Non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method of any one of claims 1-7 is performed.