Heuristic online log analysis method and system based on adaptive deep analysis tree

Abstract

The invention discloses a heuristic online log analysis method and system based on a self-adaptive deep analysis tree, and the method comprises the steps: constructing an analysis tree, inputting the field number of each log message into different log length matching nodes, and matching the preprocessed log message into a related sub-tree according to a matching result; determining the depth of the current sub-tree according to the length of the preprocessed log message and the preset fitness; matching a plurality of fields of the preprocessed log message into nodes of the current sub-tree by adopting a head-to-tail alternation mode; after multi-layer node traversal, leaf nodes of the current sub-tree are reached; finding a log group which is most matched with the preprocessed log message in leaf nodes of the current sub-tree; and updating the log template and the log ID list of the most matched log group, and after all log messages are processed, obtaining structured data of the to-be-analyzed log message set.

Description

Technical Field

[0001] The present invention relates to the technical field of log data parsing, and in particular to a heuristic online log parsing method and system based on an adaptive deep parsing tree. Background Art

[0002] With the rapid development of information technology, log data has become an important input data in many application scenarios such as system operation, fault diagnosis, and performance monitoring. Especially in complex computing environments such as distributed systems, supercomputers, and mobile devices, the amount and diversity of log data generated are becoming increasingly large, and the structure of log data is becoming increasingly complex. This makes the task of log parsing increasingly difficult, and traditional parsing methods can no longer meet the needs of modern large-scale, dynamically changing, and diverse log data sets.

[0003] 1. Challenges of log parsing. Log parsing aims to convert unstructured or semi-structured log messages into structured data for further analysis. However, log data has the following characteristics, which bring challenges to log parsing. Large volume of log data and strong real-time requirements: Especially in distributed systems, the volume of log data is huge and the real-time requirements are high, requiring the parsing algorithm to be efficient and scalable. Diversity and heterogeneity of log data: The log data formats generated by different operating systems, applications, and platforms vary greatly, which makes it difficult for a single parsing method to adapt to various data formats. The log structure is complex and changes dynamically: Many modern application systems dynamically adjust the log format and structure according to changes in the operating environment, which requires the parsing method to have dynamic adaptability.

[0004] 2. Deficiencies of the existing technology. At present, the existing log parsing algorithms mostly rely on methods such as frequent pattern mining, cluster analysis, and heuristic rules. However, these methods often have the following deficiencies on large-scale, highly dynamically changing and heterogeneous log data sets: Algorithms based on frequent pattern mining (such as LFA, FT-Tree) rely on frequent patterns in log messages to extract log templates, but when the amount of log data is huge and the field distribution is uneven, its computational complexity is high and the parsing efficiency is poor. Algorithms based on cluster analysis (such as LogSig) cluster by calculating the similarity between log messages, but the clustering process requires calculating a large number of similarities, especially on large-scale data sets, and the computational time overhead is too large. Algorithms based on heuristic rules (such as Drain, Spell) usually rely on manually set rules or fixed deep tree structures, lack sufficient flexibility, and cannot effectively adapt to dynamically changing log data formats. These methods all face the problems of low processing efficiency, poor accuracy, and poor adaptability to data changes to varying degrees, especially when facing large-scale, dynamically changing log data, their performance is significantly reduced.

[0005] 3. The need for efficient and accurate log parsing. To solve these problems, modern log parsing technology requires a solution that can balance high efficiency and high accuracy, especially in large-scale data sets and heterogeneous environments, and can automatically adapt to changes in log data structures and respond to failure events in real time. Therefore, it is an urgent need to develop an algorithm with high adaptability, low time complexity, and the ability to parse logs from different sources and formats in real time. Summary of the invention

[0006] In order to address the deficiencies of the prior art, the present invention provides a heuristic online log parsing method and system based on an adaptive deep parsing tree, aiming to solve the problems of low efficiency and poor accuracy of log parsing in the prior art, especially when processing large-scale and complex log data sets, which can significantly improve the parsing speed and accuracy.

[0007] On the one hand, a heuristic online log parsing method based on an adaptive deep parsing tree is provided, including:

[0008] Obtain a set of log messages to be parsed, and pre-process the set of log messages to be parsed;

[0009] Constructing a parse tree, the parse tree comprising a main root node, the main root node being connected to a plurality of log length matching nodes; each log length matching node being a root node of a subtree;

[0010] Input the number of fields of each log message into different log length matching nodes, and match the preprocessed log message to the relevant subtree according to the matching result; determine the depth of the current subtree according to the length of the preprocessed log message and the preset fitness; match several fields of the preprocessed log message to the nodes of the current subtree in an alternating manner; after traversing multiple layers of nodes, reach the leaf node of the current subtree, each leaf node includes several log groups, and each log group includes a log template and a corresponding log ID list;

[0011] In the leaf node of the current subtree, find the log group that best matches the preprocessed log message; update the log template and log ID list of the best matching log group, which are the structured data of the log message to be parsed; when all log messages are processed, obtain the structured data of the log message set to be parsed.

[0012] On the other hand, a heuristic online log parsing system based on an adaptive deep parsing tree is provided, including:

[0013] An acquisition module is configured to: acquire a set of log messages to be parsed, and pre-process the set of log messages to be parsed;

[0014] A construction module is configured to: construct a parse tree, the parse tree comprising a main root node, the main root node is connected to a plurality of log length matching nodes; each log length matching node is a root node of a subtree;

[0015] The matching module is configured to: input the number of fields of each log message into different log length matching nodes, and match the preprocessed log message to the relevant subtree according to the matching result; determine the depth of the current subtree according to the length of the preprocessed log message and the preset fitness; match several fields of the preprocessed log message to the nodes of the current subtree in an alternating manner; after traversing multiple layers of nodes, reach the leaf node of the current subtree, each leaf node includes several log groups, and each log group includes a log template and a corresponding log ID list;

[0016] The output module is configured to: find the log group that best matches the preprocessed log message in the leaf node of the current subtree; update the log template and log ID list of the best matching log group, wherein the log template and log ID list of the best matching log group are the structured data of the log message to be parsed; when all log messages are processed, obtain the structured data of the set of log messages to be parsed.

[0017] On the other hand, there is also provided an electronic device, comprising:

[0018] a memory for non-transitory storage of computer-readable instructions; and

[0019] a processor for executing the computer readable instructions,

[0020] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0021] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described in the first aspect is executed.

[0022] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0023] The above technical solution has the following advantages or beneficial effects:

[0024] Based on the adaptive deep tree structure, combined with heuristic algorithms and domain general knowledge preprocessing, the log parsing process is optimized, which can effectively adapt to the diversity and dynamic changes of log data and improve the accuracy and efficiency of parsing.

[0025] 1. Adaptive deep tree structure: One of the core innovations of the present invention is the construction of an adaptive deep tree structure. By adaptively adjusting the depth and branches of the tree structure, the shape of the tree is dynamically optimized according to the characteristics of the log message (such as the number of fields, field types and their changing patterns). Specifically, during the parsing process, the depth and number of branches of the tree will be automatically adjusted according to the complexity of the log, thereby reducing redundant calculations and improving the parsing speed while ensuring the accuracy of the parsing. This adaptive mechanism enables the algorithm to flexibly adapt to log data from different sources, formats and change frequencies.

[0026] 2. Heuristic field selection strategy: In order to further improve the parsing efficiency, the present invention introduces a heuristic field selection strategy. When parsing logs, the present invention constructs log groups by alternately selecting the first and last fields of the log. This strategy is based on the structural characteristics of the log data, and can effectively identify the key fields in the log message and quickly locate the log template, thereby reducing unnecessary field interference and shortening the parsing time. Compared with the traditional field-by-field matching method, this method not only improves the parsing speed, but also enhances the adaptability of the algorithm when facing complex and irregular log data.

[0027] 3. Preprocessing driven by general domain knowledge: In order to improve the accuracy of parsing and reduce the computational complexity, the present invention preprocesses the log data in combination with general domain knowledge. Domain knowledge preprocessing mainly uses rule engines and pattern recognition technology to conduct preliminary analysis of log data, simplifying and annotating log messages according to their potential structure. In this way, the algorithm can avoid interference from irrelevant data during the parsing process, identify the structural characteristics of the log template in advance, and help the parsing process proceed more quickly. This method not only improves the accuracy of parsing, but also effectively reduces the computational overhead, especially on large-scale log data sets.

[0028] In summary, the heuristic online log parsing algorithm based on the adaptive deep tree of the present invention realizes efficient and accurate log parsing through flexible tree structure adjustment, heuristic field selection, domain knowledge preprocessing and other means. This innovative method has significant advantages, especially in its application performance on large-scale and complex log data sets. Through the adaptive deep tree structure, the algorithm can dynamically adjust the depth and branches of the parsing tree according to the characteristics of the log data, and flexibly adapt to diverse and complex log formats, thereby improving parsing efficiency and accuracy. The heuristic field selection strategy helps to quickly identify core fields, reduce the interference of irrelevant fields, and further improve the parsing speed. At the same time, combined with the preprocessing of general knowledge in the field, it can reduce the computational complexity of the parsing and improve the accuracy and overall performance of the algorithm.

[0029] This method effectively overcomes the shortcomings of traditional log parsing algorithms in processing large amounts of data, diverse and dynamically changing logs, especially in applications in heterogeneous environments such as large-scale distributed systems, operating systems and mobile systems. It has excellent parsing capabilities and is real-time and efficient. In the log parsing process, this algorithm can not only flexibly parse key information from log data of different formats and sources, but also achieve efficient processing under large-scale data sets, meeting the needs of high accuracy and real-time performance in practical applications, and becoming an important breakthrough in large-scale log parsing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 An example diagram that outlines a log parsing task.

[0032] Figure 2 Flowchart of the heuristic online log parsing algorithm based on adaptive depth tree.

[0033] Figure 3 This is an example diagram of the adaptive depth tree structure.

[0034] Figure 4(a) and Figure 4(b) are examples of updating the parse tree structure.

[0035] Figure 5(a) to Figure 5(c) This is a comparison chart of the log parsing efficiency experiment results. DETAILED DESCRIPTION

[0036] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0037] Embodiment 1

[0038] This embodiment provides a heuristic online log parsing method based on an adaptive deep parsing tree;

[0039] like Figure 2 As shown in FIG. 1 , the heuristic online log parsing method based on the adaptive deep parsing tree includes:

[0040] S101: obtaining a set of log messages to be parsed, and preprocessing the set of log messages to be parsed;

[0041] S102: construct a parse tree, the parse tree including a main root node, the main root node is connected to a plurality of log length matching nodes; each log length matching node is a root node of a subtree;

[0042] S103: input the number of fields of each log message into different log length matching nodes, and match the preprocessed log message to the relevant subtree according to the matching result; determine the depth of the current subtree according to the length of the preprocessed log message and the preset fitness; match several fields of the preprocessed log message to the nodes of the current subtree in an alternating manner; after traversing multiple layers of nodes, reach the leaf node of the current subtree, each leaf node includes several log groups, and each log group includes a log template and a corresponding log ID list;

[0043] S104: Find the log group that best matches the preprocessed log message in the leaf node of the current subtree; update the log template and log ID list of the best matching log group, where the log template and log ID list of the best matching log group are structured data of the log message to be parsed; when all log messages are processed, obtain the structured data of the log message set to be parsed.

[0044] Furthermore, after inputting the number of fields of each log message into different log length matching nodes, the process includes: if the number of fields of the current log message is different from the log length of any log length matching node, adding a new log length matching node, and the log length of the new log length matching node is the number of fields of the current log message.

[0045] Further, after determining the depth of the current subtree according to the length of the preprocessed log message and the preset fitness, the method further includes: if the actual depth of the current subtree is h and the determined depth of the current subtree is H, then the current subtree will be supplemented with (Hh) layers.

[0046] Furthermore, the method of matching several fields of the preprocessed log message to the nodes of the current subtree in an alternating manner includes: if there is no matching node in the current subtree, adding a node to the current subtree and setting the content of the added node to the corresponding field.

[0047] Adding a new node to the current subtree includes two forms:

[0048] Form 1: Generate a sibling node B next to the existing node A. The sibling node B and node A share the same parent node.

[0049] Form 2: Generate a child node a connected to node A below the existing node A, and the parent node of child node a is node A;

[0050] If the content of the newly added node and node A belongs to the same log message, form 2 is used;

[0051] If the content of the newly added node and node A do not belong to the same log message, form 1 is used.

[0052] Furthermore, after traversing multiple layers of nodes and reaching the leaf node of the current subtree, the following steps are performed:

[0053] If there is no log group in the current leaf node that matches the current log message similarity, a new log group is added, and the log template and the corresponding log ID list corresponding to the current log message are filled into the log group;

[0054] If there is a log group in the current leaf node that matches the current log message in similarity, determine whether the content corresponding to the similarity matching log group is consistent with that of the current log message. If they are consistent, do not update; if they are inconsistent, update the content of the similarity matching log group.

[0055] Furthermore, the step S101: obtaining the log message to be parsed includes: obtaining the log message to be parsed by crawling the network.

[0056] The log message to be parsed includes a fixed part (log template) and a variable part (log parameter).

[0057] Furthermore, the preprocessing of the log message to be parsed includes:

[0058] (1-1) Process the setting symbol in the log message, add spaces before and after the setting symbol, separate the setting symbol from the log content, and make it an independent field;

[0059] (1-2) Replace the delimiter in the log message with a space, and divide the log message into several fields based on the space;

[0060] (1-3) Replace dynamic fields in log messages with wildcards.

[0061] For example, (1-1) for some symbols in the log message (such as '=', '(', ')', '[', ']', '{', '}'), spaces are added before and after the symbols to separate these symbols from the log content and make them independent fields. For example, the symbol =10 will become "space=space10".

[0062] Exemplarily, (1-2) replaces the delimiters (such as ',', ';', '|') in the log with spaces, and divides the log message into multiple fields according to the space delimiters. This process ensures that each field becomes an independent part of the log message, which is convenient for subsequent parsing operations.

[0063] For example, (1-3) a domain-wide regular expression is designed to match dynamic parts in the log (such as IP addresses, timestamps, numbers, paths, etc.), and these dynamic fields are replaced with wildcards "<*>". For example, the matched IP address "10.250.19.16" will be replaced with "<*>", and the date "2024.10.01" will also be replaced with "<*>".

[0064] It should be understood that after the log message arrives, it is first pre-processed so as to be converted into a standard format that is easy to parse.

[0065] As attached Figure 1 As shown in the figure, the log parsing task aims to convert unstructured or semi-structured log messages into structured data. The present invention provides an online log parsing method AD-Tree based on an adaptive depth tree, which achieves efficient and accurate log parsing by dynamically adjusting the depth of the tree and a flexible field matching strategy. The structure of the parsing tree is shown in the figure. Figure 3 shown.

[0066] Furthermore, the number of fields of each log message is input into different log length matching nodes, and according to the matching result, the preprocessed log message is matched into the relevant subtree, including:

[0067] The number of fields M of the log message is analyzed and input into the log length matching node. If the log length of the log length matching node is also M, it means that the current log message successfully matches the current subtree and the current log message is allowed to match the corresponding subtree.

[0068] If the log length of the log length matching node is not equal to M, it means that the current log message fails to match the current subtree, and the current log message is not allowed to match the current subtree.

[0069] It should be understood that after the preprocessing is completed, the log message is parsed. First, a suitable subtree is selected in the root node of the tree according to the number of fields (ie, length) of the log message, and then a matching log group is searched in the subtree.

[0070] Exemplarily, at the first level of the parse tree, the nodes are divided according to the number of fields in the log message. The parse tree selects the corresponding subtree for matching according to the number of fields in the log message (i.e., the message length). For example, if the log message contains 6 fields, the parse tree will enter the subtree associated with "Length = 6". This can effectively reduce the number of nodes that need to be compared, thereby improving matching efficiency.

[0071] Furthermore, the method of determining the depth of the current subtree according to the length of the preprocessed log message and the preset fitness includes:

[0072] Calculate the length of the preprocessed log message and the score of the preset fitness, take an integer for the score, and get the depth of the current subtree. The depth of the subtree is also the total number of layers of nodes in the subtree.

[0073] Exemplarily, after selecting a suitable subtree, the depth of the subtree is calculated according to the length of the log message and the predefined fitness degree. For example, when degree = 0.4 and the log length is 8, the depth of the subtree will be round (8 * 0.4) = 3, that is, 3 layers of node matching will be performed in the subtree. The depth value is adaptively adjusted according to the length of the log to ensure that logs of different lengths are optimally matched.

[0074] Furthermore, the method of matching several fields of the preprocessed log message to the nodes of the current subtree in an alternating manner includes:

[0075] Divide the preprocessed log message into several fields, and number the fields of the log message in sequence from 1 to N;

[0076] First, take the field numbered 1 and match the field numbered 1 with the node of the current subtree;

[0077] If the content of the field numbered 1 is different from the content of the first-level node of the current subtree, it means that the match fails, and a new node is added to the first level of the current subtree, and the content of the new node is set to the field numbered 1;

[0078] If the content of the field numbered 1 is the same as the content of the first-level node of the current subtree, it means that the match is successful; if the match is successful, the field numbered N is taken in an alternating manner and matched with the second-level node of the current subtree;

[0079] If the content of the field numbered N is the same as the content of the second-level node of the current subtree, it means that the match is successful. In the alternating manner, the node numbered 2 is taken to match the third-level node of the current subtree;

[0080] If the content of the field numbered N is different from the content of the second-level node of the current subtree, it means that the match fails, and a new node is added to the second level of the current subtree, and the content of the new node is set to the field numbered N;

[0081] And so on, until all fields of the preprocessed log message match the nodes of the current subtree, or the depth and breadth of the current subtree reach the set threshold, the process ends.

[0082] It should be understood that the alternating between the first and the last means taking the 1st field first, then the Nth field, then the 2nd field, then the N-1th field, then the 3rd field, then the N-2th field, then the 4th field, then the N-3th field, and so on.

[0083] It should be understood that within the depth of the parse tree, matching is performed according to the order of the fields of the log message. Field matching is done in an alternating manner from head to tail to optimize the search path. For example, for the log message "PacketResponder<*>forblock blk_3328terminating", the parse tree will first match the field "PacketResponder" with the nodes in the tree, and then continue to match "terminating". During the matching process, if the field contains a number (such as blk_3328), the parse tree will use a special "<->" node for matching to avoid generating too many child nodes.

[0084] To avoid branch explosion, the maximum number of child nodes of each node is controlled during the parse tree search. When the number of child nodes of a node reaches the maximum value, any unmatched token will be directly matched to the special node "<->" under the node to ensure that the depth and breadth of the tree will not be over-expanded.

[0085] Furthermore, after traversing multiple layers of nodes, the leaf node of the current subtree is reached, each leaf node includes several log groups, each log group includes a log template and a corresponding log ID list,

[0086] Log template is an abstract form generated during log parsing. It contains the core information of system operation and is used to represent a class of log messages with similar structures. It retains the fixed / constant part of the log message and replaces the variable parameter part with wildcards.

[0087] Log list, which indicates the collection of log messages belonging to the current log template and records the IDs of these log messages;

[0088] The log template is extracted from the fixed part of the log message and is used to describe the characteristics of the group of logs; the log ID list records the unique identifier of the group of log messages. Once the leaf node matching the log message is found, it is necessary to further determine the matching degree between the log group under the node and the current log message.

[0089] Furthermore, the S103: finding the log group that best matches the preprocessed log message in the leaf nodes of the current subtree includes:

[0090] (2-1): For each log group, calculate the similarity between its template and the current log message. The similarity calculation is completed by comparing the matching degree of two fields. The specific formula is as follows:

[0091]

[0092] Among them, field i and field′ i The i-th fields of the log message and log template are respectively. The Match function is used to determine whether two fields match. If the two fields are the same, it means a match. If the two fields are different, it means a mismatch. n is the number of fields.

[0093] (2-2): Select matching log groups: When the similarity between a log message and a log group is greater than the preset threshold, the log message is considered to have successfully matched. If there are multiple log groups with the same similarity, the log group containing the most wildcards "<*>" will be selected as the final matching result.

[0094] If a log group matching the log message is found in the parsing tree, the log ID list of the log group is updated and the template is updated. If no matching log group is found, a new log group needs to be created and the parsing tree structure may need to be updated.

[0095] Furthermore, the S104: updating the log template and log ID list of the most matching log group includes:

[0096] If a matching log group is found, compare the fields in the log message and the log template one by one;

[0097] If the fields are consistent, keep the template fields unchanged;

[0098] If the fields do not match, the template fields are replaced with wildcard characters "<*>".

[0099] If no matching log group is found, a new log group is created; the template of the new log group is exactly the same as the current log message, and the ID list only contains the unique identifier of the current log message.

[0100] After creating a new log group, the structure of the parse tree is updated according to the characteristics of the logs.

[0101] At this time, the new log group will be inserted into the corresponding position of the tree, and a new node will be created or the existing node will be adjusted. Examples of parsing tree structure updates are shown in Figure 4(a) and Figure 4(b). Figure 5(a) to Figure 5(c) This is a comparison chart of the log parsing efficiency experiment results.

[0102] Table 1 Comparison of log parsing accuracy experimental results

[0103]

[0104]

[0105] Embodiment 2

[0106] This embodiment provides a heuristic online log parsing system based on an adaptive deep parsing tree, including:

[0107] An acquisition module is configured to: acquire a set of log messages to be parsed, and pre-process the set of log messages to be parsed;

[0108] A construction module is configured to: construct a parse tree, the parse tree comprising a main root node, the main root node is connected to a plurality of log length matching nodes; each log length matching node is a root node of a subtree;

[0109] The matching module is configured to: input the number of fields of each log message into different log length matching nodes, and match the preprocessed log message to the relevant subtree according to the matching result; determine the depth of the current subtree according to the length of the preprocessed log message and the preset fitness; match several fields of the preprocessed log message to the nodes of the current subtree in an alternating manner; after traversing multiple layers of nodes, reach the leaf node of the current subtree, each leaf node includes several log groups, and each log group includes a log template and a corresponding log ID list;

[0110] The output module is configured to: find the log group that best matches the preprocessed log message in the leaf node of the current subtree; update the log template and log ID list of the best matching log group, wherein the log template and log ID list of the best matching log group are the structured data of the log message to be parsed; when all log messages are processed, obtain the structured data of the set of log messages to be parsed.

[0111] It should be noted that the acquisition module, construction module, matching module and output module described above correspond to steps S101 to S104 in Embodiment 1, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the modules described above as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0112] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0114] Embodiment 3

[0115] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0116] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0117] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0118] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0119] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0120] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0121] Embodiment 4

[0122] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A heuristic online log parsing method based on adaptive deep parsing tree, characterized by: include: Obtain a set of log messages to be parsed, and pre-process the set of log messages to be parsed; Constructing a parse tree, the parse tree comprising a main root node, the main root node being connected to a plurality of log length matching nodes; each log length matching node being a root node of a subtree; The number of fields of each log message is input into different log length matching nodes, and the preprocessed log message is matched to the relevant subtree according to the matching result; Determine the depth of the current subtree according to the length of the preprocessed log message and the preset fitness; match several fields of the preprocessed log message to the nodes of the current subtree in a head-to-tail alternating manner; After traversing multiple layers of nodes, the leaf node of the current subtree is reached. Each leaf node includes several log groups. Each log group includes a log template and a corresponding log ID list. In the leaf node of the current subtree, find the log group that best matches the preprocessed log message; update the log template and log ID list of the best matching log group, which are the structured data of the log message to be parsed; when all log messages are processed, obtain the structured data of the log message set to be parsed.

2. The heuristic online log parsing method based on the adaptive deep parsing tree as claimed in claim 1 is characterized in that: After the number of fields of each log message is input into different log length matching nodes, the method further comprises: if the number of fields of the current log message is different from the log length of any log length matching node, a new log length matching node is added, and the log length of the new log length matching node is the number of fields of the current log message; After determining the depth of the current subtree according to the length of the preprocessed log message and the preset fitness, the method further comprises: if the actual depth of the current subtree is h and the determined depth of the current subtree is H, then adding (Hh) layers to the current subtree; The method adopts the head-to-tail alternating method to match the fields of the preprocessed log message to the nodes of the current subtree, and includes: if there is no matching node in the current subtree, adding a node to the current subtree, and setting the content of the added node to the corresponding field; The method of adding a new node to the current subtree includes two forms: Form 1: generating a brother node B next to the existing node A, and the brother node B shares a parent node with node A; Form 2: generating a child node a connected to node A below the existing node A, and the parent node of the child node a is node A; if the content of the new node and node A belongs to the same log message, form 2 is used; if the content of the new node and node A does not belong to the same log message, form 1 is used.

3. The heuristic online log parsing method based on the adaptive deep parsing tree as claimed in claim 1, characterized in that: After the multi-layer node traversal, after reaching the leaf node of the current subtree, the steps include: if there is no log group matching the similarity of the current log message in the current leaf node, a new log group is added, and the log template and the corresponding log ID list corresponding to the current log message are filled into the log group; if there is a log group matching the similarity of the current log message in the current leaf node, it is determined whether the content of the similarity matching log group is consistent with the content corresponding to the current log message, if they are consistent, then no update is performed; if they are inconsistent, the content of the similarity matching log group is updated; The preprocessing of the log message to be parsed includes: processing the setting symbol in the log message, adding spaces before and after the setting symbol, separating the setting symbol from the log content, and making them independent fields; replacing the separator in the log message with a space, and dividing the log message into a plurality of fields according to the space; replacing the dynamic field in the log message with a wildcard.

4. The heuristic online log parsing method based on adaptive deep parsing tree as claimed in claim 1, characterized in that: The number of fields of each log message is input into different log length matching nodes. According to the matching results, the preprocessed log message is matched to the relevant subtrees, including: Analyze the number of fields M of the log message and input the number of fields M of the log message into the log length matching node. If the log length of the log length matching node is also M, it means that the current log message successfully matches the current subtree and the current log message is allowed to match the corresponding subtree. If the log length of the log length matching node is not equal to M, it means that the current log message fails to match the current subtree, and the current log message is not allowed to match the current subtree; The method of determining the depth of the current subtree according to the length of the preprocessed log message and the preset fitness comprises: Calculate the length of the preprocessed log message and the score of the preset fitness, take an integer for the score, and get the depth of the current subtree. The depth of the subtree is also the total number of layers of nodes in the subtree.

5. The heuristic online log parsing method based on adaptive deep parsing tree as claimed in claim 1, characterized in that: The method of matching several fields of the preprocessed log message to the nodes of the current subtree in an alternating manner includes: Divide the preprocessed log message into several fields, and number the fields of the log message in sequence from 1 to N; First, take the field numbered 1 and match the field numbered 1 with the node of the current subtree; If the content of the field numbered 1 is different from the content of the first-level node of the current subtree, it means that the match fails, and a new node is added to the first level of the current subtree, and the content of the new node is set to the field numbered 1; If the content of the field numbered 1 is the same as the content of the first-level node of the current subtree, it means that the match is successful; if the match is successful, the field numbered N is taken in an alternating manner and matched with the second-level node of the current subtree; If the content of the field numbered N is the same as the content of the second-level node of the current subtree, it means that the match is successful. In the alternating manner, the node numbered 2 is taken to match the third-level node of the current subtree; If the content of the field numbered N is different from the content of the second-level node of the current subtree, it means that the match fails, and a new node is added to the second level of the current subtree, and the content of the new node is set to the field numbered N; And so on, until all fields of the preprocessed log message match the nodes of the current subtree, or the depth and breadth of the current subtree reach the set threshold, the process ends.

6. The heuristic online log parsing method based on the adaptive deep parsing tree as claimed in claim 1 is characterized in that In the leaf nodes of the current subtree, find the log group that best matches the preprocessed log message, including: For each log group, calculate the similarity between its template and the current log message; the similarity calculation is completed by comparing the matching degree of the two fields. The specific formula is as follows: Among them, field i and field′ i are the i-th fields of the log message and log template respectively. The Match function is used to determine whether two fields match. If the two fields are the same, it means a match. If the two fields are different, it means a mismatch. n is the number of fields. Select matching log groups: When the similarity between a log message and a log group is greater than the preset threshold, the log message is considered to have successfully matched; if there are multiple log groups with the same similarity, the log group containing the most wildcards "<*>" will be selected as the final matching result; If a log group matching the log message is found in the parsing tree, the log ID list of the log group is updated and the template is updated; if no matching log group is found, a new log group is created and the parsing tree structure is updated.

7. The heuristic online log parsing method based on adaptive deep parsing tree as claimed in claim 1 is characterized in that: Update the log template and log ID list that best matches the log group, including: If a matching log group is found, compare the fields in the log message and the log template one by one; If the fields are consistent, keep the template fields unchanged; If the fields are inconsistent, replace the template fields with wildcards "<*>"; If no matching log group is found, a new log group is created; the template of the new log group is exactly the same as the current log message, and the ID list only contains the unique identifier of the current log message; After creating a new log group, the structure of the parse tree is updated according to the characteristics of the logs.

8. A heuristic online log parsing system based on an adaptive deep parsing tree, characterized by: include: An acquisition module is configured to: acquire a set of log messages to be parsed, and pre-process the set of log messages to be parsed; A construction module is configured to: construct a parse tree, the parse tree comprising a main root node, the main root node is connected to a plurality of log length matching nodes; each log length matching node is a root node of a subtree; A matching module, which is configured to: input the number of fields of each log message into different log length matching nodes, and match the preprocessed log message to a related subtree according to the matching result; Determine the depth of the current subtree according to the length of the preprocessed log message and the preset fitness; match several fields of the preprocessed log message to the nodes of the current subtree in a head-to-tail alternating manner; After traversing multiple layers of nodes, the leaf node of the current subtree is reached. Each leaf node includes several log groups. Each log group includes a log template and a corresponding log ID list. The output module is configured to: find the log group that best matches the preprocessed log message in the leaf node of the current subtree; update the log template and log ID list of the best matching log group, wherein the log template and log ID list of the best matching log group are the structured data of the log message to be parsed; when all log messages are processed, obtain the structured data of the set of log messages to be parsed.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the method according to any one of claims 1 to 7 is performed.

Images