Log analysis method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a log analysis method and device, electronic equipment and storage medium. The method comprises: constructing a plurality of log analysis rules of a target log type into an analysis dictionary tree, each node of the analysis dictionary tree representing an analysis operator; obtaining a log to be analyzed of the target log type; matching the log to be analyzed with the analysis dictionary tree, and analyzing the log to be analyzed based on an analysis path successfully matched in the analysis dictionary tree to obtain an analysis result, the analysis path comprising a plurality of nodes. In the matching process, the same analysis operator does not need to be repeatedly matched, and compared with matching each log analysis rule one by one, the matching time can be reduced and the log analysis efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a log parsing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In existing technologies, regular expressions are generally used to parse logs. This involves matching each log entry against a regular expression to obtain the parsing result. Regular expressions are a method for non-deterministic pattern matching. Log pattern matching essentially corresponds to the state transitions within the parsing engine. Based on the mechanism of state symbol transitions, it can be divided into NFA (Non-deterministic Finite Automaton) and DFA (Deterministic Finite Automaton). Most regular expression-based parsing engines use pattern matching similar to PCRE (Perl Compatible Regular Expressions). PCRE is implemented as an NFA by default. At runtime, there are multiple matching paths from the initial state to an acceptable or matched state. The set of all these paths is the set of strings accepted by the NFA.

[0003] As the product business develops, the number of parsing rules increases, and the corresponding regular expressions also grow. Log parsing is performed by matching each regular expression one by one, resulting in low log parsing efficiency. Summary of the Invention

[0004] This application provides a log parsing method, apparatus, electronic device, and storage medium, which helps to improve log parsing efficiency.

[0005] To address the aforementioned problems, firstly, embodiments of this application provide a log parsing method, including:

[0006] Multiple log parsing rules for the target log type are constructed into a parsing trie, where each node of the parsing trie represents a parsing operator.

[0007] Obtain the logs to be parsed for the target log type;

[0008] The log to be parsed is matched with the parsing trie, and the log to be parsed is parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. The parsing path includes multiple nodes.

[0009] Secondly, embodiments of this application provide a log parsing apparatus, including:

[0010] The trie construction module is used to construct a parse trie from multiple log parsing rules of the target log type, where each node of the parse trie represents a parsing operator.

[0011] The log acquisition module is used to acquire the logs to be parsed of the target log type;

[0012] The log parsing module is used to match the log to be parsed with the parsing trie, and parse the log to be parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. The parsing path includes multiple nodes.

[0013] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the log parsing method described in embodiments of this application.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, discloses the log parsing method in embodiments of this application.

[0015] The log parsing method, apparatus, electronic device, and storage medium provided in this application construct a parsing trie for multiple log parsing rules of a target log type. Each node of the parsing trie represents a parsing operator. After obtaining the log to be parsed for the target log type, the log to be parsed is matched with the parsing trie, and the log to be parsed is parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. Since the parsing trie is a tree structure composed of multiple log parsing rules of the target log type, multiple log parsing rules have the same prefix. During the matching process, it is not necessary to repeatedly match the same parsing operator. Compared with matching each log parsing rule one by one, the matching time can be reduced and the log parsing efficiency can be improved. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a log parsing method provided in an embodiment of this application;

[0018] Figure 2 This is an example diagram of a directed acyclic graph in the embodiments of this application;

[0019] Figure 3 This is a flowchart of a log parsing method provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the structure of a log parsing device provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Figure 1 This is a flowchart of a log parsing method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes steps 110 to 130.

[0024] Step 110: Construct a parsing trie from multiple log parsing rules for the target log type, where each node of the parsing trie represents a parsing operator.

[0025] The target log type refers to the type of logs accessed by the log parsing system. This can be differentiated based on the log source, such as system logs, host logs, firewall logs, traffic logs, etc. The trie (prefix tree) refers to the trie corresponding to multiple log parsing rules. Also known as a Trie or prefix tree, it's a tree-like data structure used to store a set of strings and supports fast retrieval, insertion, and deletion operations. A key feature of a trie is that each node has one or more child nodes, each representing a character. Strings along the path from the root node to any node share the same prefix. In this embodiment, each node represents a parsing operator. A parsing operator is used to parse a portion of the log characters. Parsing operators can include constant parsing operators or operator operators. Constant parsing operators are used to parse constant strings, while operator operators are used to parse strings other than constant strings. Constant parsing operators can be implemented using the KMP (Knuth-Morris-Pratt) algorithm, an improved string matching algorithm proposed by DEKnuth, JH Morris, and VRPratt, and is therefore also known as the Knuth-Morris-Pratt operation. Log parsing rules are typically parsed grammars; given a rule, text position, and a specific keyword, its parsing result is usually unique.

[0026] In log parsing, parse trie trees can be used to quickly find and classify matching log messages (i.e., matching information accumulated in the KMP algorithm). Parsing trie trees can preprocess a series of keywords or patterns, such as "error," "warning," and "info," during the parsing rule pre-compilation process, constructing different trie trees according to different log types. This allows for rapid matching and filtering of specific types of log messages when traversing the logs to be parsed.

[0027] Different log types have different log parsing rules, and a separate parsing trie can be built for each log type. When constructing the parsing trie for a target log type, multiple log parsing rules corresponding to that target log type are obtained. For each log parsing rule, the rule configuration is defined, and the parsing operators in each log parsing rule are identified. The parsing operators are arranged according to their order of appearance in the log parsing rules, and the same parsing operator appearing in the same order in different log parsing rules is merged to obtain the nodes corresponding to each parsing operator. The nodes corresponding to each parsing operator are then connected based on their order of appearance in the log parsing rules, resulting in the parsing trie corresponding to the multiple log parsing rules for the target log type. Each node can be a root node, a child node, or a leaf node. Log parsing rules are usually written by parsing rule specialists based on on-site logs. They are expressions used to parse logs, such as <%pri:number%>%date:date-iso%%time:time-24hr%%host:word%%program:char-to::%:%kv:name-value-list:,).

[0028] A parse trie (DFA) is a finite state machine where each node represents a state, and the connections between nodes represent state transitions. For any structure, a finite state machine can be quickly constructed using a trie. As the log parsing process increases, the depth of the DFA recursively increases, forming the log parsing dictionary. Ultimately, we can obtain... Figure 2 The directed acyclic graph shown can be easily queried using a parse trie (directed acyclic graph) structure to obtain the approximate structure of the log to be parsed. Figure 2 The directed acyclic graph shown can quickly resolve the log levels corresponding to WARN or INFO. It should be noted that... Figure 2 This is merely an example of a directed acyclic graph and does not limit the parsed trie in this application embodiment. The nodes in the parsed trie in this application embodiment represent parsing operators and are not necessarily characters.

[0029] Based on the characters that make up the log parsing rules, the strings that make up the parsing operators are identified, and the log parsing rules are then segmented into nodes according to the strings of the parsing operators. A node is the basic unit for building the processing engine; it can represent a condition or an operation, etc. A node can contain one or more child nodes and can have a parent node. Each node can be a root node, a child node, or a leaf node. Finally, the nodes from multiple log parsing rules corresponding to the same target log type are constructed into a parsing trie. The parsing path from the root node to a leaf node in the parsing trie corresponds to a log parsing rule. The root node is the root of the node tree and typically represents the entire parsing rule system performing log parsing. The node structure is defined as follows:

[0030] typedef struct tPolicy{

[0031] / / Implement the adjustment of the weights (number of successful parses) of the Trie tree.

[0032] Policy;

[0033] typedef struct tContext{

[0034] / / Recursion result cache table

[0035] int[][]stats;

[0036] / / TopN update strategy

[0037] Policy update;

[0038] }Context;

[0039] Declare Trie Node structure

[0040] typedef struct tNode{

[0041] / / Engine context

[0042] Context*ctx;

[0043] / / Rule ID range 0-225

[0044] int id;

[0045] / / rule

[0046] char*rule;

[0047] / / rule prefix

[0048] char*prefix;

[0049] / / 1: Root node, 2: Child node, 3: Leaf node

[0050] int type;

[0051] Node*next; ...

[0053] }Node;

[0054] The process constructs the context information for the log parsing engine, obtains multiple log parsing rules for the target log type, and builds these rules into a parsing trie. Context information is the environment information during the execution of the log parsing program, typically containing information related to program execution. In the processing engine, context information can include configuration information for input and output data, shared data, and other configuration information. Context information can be represented by an object, which can be updated continuously during program execution. Corresponding log parsing rules are pre-compiled. Log parsing rules are the rules and conditions used to guide the log parsing program in processing input data (logs to be parsed). In the processing engine, log parsing rules can include regular expressions, DTD (Document Type Definition), XSD (XML Schema Definition), etc. Log parsing rules are the foundation for parsing and validating input data and the basis for building the parsing trie. Each log parsing rule is compiled into multiple parsing operators, each parsing operator corresponding to a node. Nodes are divided into three types: root node, child node, and leaf node.

[0055] When constructing the context information for the processing engine and compiling log parsing rules, it's essential to understand the specific application scenarios and requirements. Pre-compiling the corresponding log parsing rules and breaking them down into multiple nodes can improve the execution efficiency and processing accuracy of the log parsing program. Furthermore, for large processing engines, optimizing and eliminating useless rules before constructing the parsing trie can enhance program performance and response speed.

[0056] Step 120: Obtain the log to be parsed for the target log type.

[0057] Obtain the log to be parsed for the target log type, and allocate memory for the log to be parsed, so as to use the allocated memory to parse and process the log.

[0058] Step 130: Match the log to be parsed with the parsing trie, and parse the log to be parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. The parsing path includes multiple nodes.

[0059] Starting from the root node of the parsing trie, the log to be parsed is matched with each node of the parsing trie. The successfully matched nodes are used to parse the successfully matched strings in the log to be parsed, and the parsing result of the string is obtained. After the log to be parsed is matched, the parsing path in the parsing trie that successfully matches the log to be parsed is obtained. This parsing path consists of multiple nodes. Based on the parsing result of the successfully matched strings of each node, the parsing result of the entire log to be parsed is obtained.

[0060] The process of matching the log to be parsed with the trie is a recursive iterative process of parsing the nodes on the trie based on the log to be parsed. When the string of a node in the trie matches a part of the string of the log to be parsed (i.e., node.str.startWith(input.str)), the process recursively moves to the next node (node.next) of that node and continuously updates the status (stats) record of that node in the context information until a leaf node is parsed and the log to be parsed is completely matched (node.type == 3 && input.pos == input.len), it means that there is a parsing path that completely matches the log to be parsed, and the match is determined to be successful; otherwise, the match fails.

[0061] The log parsing method provided in this application constructs a parsing trie (DJ) of multiple log parsing rules for a target log type. Each node of the DJ represents a parsing operator. After obtaining the log to be parsed for the target log type, the log to be parsed is matched with the DJ, and the log to be parsed is parsed based on the parsing path that is successfully matched in the DJ to obtain the parsing result. Since the DJ is a tree structure composed of multiple log parsing rules for the target log type, the multiple log parsing rules have the same prefix. During the matching process, it is not necessary to repeatedly match the same parsing operator. Compared with matching each log parsing rule one by one, the matching time can be reduced and the log parsing efficiency can be improved.

[0062] Based on the above technical solution, the parsing operator includes constant parsing operators or operation operators, and the node corresponding to the operation operator is the operation node;

[0063] After parsing the log to be parsed based on the parsing path that is successfully matched in the parsing trie and obtaining the parsing result, the method further includes: updating the number of successful parsing operations corresponding to each operation node in the parsing path; and increasing the priority of the operation node when the number of successful parsing operations is greater than or equal to the target number.

[0064] The constant parsing operator can be implemented using the KMP algorithm. Operation operators are used to implement a parsing operation. For example, an operation operator represented as date:yyyy-MM-dd can be used to parse strings with a date format of yyyy-MM-dd. This operation operator can quickly parse strings like 2023-08-08.

[0065] The number of successful parses for each operation node in the parsing trie is recorded. After a log to be parsed is successfully matched, the number of successful parses for each operation node in the successfully matched parsing path is updated based on the original number of successful parses. That is, the number of successful parses for each operation node is incremented by 1. When the number of successful parses for an operation node is greater than or equal to the target number, the priority of the operation node is increased.

[0066] The priority of multiple operation nodes under the same parent node represents the matching priority order of operation nodes under that parent node. When matching with the log to be parsed, multiple operation nodes are matched with the log to be parsed according to the priority order of multiple operation nodes under the same parent node. That is, after the parent node successfully matches the log to be parsed, when matching the child nodes (i.e., multiple operation nodes) of that parent node, the operation node with the highest priority is matched first. If the operation node matches successfully and its child node matches successfully, there is no need to match other operation nodes under the same parent node. If the operation node matches successfully but its child node does not match successfully, or if the operation node fails to match, then the matching starts from the next priority operation node under the parent node, until the operation node matches successfully and its child node matches successfully, thus completing the matching of the child nodes of the parent node.

[0067] By updating the number of successful parses for each operation node in the successfully matched parsing path, and increasing the priority of the operation node when the number of successful parses is greater than or equal to the target number, it is easier to prioritize matching operation nodes with higher priority. Since operation nodes with higher priority are frequently used operation nodes, parsing efficiency can be further improved.

[0068] The main difference between this application's embodiment based on a Trie tree and a variant lies in using characters with specific algorithmic functions as type regular expressions. After the log parsing rules are constructed, they are compiled to generate operator functions. These operator functions can accelerate the matching efficiency of ordinary characters and can handle some frequently changing log positions, such as log time processing. Using the operator date:yyyy-MM-dd, strings like 2023-08-08 can be quickly parsed. For constant strings, the basic KMP algorithm is still used to quickly search whether they conform to the current rule pattern, thereby accelerating the parsing of the entire log.

[0069] Based on the above technical solution, the step of matching the log to be parsed with the parsing trie, and parsing the log to be parsed based on the successfully matched parsing paths in the parsing trie to obtain the parsing result, includes:

[0070] The matching process begins from the root node of the parsed trie and is performed against the log to be parsed.

[0071] When a node in the parsed trie successfully matches a partial string in the log to be parsed, the parsing result of the node for the partial string is obtained, and the child nodes are matched with the characters following the partial string in the log to be parsed in descending order of priority of the node's multiple child nodes, until the child nodes successfully match the characters following the partial string.

[0072] When the log to be parsed successfully matches the parsing trie, the parsing result of the log to be parsed is determined based on the parsing result of each node in the successfully matched parsing path.

[0073] When matching the log to be parsed begins, the log is first matched against the root node of the parsing trie. Since the root node represents the parsing rule system for log parsing, the log to be parsed matches the root node successfully. Next, the log to be parsed is matched against the child nodes of the root node. If the root node has multiple child nodes, they are matched against the log to be parsed in descending order of priority. That is, the child node with the highest priority is matched first. If the match is successful, no further matching is needed. If the match fails, the next highest priority child node is matched, and so on, until a child node that matches the log is found. The process continues, matching the log to be parsed against the child nodes of the successfully matched nodes, until the entire log to be parsed matches the parsing trie. The parsing results of each node in the successfully matched parsing path in the parsing trie are then obtained. The parsing results of each node are concatenated according to the order of the nodes in the parsing path to obtain the parsed result of the log to be parsed. By matching the child nodes of the same node with the log to be parsed in descending order of priority, higher priority child nodes can be matched first. Since higher priority child nodes are frequently used, they can be matched more quickly, thus improving the efficiency of log parsing.

[0074] Based on the above technical solution, before matching the log to be parsed starting from the root node of the parsing trie, the method further includes: allocating memory for the log to be parsed and storing the log to be parsed in the allocated memory; initializing the structure corresponding to the log to be parsed, the structure including the log to be parsed, the current parsing position, and the string length of the log to be parsed.

[0075] First, declare an NPB (Next Processor Block) structure as follows:

[0076] typedef struct tNPB{

[0077] / / Engine context

[0078] Context*ctx;

[0079] / / Raw string

[0080] const char*str;

[0081] / / Current parsing position

[0082] size_t pos;

[0083] / / String length

[0084] size_t len;

[0085] NPB;

[0086] In the main function of log parsing, a string pointer `input_` is declared to store the input string (the log to be parsed). The `malloc` function can be used to allocate memory for this string and copy the input string into that memory.

[0087] After obtaining the log to be parsed, the `malloc` function is used to allocate memory for the log, and the log is copied and stored in the allocated memory. The structure object corresponding to the log is initialized, the original string in the structure object is initialized to the log to be parsed, the current parsing position in the structure object is initialized to 0, and the string length in the structure object is initialized to the string length of the log to be parsed. In other words, the initialization operation of the log to be parsed is achieved by running the following code:

[0088] / / Input string

[0089] char* str = "This is a sample log to be parsed.";

[0090] / / Allocate memory and copy the input string into that memory.

[0091] char*str_copy=(char*)malloc(strlen(str)+1);

[0092] strncpy(str_copy,str,strlen(str)+1);

[0093] / / Initialize NPB structure

[0094] NPB input;

[0095] input.str = str_copy;

[0096] input.pos = 0;

[0097] input.len=strlen(input.str);

[0098] / / Perform other operations...

[0099] / / Release allocated memory

[0100] free(str_copy);

[0101] By allocating and initializing memory for the log to be parsed, the log can be matched and parsed in the allocated memory, and the structure object can be updated based on the matching process. This facilitates the recording of the matching process and ensures the accuracy of the parsing results.

[0102] Based on the above technical solution, the method further includes: when a node of the parsed trie successfully matches a portion of the string in the log to be parsed, updating the current parsing position in the structure object; and when a leaf node of the parsed trie is matched and the current parsing position is equal to the length of the string, determining that the log to be parsed has been successfully matched.

[0103] When a node in the parsed trie successfully matches a portion of the log string to be parsed, the current parsing position in the structure object is updated to the position of the last character of that portion of the string. After matching, if a leaf node of the parsed trie is matched and the current parsing position in the structure object is equal to the string length, the log string to be parsed is considered to have matched successfully. By updating the current parsing position recorded in the structure object during the matching process, and determining a successful match when the current parsing position is equal to the string length, the current parsing position in the log string to be parsed can be accurately recorded, and the success of parsing can be accurately determined. This can further improve the efficiency and accuracy of log parsing.

[0104] In one embodiment of this application, the method further includes: updating a preset number of parsing paths stored in the cache according to the successfully matched parsing paths, wherein the parsing paths in the cache are used to preferentially match the logs to be parsed.

[0105] The cached parsing paths can be updated based on the parsing time or the number of successful parsing attempts of the successfully matched parsing paths. In one exemplary embodiment, the parsing time of a successfully matched parsing path is obtained and compared with the parsing time of the preset number of parsing paths stored in the cache. If the parsing time of a successfully matched parsing path is less than the maximum parsing time among the preset number of parsing paths, the successfully matched parsing path can be stored in the cache, and the parsing path with the maximum parsing time can be deleted from the cache, ensuring that the cache stores a preset number (Top N) of parsing paths with the smallest parsing time. In another exemplary embodiment, the number of successful parsing attempts of a successfully matched parsing path is updated and compared with the number of successful parsing attempts of the preset number of parsing paths stored in the cache. If the number of successful parsing attempts of a successfully matched parsing path is greater than the minimum number of successful parsing attempts among the preset number of parsing paths, the successfully matched parsing path can be stored in the cache, and the parsing path with the minimum number of successful parsing attempts can be deleted from the cache, ensuring that the cache stores a preset number (Top N) of parsing paths with the largest number of successful parsing attempts.

[0106] The Round-Robin algorithm is used iteratively for a preset number of parsing paths in the cache to ensure that the parser performing log parsing is deeply optimized, and the performance of the operator will be significantly improved.

[0107] By updating the preset number of parsing paths in the cache, the optimal parsing path is ensured to be cached. When parsing subsequent logs, the logs are preferentially matched with the cached parsing paths, thereby further improving the log parsing efficiency.

[0108] Based on the above technical solution, before matching the log to be parsed with the parsing trie, the method further includes: matching the log to be parsed with each parsing path in the cache, and parsing the log to be parsed based on the successfully matched parsing paths to obtain the parsing result of the log to be parsed; if the log to be parsed fails to match each parsing path in the cache, the operation of matching the log to be parsed with the parsing trie is triggered.

[0109] Before matching the log to be parsed against the parsing trie, the cached parsing paths are used to parse the log. If a cached parsing path matches the log, that path is used directly to parse the log and obtain the result. If neither a cached path nor the log matches, the log is further matched against the parsing trie to obtain the parsing result. Since the cache stores a preset number of parsing paths that are either time-efficient or frequently used, it can parse a larger number of logs, thus further improving log parsing efficiency.

[0110] Figure 3 This is a flowchart of a log parsing method provided in an embodiment of this application, such as... Figure 3 As shown, the method includes:

[0111] Step 310: Construct a parsing trie from multiple log parsing rules for the target log type, where each node of the parsing trie represents a parsing operator.

[0112] Step 320: Obtain multiple logs of the target log type to be parsed.

[0113] Step 330: Identify similar keywords in the multiple logs to be parsed based on the matrix factorization model, and replace the similar keywords in the multiple logs to be parsed with the same similar keyword.

[0114] Based on the multiple logs to be parsed and the keywords in each log, a log keyword matrix corresponding to the multiple logs to be parsed is constructed. The log keyword matrix is ​​then decomposed using a matrix factorization model to obtain a log matrix and a keyword matrix. The inner product of the vectors corresponding to every two logs to be parsed in the log matrix is ​​determined to obtain the semantic similarity between every two logs to be parsed. Two logs to be parsed whose semantic similarity is greater than a similarity threshold are selected. These two logs are matched to obtain similar keywords, and one of the similar keywords is used to replace a similar keyword in the other log to obtain the processed log to be parsed. The processed log to be parsed can be parsed using a trie. The keywords in the logs to be parsed can be words composed of parts of the strings in the logs to be parsed.

[0115] Assuming there are m logs to be parsed and n keywords, we can construct an m x n log-keyword matrix R. The values ​​in R represent specific score values, such as whether a keyword appears in a log (1 indicates appearance, 0 indicates absence). The matrix factorization model decomposes matrix R into two low-rank log matrices U and V, and then approximates R using their product, as shown in the following formula:

[0116] R≈U T V

[0117] Where, U∈R k×m , V∈R k×n And k << min(m,n). Column vector U i V represents the log latent vector, i.e., the vector corresponding to the log to be parsed. j This represents the latent vector of the keywords. To learn the latent vector, we can use the idea of ​​linear regression to minimize the original score R and the predicted score U. T The squared error between V values ​​yields the objective function as follows:

[0118]

[0119] Where K represents (i,j) pairs with known scores, a regularization term is added to prevent overfitting, and λ represents the regularization factor. Represents the matrix norm.

[0120] The log matrix U and keyword matrix V corresponding to the minimum objective function are used as the final decomposition matrix. Various optimization methods can be used to find the local minimum of the objective function, such as gradient descent and ALS (Alternating Least Squares). Compared to gradient descent, ALS is more advantageous for distributed scaling and improves computational efficiency. However, as an offline method, ALS cannot accurately perform real-time recommendations and also suffers from the cold start problem. Therefore, gradient descent and ALS each have their advantages and disadvantages, and the appropriate algorithm can be selected based on the requirements.

[0121] Step 340: For each log to be parsed, match the log to be parsed with the parsing trie, and parse the log to be parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. The parsing path includes multiple nodes.

[0122] Each log file to be parsed is parsed using a parse trie to obtain the parsing result for each log file.

[0123] The log parsing method provided in this embodiment identifies similar keywords in multiple logs to be parsed based on a matrix factorization model before parsing the logs, and replaces the similar keywords in multiple logs with the same similar keyword among the multiple similar keywords, thereby avoiding the problem of inaccurate parsing caused by possible semantic ambiguity.

[0124] Figure 4 This is a schematic diagram of the structure of a log parsing device provided in an embodiment of this application, as shown below. Figure 4 As shown, the device includes:

[0125] The trie construction module 410 is used to construct a parse trie from multiple log parsing rules of the target log type, wherein each node of the parse trie represents a parsing operator.

[0126] Log acquisition module 420 is used to acquire the log to be parsed of the target log type;

[0127] The log parsing module 430 is used to match the log to be parsed with the parsing trie, and parse the log to be parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. The parsing path includes multiple nodes.

[0128] Optionally, the parsing operator includes a constant parsing operator or an operation operator, and the node corresponding to the operation operator is an operation node;

[0129] The device further includes:

[0130] The count update module is used to update the number of successful parsings corresponding to each operation node in the parsing path;

[0131] The priority enhancement module is used to increase the priority of the operation node when the number of successful parsings is greater than or equal to the target number.

[0132] Optionally, the log parsing module is specifically used for:

[0133] The matching process begins from the root node of the parsed trie and is performed against the log to be parsed.

[0134] When a node in the parsed trie successfully matches a partial string in the log to be parsed, the parsing result of the node for the partial string is obtained, and the child nodes are matched with the characters following the partial string in the log to be parsed in descending order of priority of the node's multiple child nodes, until the child nodes successfully match the characters following the partial string.

[0135] When the log to be parsed successfully matches the parsing trie, the parsing result of the log to be parsed is determined based on the parsing result of each node in the successfully matched parsing path.

[0136] Optionally, the device further includes:

[0137] A memory allocation module is used to allocate memory for the log to be parsed and store the log to be parsed in the allocated memory;

[0138] An initialization module is used to initialize the structure object corresponding to the log to be parsed. The structure object includes the log to be parsed, the current parsing position, and the string length of the log to be parsed.

[0139] Optionally, the log parsing module further includes:

[0140] The parsing position update unit is used to update the current parsing position in the structure object when a node of the parsing trie successfully matches a partial string of the log to be parsed;

[0141] The matching success determination unit is used to determine that the log to be parsed is successfully matched when a leaf node of the parsing trie is matched and the current parsing position is equal to the length of the string.

[0142] Optionally, the device further includes:

[0143] The cache update module is used to update a preset number of parsing paths stored in the cache based on the successfully matched parsing paths. The parsing paths in the cache are used to preferentially match the logs to be parsed.

[0144] Optionally, the log acquisition module is specifically used for:

[0145] Obtain multiple logs of the target log type to be parsed;

[0146] The device further includes:

[0147] The similar keyword processing module is used to identify similar keywords in the multiple logs to be parsed based on a matrix factorization model before matching the logs to be parsed with the parsing trie, and to replace the similar keywords in the multiple logs to be parsed with the same similar keyword.

[0148] The log parsing apparatus provided in this application embodiment is used to implement the steps of the log parsing method described in this application embodiment. The specific implementation of each module of the apparatus is described in the corresponding steps, and will not be repeated here.

[0149] The log parsing apparatus provided in this application constructs a parsing trie of multiple log parsing rules for a target log type. Each node of the parsing trie represents a parsing operator. After obtaining the log to be parsed for the target log type, the log to be parsed is matched with the parsing trie, and the log to be parsed is parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. Since the parsing trie is a tree structure composed of multiple log parsing rules for the target log type, the multiple log parsing rules have the same prefix. During the matching process, it is not necessary to repeatedly match the same parsing operator. Compared with matching each log parsing rule one by one, the matching time can be reduced and the log parsing efficiency can be improved.

[0150] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device 500 may include one or more processors 510 and one or more memories 520 connected to the processors 510. The electronic device 500 may also include an input interface 530 and an output interface 540 for communicating with another device or system. Program code executed by the processor 510 may be stored in the memory 520.

[0151] The processor 510 in the electronic device 500 calls the program code stored in the memory 520 to execute the log parsing method in the above embodiment.

[0152] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the log parsing method as described in this application.

[0153] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus embodiments, since they are fundamentally similar to the method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0154] The above provides a detailed description of a log parsing method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

Claims

1. A log parsing method, characterized in that, include: Multiple log parsing rules for the target log type are constructed into a parsing trie, where each node of the parsing trie represents a parsing operator. Obtain the logs to be parsed for the target log type; The log to be parsed is matched with the parsing trie, and the log to be parsed is parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. The parsing path includes multiple nodes. Obtain multiple log parsing rules corresponding to the target log type, configure the rules for each log parsing rule, identify the parsing operators in each log parsing rule, arrange the parsing operators according to their order of appearance, merge the same parsing operator with the same order of appearance in different log parsing rules, obtain the nodes corresponding to each parsing operator, and connect the nodes corresponding to each parsing operator based on their order of appearance in the log parsing rules to obtain the parsing trie; The parsing operator is used to parse a portion of the log characters. The parsing operators include constant parsing operators or operational operators, and the nodes corresponding to the operational operators are operational nodes; After parsing the log to be parsed based on the parsing path that is successfully matched in the parsing trie and obtaining the parsing result, the process further includes: Update the number of successful parsing operations for each operation node in the parsing path; When the number of successful parsing attempts is greater than or equal to the target number, the priority of the operation node is increased.

2. The method according to claim 1, characterized in that, The step of matching the log to be parsed with the parsing trie and parsing the log based on the successfully matched parsing paths in the parsing trie to obtain the parsing result includes: The matching process begins from the root node of the parsed trie and is performed against the log to be parsed. When a node in the parsed trie successfully matches a partial string in the log to be parsed, the parsing result of the node for the partial string is obtained, and the child nodes are matched with the characters following the partial string in the log to be parsed in descending order of priority of the node's multiple child nodes, until the child nodes successfully match the characters following the partial string. When the log to be parsed successfully matches the parsing trie, the parsing result of the log to be parsed is determined based on the parsing result of each node in the successfully matched parsing path.

3. The method according to claim 2, characterized in that, Before matching the log to be parsed starting from the root node of the parsed trie, the method further includes: Allocate memory for the log to be parsed, and store the log in the allocated memory; Initialize the structure object corresponding to the log to be parsed. The structure object includes the log to be parsed, the current parsing position, and the string length of the log to be parsed.

4. The method according to claim 3, characterized in that, Also includes: When a node in the parsed trie successfully matches a portion of the string in the log to be parsed, update the current parsing position in the structure object; When a leaf node of the parsing trie is matched and the current parsing position is equal to the length of the string, the log to be parsed is determined to be successfully matched.

5. The method according to claim 1, characterized in that, Also includes: The cached parsing paths are updated based on the successfully matched parsing paths. The parsing paths in the cache are used to preferentially match the logs to be parsed.

6. The method according to claim 1, characterized in that, The step of obtaining the log to be parsed of the target log type includes: Obtain multiple logs of the target log type to be parsed; Before matching the log to be parsed with the parsed trie, the method further includes: Based on the matrix factorization model, similar keywords in the multiple logs to be parsed are identified, and the similar keywords in the multiple logs to be parsed are replaced with the same similar keyword.

7. A log parsing device, characterized in that, include: The trie construction module is used to construct a parse trie from multiple log parsing rules of the target log type, where each node of the parse trie represents a parsing operator. The log acquisition module is used to acquire the logs to be parsed of the target log type; The log parsing module is used to match the log to be parsed with the parsing trie, and parse the log to be parsed based on the parsing path that is successfully matched in the parsing trie to obtain the parsing result. The parsing path includes multiple nodes. Obtain multiple log parsing rules corresponding to the target log type, configure the rules for each log parsing rule, identify the parsing operators in each log parsing rule, arrange the parsing operators according to their order of appearance, merge the same parsing operator with the same order of appearance in different log parsing rules, obtain the nodes corresponding to each parsing operator, and connect the nodes corresponding to each parsing operator based on their order of appearance in the log parsing rules to obtain the parsing trie; The parsing operator is used to parse a portion of the log characters. The parsing operators include constant parsing operators or operational operators, and the nodes corresponding to the operational operators are operational nodes; The log parsing module is also used to update the number of successful parsing operations corresponding to each operation node in the parsing path; When the number of successful parsing attempts is greater than or equal to the target number, the priority of the operation node is increased.

8. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the log parsing method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the log parsing method according to any one of claims 1 to 6.

Images