Log analysis method and device

Abstract

The embodiment of the invention provides a log analysis method and device, and the method comprises the steps: determining a to-be-analyzed log, and carrying out the word segmentation of the to-be-analyzed log, and obtaining a log word segmentation sequence; the log word segmentation sequence is matched with nodes in a prefix tree, candidate grammar templates corresponding to the log word segmentation sequence are determined, the prefix tree is constructed through the nodes and paths, the nodes are segmented words in the log grammar templates, the paths are association relationships of the segmented words in the log grammar templates, and the candidate grammar templates correspond to the log word segmentation sequence. The log grammar template is determined according to an initial log; according to a preset matching strategy, matching the log word segmentation sequence with the candidate grammar template to obtain a matching result; and under the condition that the matching result does not meet a preset matching condition, analyzing the to-be-analyzed log by utilizing a large language model to obtain an analysis result. Through the method, the accuracy of log analysis can be improved, and the cost of log analysis can be reduced.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of log analysis, and in particular, to a log parsing method. One or more embodiments of this specification also relate to a log parsing device, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Generally, the amount of log data is relatively large. For example, for large-scale cloud services, it is common to generate trillions of lines of log data every day. Log data plays an indispensable role in system diagnosis, security analysis, and performance tuning. Therefore, it is necessary to perform data analysis on log data. However, in the face of the analysis of log data with a relatively large amount of data, manual analysis is very inefficient. Therefore, it is necessary to perform automated analysis on log data. In order to better achieve automated log analysis, log parsing technology is required to convert the original log data into a structured format for subsequent downstream task analysis.

[0003] However, the complexity of contemporary systems and the diversity and variability of logs pose significant challenges to existing automatic log parsing technologies. Therefore, there is an urgent need for a log parsing solution with lower cost and higher accuracy to solve the above problems. Summary of the Invention

[0004] In view of this, the embodiments of this specification provide a log parsing method. One or more embodiments of this specification also relate to a log parsing device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] According to the first aspect of the embodiments of this specification, a log parsing method is provided, including:

[0006] Determine the log to be parsed, and perform word segmentation processing on the log to be parsed to obtain a log word segmentation sequence;

[0007] Match the log word segmentation sequence with the nodes in the prefix tree to determine the candidate grammar template corresponding to the log word segmentation sequence, where the prefix tree is constructed by nodes and paths, the nodes are the word segments in the log grammar template, the paths are the association relationships of the word segments in the log grammar template, and the log grammar template is determined according to the initial log;

[0008] According to a preset matching strategy, match the log word segmentation sequence with the candidate grammar template to obtain a matching result;

[0009] In the case where the matching result does not meet the preset matching condition, use a large language model to parse the log to be parsed to obtain a parsing result.

[0010] According to the second aspect of the embodiments of this specification, a log parsing device is provided, including:

[0011] A log word segmentation sequence obtaining module, configured to determine a log to be parsed, perform word segmentation processing on the log to be parsed, and obtain a log word segmentation sequence;

[0012] A candidate grammar template determining module, configured to match the log word segmentation sequence with the nodes in the prefix tree to determine the candidate grammar template corresponding to the log word segmentation sequence, where the prefix tree is constructed by nodes and paths, the nodes are the word segments in the log grammar template, the paths are the association relationships of the word segments in the log grammar template, and the log grammar template is determined according to the initial log;

[0013] A grammar template matching module, configured to match the log word segmentation sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result;

[0014] A log parsing module, configured to, when the matching result does not meet the preset matching condition, use a large language model to parse the log to be parsed to obtain a parsing result.

[0015] According to the third aspect of the embodiments of this specification, a computing device is provided, including:

[0016] A memory and a processor;

[0017] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above log parsing method are implemented.

[0018] According to the fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by the processor, the steps of the above log parsing method are implemented.

[0019] According to the fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions, and when the computer program / instructions are executed by the processor, the steps of the above log parsing method are implemented.

[0020] An embodiment of this specification provides a log parsing method, including: determining the log to be parsed, and performing word segmentation processing on the log to be parsed to obtain a log word segmentation sequence; matching the log word segmentation sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log word segmentation sequence, where the prefix tree is constructed by nodes and paths, the nodes are word segments in a log grammar template, the paths are the association relationships of the word segments in the log grammar template, and the log grammar template is determined according to initial logs; according to a preset matching strategy, matching the log word segmentation sequence with the candidate grammar template to obtain a matching result; in the case that the matching result does not meet the preset matching condition, using a large language model to parse the log to be parsed to obtain a parsing result.

[0021] Specifically, this method first performs word segmentation on the log to be parsed to obtain a log word segmentation sequence, and improves the log parsing efficiency through the matching between word segments. Then, it matches the log to be parsed with a prefix tree containing information about log grammar templates to find a candidate grammar template close to the log to be parsed, so as to determine all possible log grammar templates that can be used as the log grammar template corresponding to the log to be parsed. According to a preset matching strategy, it matches the log word segmentation sequence with the candidate grammar template to determine whether there is a candidate grammar template that matches the current log to be parsed and meets the preset matching condition. In the case that there is no candidate grammar template that meets the preset matching condition, it uses a large model to parse the log to be parsed, understands the semantics of the log to be parsed, and realizes more accurate log parsing. Moreover, since the large language model is only used when no candidate grammar template that meets the matching condition is found, the cost of using the large language model is reduced, so that the log parsing cost of applying this solution is lower. Description of the Drawings

[0022] Figure 1 is a specific application scenario diagram of a log parsing method provided by an embodiment of this specification;

[0023] Figure 2 is a flowchart of a log parsing method provided by an embodiment of this specification;

[0024] Figure 3 is a schematic diagram of a data structure example of a log parsing method provided by an embodiment of this specification;

[0025] Figure 4 is a schematic structural diagram of a log parsing device provided by an embodiment of this specification;

[0026] Figure 5 is a structural block diagram of a computing device provided by an embodiment of this specification. Detailed Embodiments

[0027] In the following description, numerous specific details are set forth in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the connotation of this specification. Therefore, this specification is not limited by the specific implementations disclosed below.

[0028] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a", "the", and "said" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0029] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining".

[0030] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for the user to choose to authorize or reject.

[0031] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, usually including hundreds of millions, tens of billions, hundreds of billions, trillions or even more than one quadrillion model parameters. A large model can also be referred to as a Foundation Model. Through the pre-training of a large model with a large amount of unlabeled corpus, a pre-trained model with more than one hundred million parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLMs), multi-modal pre-training models, etc.

[0032] When the large model is actually applied, it only needs a small number of samples to fine-tune the pre-trained model and then can be applied to different tasks. The large model can be widely applied in fields such as Natural Language Processing (NLP) and computer vision. Specifically, it can be applied to tasks in the field of computer vision such as Visual Question Answering (VQA), Image Caption (IC), and image generation, as well as tasks in the field of natural language processing such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the large model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0033] First, explain the noun terms involved in one or more embodiments of this specification.

[0034] Log parsing: A technology that extracts log templates and variables from raw logs to convert log data into structured data. A log template can be understood as a static keyword that contains the events corresponding to the log data, such as "receive", "send", "node", etc., which can usually be represented in natural language. A variable can be understood as a dynamic variable in the log data, such as data representing an IP address, time, etc.

[0035] Prefix parsing tree: A tree structure with a common prefix form used for log parsing, including a root node, non-leaf nodes, and leaf nodes. Each node represents a word or a placeholder. The path from the root node to any node, concatenated in the order before and after the nodes, can obtain a log template, that is, a log syntax template. For the convenience of explanation, the prefix parsing trees in the following embodiments of the specification are all abbreviated as prefix trees, and will not be elaborated hereinafter.

[0036] To ensure the stable operation of the cloud system and improve the observability of the cloud system, a large amount of operation process data will be generated during the system operation for analysis, such as log data, trace data, metric data, etc. Among them, log data is very flexible and is the most commonly used data for analysis. By analyzing the log data, various anomalies and faults encountered during the operation of the cloud system can be determined; however, the amount of log data is relatively huge. For large-scale cloud services, it is very common to generate trillions of lines of log data every day. Therefore, the efficiency of manual analysis is very low, and there is an urgent need for automated analysis of log data.

[0037] To facilitate log output, there are various log formats. Therefore, in the process of automated log analysis, it is important to automatically identify the variable part (i.e., variables) and the template structure of the log (i.e., log template) in the log; when the variables in each log line can be automatically identified and structured into high-dimensional metric data (i.e., structured data), there will be many systematic methods for high-dimensional metric data analysis that can be used, which can be used for anomaly detection, root cause analysis, etc. This helps the relevant personnel of the cloud system quickly identify and detect system problems, nip the system problems in the bud, and locate and solve the problems in a timely manner.

[0038] The description method of log data is very similar to natural language. Traditional log automatic parsing methods only consider the statistical changes of words in log data and cannot explain the semantic information existing in natural language. Therefore, the accuracy of traditional log automatic parsing methods is relatively low. With the development of large language models, relevant personnel have begun to use large language models for log parsing. However, due to the large amount of computing resources required by large language models, the cost of using large language models for log parsing is very high.

[0039] In view of this, we have innovatively developed a method that seamlessly integrates the semantic analysis from large language models and traditional log automatic parsing methods (i.e., statistical machine learning algorithms for log parsing), thereby leveraging the advantages of both technologies and allowing the statistical machine learning algorithms to enable large language models to extract log templates in specific critical situations, which not only improves accuracy but also saves the cost of using large language models.

[0040] Specifically, in this specification, a log parsing method is provided. This specification also relates to a log parsing device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail one by one in the following embodiments.

[0041] In practical applications, the log parsing method provided in the embodiments of this specification can be applied to any application scenario that requires log parsing, including but not limited to mobile system log parsing, software running log parsing, computer log parsing, cloud system log parsing, etc. For the sake of easy understanding, the following will take the log parsing method provided in the embodiments of this specification applied to cloud system log parsing as an example for exemplary explanation.

[0042] The following combines the attached Figure 1 , taking the application of the log parsing method provided in this specification to cloud system log parsing as an example, to further illustrate the log parsing method. See Figure 1 , Figure 1 is a specific application scenario diagram of a log parsing method provided according to an embodiment of this specification.

[0043] As Figure 1 described Figure 1 it includes a client 102 and a cloud server 104. A cloud system interface for providing cloud system usage services to users is displayed on the client 102. The cloud server 104 is used to complete the task of parsing the logs generated during the operation of the cloud system. Among them, the client 102 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, etc. The cloud server 104 can be understood as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms. In other achievable application scenarios (such as mobile phone system log parsing, local computer log parsing, etc.), the cloud server can be replaced by other servers, such as physical servers, a server cluster composed of multiple physical servers, a distributed system, etc.

[0044] Specifically, during implementation, the user operates the cloud system through the cloud system interface displayed on the client 102. During the operation of the cloud system, logs to be parsed are generated. The client 102 sends the logs to be parsed to the cloud server 104 of the cloud system. After receiving the logs to be parsed, the cloud server 104 first performs word segmentation on the logs to be parsed to obtain a sequence of log word segments after word segmentation. Then, the cloud server 104 matches the sequence of log word segments with a prefix tree representing the log grammar template information obtained by parsing the initial logs. Specifically, the nodes of this prefix tree represent the word segments in the log grammar template, and the paths represent the association relationships between the word segments. After matching, a candidate grammar template corresponding to the sequence of log word segments is obtained. After obtaining the candidate grammar template, the sequence of log word segments is matched with the candidate grammar template through a preset matching strategy. Then, in the case where the matching result does not meet the preset matching conditions, that is, when none of the candidate grammar templates match the sequence of log word segments, in this case, a large language model is used to perform semantic analysis on the logs to be parsed, so as to achieve the parsing of the logs to be parsed and obtain the log parsing result.

[0045] The log parsing method provided in the embodiments of this specification enables the cloud server to segment the log to be parsed generated during the user's use of the cloud system on the client side, obtaining a log segmentation sequence, and improving the log parsing efficiency through the matching between segments. Subsequently, the log to be parsed is matched with a prefix tree containing information of log grammar templates to find candidate grammar templates close to the log to be parsed, so as to determine all possible log grammar templates that can correspond to the log to be parsed. According to a preset matching strategy, the log segmentation sequence is matched with the candidate grammar templates to determine whether there is a candidate grammar template that matches the current log to be parsed and meets the preset matching conditions. In the case where there is no candidate grammar template that meets the preset matching conditions, a large model is used to parse the log to be parsed, obtaining a log parsing result, thereby understanding the semantics of the log to be parsed, achieving more accurate log parsing. Moreover, since the large language model is only used when no candidate grammar template that meets the matching conditions is found, the cost of using the large language model is reduced, and thus the cloud server makes the log parsing cost of the cloud system lower.

[0046] See Figure 2 , Figure 2 is a flowchart of a log parsing method provided according to an embodiment of this specification, which specifically includes the following steps.

[0047] Step 202: Determine the log to be parsed and perform a segmentation process on the log to be parsed to obtain a log segmentation sequence.

[0048] Among them, the log to be parsed can be understood as the log to be subjected to log parsing, the segmentation process can be understood as the processing step of separating each word and variable included in the log to be parsed, and the log segmentation sequence can be understood as the segmentation sequence obtained by separating each word, variable and other elements in the log to be parsed, which is in the same order as each word and variable in the log to be parsed.

[0049] In practical applications, since the logs generated during system operation contain invalid information or interfering information, which are meaningless or interfering with log parsing, in order to ensure the accuracy of log parsing, it is necessary to perform preliminary processing on the original initial log to be parsed generated by the system. The specific implementation method is as follows:

[0050] The step of determining the log to be parsed and performing a segmentation process on the log to be parsed to obtain a log segmentation sequence includes:

[0051] Determine the initial log to be parsed, extract the log content in the initial log to be parsed according to a preset regular expression, and determine it as the log to be parsed;

[0052] Perform word segmentation on the log to be parsed according to a preset segmentation method to obtain a log word segmentation sequence.

[0053] Among them, the initial log to be parsed can be understood as the original log generated during the system operation and to be parsed. The preset regular expression can be understood as a regular expression used to extract the template part and variable part in the log. Specifically, the preset regular expression can be set according to requirements during implementation. The preset segmentation method can be understood as a preset method of using a preset delimiter to segment the log to be parsed. The preset delimiters include but are not limited to spaces, commas, semicolons, tab characters, etc.

[0054] Preferably, the preset segmentation method can be understood as a method of using spaces to segment each word and each variable in the log to be parsed. So that when using a large language model to perform semantic analysis on the log to be parsed later, the log to be parsed is reversed by the large language model, and then the reversed log to be parsed is segmented and semantically analyzed by using the word segmentation method of the large language model itself, increasing the accuracy of the semantic analysis of the large language model on the log to be parsed; for example, a log to be parsed "Receive 2", after being segmented by spaces, is "Receive" and "2", or a log to be parsed "Sendto node4", after being segmented by spaces, is "Send", "to", "node", "4".

[0055] The log parsing method provided by the embodiments of this specification extracts the log content by using a preset regular expression, determines the extracted log content as the log to be parsed, eliminates the interference information in the initial log to be parsed, ensures the accuracy of the log to be parsed, and performs word segmentation on the log to be parsed according to the preset segmentation method to obtain a log word segmentation sequence, providing a prerequisite convenience for subsequent matching with the prefix tree.

[0056] Step 204: Match the log word segmentation sequence with the nodes in the prefix tree to determine the candidate grammar template corresponding to the log word segmentation sequence.

[0057] Among them, the prefix tree is constructed by nodes and paths. The nodes are the word segments in the log grammar template, and the paths are the association relationships of the word segments in the log grammar template. The log grammar template is determined according to the initial log.

[0058] Specifically, the prefix tree can be understood as a prefix parsing tree.

[0059] During specific implementation, by matching the log word segmentation sequence of the log to be parsed with the prefix tree, all the paths in the prefix tree that match the log word segmentation sequence can be obtained, so that the log grammar templates corresponding to all the paths can be determined.

[0060] In practical applications, there is no prefix tree when parsing the first log. Therefore, a prefix tree can be constructed first before that. The specific implementation method is as follows:

[0061] Before matching the log token sequence with the nodes in the prefix tree to determine the candidate grammar templates corresponding to the log token sequence, the following steps are also included:

[0062] Determine the initial log and perform tokenization on the initial log to obtain an initial log token sequence;

[0063] Input the initial log into a large language model to obtain a log semantic template corresponding to the initial log;

[0064] According to the initial log token sequence, determine a log grammar template corresponding to the log semantic template;

[0065] Use the tokens in the log grammar template as nodes and the association relationships of the tokens in the log grammar template as paths to construct a prefix tree.

[0066] Among them, the initial log can be understood as the first log in the log parsing process. The log semantic template can be understood as a log template obtained by performing semantic analysis on a certain log through a large language model and containing the semantic information of the log. The log grammar template can be understood as a log template that corresponds one-to-one with the tokens in the token sequence of the corresponding log and in which each non-variable element is the same as the corresponding log token sequence.

[0067] For example, if the first log is "Receive from node 4", the corresponding initial log token sequence is "Receive", "from", "node", "4". According to this log grouping sequence, a prefix tree containing the nodes "Receive", "from", "node" and the path "Receive-from-node-4" can be established.

[0068] The log parsing method provided by the embodiments of this specification constructs a prefix tree according to the initial log token sequence by pre-using a large language model, ensuring the corresponding relationship between each node and path in the prefix tree and the log, thereby ensuring the accuracy of log template matching based on the prefix tree.

[0069] In practical applications, in order to reduce the total number of log templates, logs can be clustered to establish log groups for subsequent statistics. The specific implementation method is as follows:

[0070] After constructing the prefix tree, the following steps are also included:

[0071] Create a corresponding log group for the initial log, store the log semantic template and the log syntax template corresponding to the log semantic template in the template pool, and establish an association relationship between the log group and the log semantic template.

[0072] Among them, the log group can be understood as the clustering group of logs. The logs under one log group are of the same type. The template pool can be understood as a storage pool containing the log semantic templates corresponding to each log group and the log syntax templates corresponding to each log semantic template.

[0073] Specifically, one log group corresponds to one log semantic template. All the logs under one log group share one log semantic template corresponding to this log group. One log semantic template can correspond to one or more log syntax templates. One or more logs under one log group correspond to one or more log syntax templates corresponding to this log group. All the log semantic templates corresponding to all the log groups, and all the log syntax templates that all the log semantic templates can correspond to, constitute the above-mentioned template pool.

[0074] The log parsing method provided by the embodiments of this specification, by creating a corresponding log group for the initial log, storing its corresponding log semantic template and log syntax template in the template pool, and associating the log group with the log semantic template and log syntax template, ensures the accuracy of subsequent log clustering. Constructing a prefix tree according to the initial log token sequence ensures the corresponding relationship between each node and path in the prefix tree and the log, thus ensuring the accuracy of log template matching according to the prefix tree, and realizing fully automatic log template matching and log clustering in this way, ensuring the efficiency of log parsing.

[0075] In practical applications, after constructing the prefix tree, the log token sequence can be matched with the nodes and paths in the prefix tree to determine the candidate syntax template. The specific implementation method is as follows:

[0076] The matching of the log token sequence with the nodes in the prefix tree to determine the candidate syntax template corresponding to the log token sequence includes:

[0077] Sequentially match each log token in the log token sequence with the nodes in the prefix tree to determine the target node, where the target node is any one or more nodes in the prefix tree;

[0078] Determine the candidate syntax template corresponding to the log token sequence according to the path between the root node and the target node in the prefix tree.

[0079] Among them, the root node in the prefix tree can be understood as the first node to start matching. The word corresponding to the root node is usually a space, indicating the start. Generally, the root node does not perform an actual matching process. That is, the first word in the log word segmentation sequence is matched with the next node of the root node; the candidate grammar template can be understood as all possible log grammar templates that match the log word segmentation sequence. The target node can be understood as the situation where all the words in the log word segmentation sequence are matched, or when no further matching can be performed in the prefix tree, and all the nodes passed by the path at that time.

[0080] Specifically, the prefix tree contains nodes and paths. Each node represents a word segment. The nodes include the root node, non-leaf nodes, and leaf nodes. A string of words formed by the path from the root node to a leaf node can correspond to a log grammar template in the template pool, and a string of words formed by the path from the root node to a non-leaf node can also correspond to a log grammar template in the template pool. Since a log semantic template corresponds to multiple log grammar templates and a log group corresponds to a log semantic template, a log group can correspond to multiple paths in the prefix parsing tree.

[0081] The log parsing method provided by the embodiments of this specification, by matching the log analysis sequence with the nodes of the prefix tree, while ensuring the accuracy of the match with the log grammar templates in the template pool, through the efficient search mechanism of the prefix tree itself, greatly improves the efficiency of determining the candidate grammar templates.

[0082] In practical applications, in order to match the subsequent determined path with the log word segmentation sequence, the matching can be carried out strictly in order. The specific implementation method is as follows:

[0083] The step of sequentially matching each log word segment in the log word segmentation sequence with the nodes in the prefix tree to determine the target node includes:

[0084] According to the order of each log word segment in the log word segmentation sequence and the front-back relationship between the nodes in the prefix tree, match each log word segment with the nodes in the prefix tree;

[0085] Determine the nodes in the prefix tree that match each log word segment as the target nodes;

[0086] Correspondingly, in the case where there are multiple target nodes, the step of determining the candidate grammar template corresponding to the log word segmentation sequence according to the path between the root node and the target nodes in the prefix tree includes:

[0087] Determine the word segmentation corresponding to each path according to the paths between the root node and each target node in the prefix tree, and determine the candidate grammar templates corresponding to the log word segmentation sequence according to the word segmentation corresponding to each path.

[0088] Among them, the order of each log word segmentation in the log word segmentation sequence can be understood as the position order of each log word segmentation in the log word segmentation sequence, and the front-back relationship between nodes in the prefix tree can be understood as the front-back relationship of the positions represented by the nodes and the positions represented by other nodes in the prefix tree.

[0089] In specific implementation, match each log word segmentation in the log word segmentation sequence with the nodes in the prefix tree one by one in order, and determine the nodes that can be successfully matched during the matching process as target nodes.

[0090] Then, after determining multiple target nodes as above, the word segmentation strung together by the paths between the root node and each target node in the prefix tree can be determined as the candidate grammar template corresponding to the log word segmentation sequence.

[0091] Specifically, the prefix tree contains nodes and paths. Each node represents a word segmentation. The nodes include a root node, non-leaf nodes, and leaf nodes. A string of words strung together by the path from the root node to a leaf node can correspond to a log grammar template in the template pool, and a string of words strung together by the path from the root node to a non-leaf node can also correspond to a log grammar template in the template pool. Since one log semantic template corresponds to multiple log grammar templates and one log group corresponds to one log semantic template, one log group can correspond to multiple paths in the prefix parsing tree.

[0092] For example, the root node of a prefix tree is "Root", the first-layer nodes, that is, the child nodes of the root node are "Receive", "Send", "<*>". The child node corresponding to "Receive" is "<*>", and the child node corresponding to "Send" is "<*>". Then, for a log to be parsed "Receive from node 4", when matching with this prefix tree, two candidate grammar templates "Receive<*>" and "<*>" can be obtained, where "Receive" and "Send" are non-leaf nodes, and "<*>" is a leaf node.

[0093] For another example, the root node of a prefix tree is "Root", the first-level nodes, i.e., the children nodes of the root node, are "Receive", "Send", "<*>". The children nodes corresponding to "Receive" are "from", "<*>". The children node corresponding to "Send" is "<*>". The children node "from" of "Receive" has children nodes "<*>". Then, for a log to be parsed "Receive from node 4", when matching it with this prefix tree, two candidate syntax templates "Receivefrom<*>","Receive<*>","<*>" can be obtained.

[0094] The log parsing method provided by the embodiments of this specification ensures the matching of the subsequent determined path with the log token sequence by sequentially matching the log token sequence with each node in the prefix tree, thereby improving the accuracy of the matching between the log token sequence and the log syntax templates in the template pool. <s

[0095] Step 206: According to a preset matching strategy, match the log token sequence with the candidate syntax templates to obtain a matching result.

[0096] Among them, the preset matching strategy can be understood as a preset strategy for performing quantity matching and alignment matching between the log token sequence and each candidate syntax template. The matching results include but are not limited to results such as loose matching, strict matching, and non-matching.

[0097] In practical applications, quantity matching can be performed first, and then alignment matching. The specific implementation methods for the case of satisfying quantity matching but not alignment matching are as follows:

[0098] There are multiple candidate syntax templates;

[0099] Correspondingly, the step of matching the log token sequence with the candidate syntax templates according to the preset matching strategy to obtain a matching result includes:

[0100] Determine the first word quantity of the log tokens included in the log token sequence and the second word quantity of the tokens in each candidate syntax template;

[0101] Determine the candidate syntax templates with the same second word quantity as the first word quantity as the candidate syntax templates to be selected;

[0102] Align the log tokens with the tokens in the candidate syntax templates to be selected;

[0103] In the case where it is determined that the log tokens after alignment do not match the tokens in the candidate syntax templates to be selected, obtain a first matching result.

[0104] Among them, the number of the first words can be understood as the number of word segments included in the log word segmentation sequence, and the number of the second words of each candidate grammar template can be understood as the number of word segments included in the candidate grammar template corresponding to each candidate grammar template. The first matching result can be understood as a loose match.

[0105] Then, filter out the candidate grammar templates with inconsistent numbers of the two words. The remaining candidate grammar templates are the candidate grammar templates to be selected.

[0106] After that, perform alignment matching, that is, first align the log word segments in the log word segmentation sequence with the word segments in the candidate grammar templates to be selected one by one in order, and then determine whether the word segments at the corresponding positions are the same. If there is no candidate grammar template with one-to-one corresponding and identical word segments, then it can be determined that the matching result is the first matching result, that is, a loose match.

[0107] The log parsing method provided by the embodiments of this specification obtains the first matching result, that is, a loose match, in the process of matching the log word segmentation sequence with each candidate grammar template when the number of word segments is the same, but the aligned log word segments do not match the word segments in each candidate grammar template to be selected, thereby improving the fault tolerance rate of template matching and making the subsequent log parsing of the log to be parsed more accurate.

[0108] In practical applications, the specific implementation methods for satisfying the quantity matching and the alignment matching are as follows:

[0109] After aligning the log word segments with the word segments in the candidate grammar template to be selected, it further includes:

[0110] When it is determined that the aligned log word segments match the word segments in the candidate grammar template to be selected, obtain the second matching result;

[0111] Specifically, following the description of the above implementation method, if there is a candidate grammar template with one-to-one corresponding and identical word segments, then it can be determined that the matching result is the second matching result, that is, a strict match, and then the candidate grammar template with one-to-one corresponding and identical word segments is the log grammar template corresponding to the log to be parsed.

[0112] The log parsing method provided by the embodiments of this specification obtains the second matching result, that is, a strict match, in the process of matching the log word segmentation sequence with each candidate grammar template when the number of word segments is the same and the aligned log word segments also match the word segments in each candidate grammar template to be selected. After the strict match, the corresponding log grammar template can be directly determined without using a large language model for subsequent operations, improving the processing efficiency of the entire process.

[0113] In practical applications, if the second matching result meets the preset matching condition, the log grouping can be directly determined, and the specific implementation method is as follows:

[0114] After obtaining the second matching result in the case of determining that the log word segments after alignment match the word segments in the candidate grammar template, the following is further included:

[0115] In the case where the second matching result meets the preset matching condition, the log grouping corresponding to the candidate grammar template is determined as the log grouping corresponding to the log to be parsed.

[0116] Specifically, in the process of matching the log word segment sequence with each candidate grammar template above, after the number of word segments is the same and the log word segments after alignment also match the word segments in each candidate grammar template, it indicates that there is the same template for the log grammar template of the log to be parsed in the template pool. Then, the log grouping corresponding to the matching candidate grammar template can be determined as the log grouping corresponding to the log to be parsed.

[0117] The log parsing method provided by the embodiments of this specification maps the log to the corresponding log grouping by determining that the match is the second matching result, that is, a strict match. While ensuring the accuracy of log parsing, it improves the log parsing efficiency, optimizes resource consumption, and reduces the cost of the entire log parsing process.

[0118] In practical applications, the specific implementation method for not meeting the quantity match is as follows:

[0119] After determining the first word quantity of the log word segments included in the log word segment sequence and the second word quantity of the word segments in each candidate grammar template, the following is further included:

[0120] In the case where the first word quantity is different from each second quantity, a third matching result is obtained.

[0121] Specifically, following the description of the above implementation method, in the case where the number of word segments in all candidate grammar templates is different from the number of word segments included in the log word segment sequence, that is, the third matching result can be obtained, and the third matching result can be understood as a non-match.

[0122] For example, a log word segment sequence is "End", and the candidate grammar templates are ["Send", "to", "node", "<*>"], ["Send", "<*>"], ["start", "prefetching", <*>"]. The number of word segments in the log word segment sequence is 1, and in each candidate grammar template, the number of word segments is not 1, that is, there is no candidate grammar template with the same number of word segments as the log word segment sequence. In this case, a third matching result of "non-match" is obtained.

[0123] The log parsing method provided by the embodiments of this specification directly obtains a second matching result, that is, a non-match, without the need to perform subsequent matching with the prefix tree during the process of matching the log tokenization sequence with each candidate grammar template when the number of tokens is inconsistent, further improving the processing efficiency of the entire process.

[0124] Step 208: When the matching result does not meet the preset matching condition, use a large language model to parse the log to be parsed to obtain a parsing result.

[0125] Among them, the preset matching condition can be understood as the existence of a strictly matching candidate grammar template.

[0126] When the matching result is that there is no strictly matching candidate grammar template, it is necessary to call a large language model to parse the log. The specific implementation method is as follows:

[0127] When the matching result does not meet the preset matching condition, using a large language model to parse the log to be parsed to obtain a parsing result includes:

[0128] When it is determined that the first matching result or the third matching result does not meet the preset matching condition, use a large prediction model to process the log to be parsed to obtain a predicted semantic template;

[0129] According to the relationship between the predicted semantic template and the log semantic templates in the template pool, determine the log grouping corresponding to the log to be parsed, where the template pool includes the log grammar templates and the log semantic templates corresponding to the log grammar templates, and there are log groupings with an associated relationship for the log semantic templates.

[0130] Among them, the relationship between the predicted semantic template and the log semantic template includes matching or not matching. Matching can be understood as the predicted semantic template and the log semantic template being exactly the same, and not matching can be understood as there being inconsistent words or orders between the predicted semantic template and the log semantic template.

[0131] Then, according to the matching or non-matching relationship between the predicted semantic template and the log semantic template, the log grouping corresponding to the log to be parsed can be determined according to different situations.

[0132] The log parsing method provided by the embodiments of this specification, when obtaining the first matching result (i.e., the matching result of the candidate grammar template that loosely matches the log tokenization sequence) and the third matching result (i.e., the matching result of the candidate grammar template that strictly matches the log tokenization sequence) as described above, uses a large language model to predict the predicted semantic template corresponding to the log grouping sequence, and then matches the predicted semantic template with the log semantic templates in the template pool. Through the accurate semantic analysis of the large language model, the intelligent analysis of the log to be parsed is improved, thereby improving the accuracy of template extraction. Moreover, through this method, the situation where logs of the same classification cannot be clustered into the same log grouping due to different description methods is reduced, improving the applicability and accuracy of log parsing in different systems and different log forms.

[0133] In practical applications, in order to reduce the resource consumption and labor costs brought by the training of the large language model, the large language model can be implemented by improving the prompt, and the specific implementation method is as follows:

[0134] The parsing of the log to be parsed using the large language model includes:

[0135] Determine the log parsing task and the log query problem according to the log to be parsed;

[0136] Determine a preset number of example semantic templates from the template pool according to the feature vector of the log to be parsed and the feature vector of the log semantic templates in the template pool, where the example semantic templates are subsets of the log semantic templates;

[0137] Parse the log to be parsed using the large language model according to the log parsing task, the log query problem, and the example semantic templates.

[0138] Among them, the log parsing task can be understood as a task used to describe what the large language model is to execute, the log query problem can be understood as a question that needs to be directly answered by the large language model, the feature vector of the log to be parsed can be understood as a vector encoded by the large language model that describes the log to be parsed, the feature vector of the log semantic template can be understood as a vector encoded by the large language model that describes the log semantic template, and the example semantic templates can be understood as a preset number of log semantic templates selected from the log semantic templates.

[0139] Then, based on the log to be parsed, the log parsing task and the log query problem corresponding to the log to be parsed can be determined. For example, if the log to be parsed is "Receive-from-node-4", then based on this log to be parsed, the log parsing task can be determined as, for example, "Please perform log template extraction", and the log query problem is "Please perform log template extraction on the following log: Receive-from-node-4". Another example is that if the log to be parsed is "Send to node 2", then based on this log to be parsed, the log parsing task can be determined as, for example, "Please execute the log template extraction task", and the log query problem is "Please perform log template extraction on the following log: Receive-from-node-4".

[0140] And determining a preset number of example semantic templates from the template pool according to the feature vector of the log to be parsed and the feature vectors of the log semantic templates in the template pool can be understood as calculating the cosine similarity between the vector description (i.e., the feature vector) encoded by the large language model that describes the log to be parsed and the vector descriptions (i.e., the feature vectors) encoded by the large language model that describe each log semantic template in the template pool, and sorting the log semantic templates according to the cosine similarity, and selecting the top k log semantic templates as the example semantic templates, where k can be understood as the above-mentioned preset number, and in practical applications, the value of k can be determined according to actual needs.

[0141] In specific implementation, identifying variable content in the log and classifying the variables helps to improve the accuracy of log parsing. Moreover, classifying these variables is also beneficial for subsequent data analysis tasks. Therefore, the embodiments of this specification construct a question text for the large language model so that the prompt text can achieve: enabling the large language model to identify variables and classify them into one of the outlined ten categories (that is, the prompt text can help the large language model understand which part of the log should be a variable and what the logical reasoning behind classifying these variables is), and performing template extraction.

[0142] Specifically, the prompt text can be designed as variable-aware prompt text. The variable-aware prompt contains three elements: instruction (i.e., the above log parsing task) for describing the task to be performed by the large language model; demonstration (i.e., the above example semantic template), a set of examples, where an example consists of a question and the true answer; query (i.e., the above log query question), which is the question that requires the large language model to directly answer. Among them, the examples can be constructed from the template pool. For the instruction to call the large language model for template extraction, multiple examples (e.g., three) can be directly constructed from the template pool according to the log semantic template; or, for the instruction of identifying and classifying variables in the log, based on constructing multiple examples according to the log semantic template, multiple additional examples (e.g., ten) can be added, with each example representing a different type of log semantic template and corresponding to a variable type, serving as seed examples. To obtain the seed examples, the cosine similarity between the vector description of the log to be parsed in the large language model and the vector description of each log semantic template existing in the template pool can be calculated, and then the log semantic templates can be sorted according to the cosine similarity, and the top k are selected from the log semantic templates as the k seed examples. Additionally, after using the large language model to perform semantic analysis on the log token sequence and extract the log semantic template, the extracted log semantic template will expand the above template pool.

[0143] Furthermore, after obtaining a preset number of seed examples, the answers of these seed examples can be marked, and the seed examples and their corresponding answers are used as demonstrations. Then, the log parsing task, the log query question, and this demonstration are input into the large language model, and thus the log parsing of the log to be parsed can be achieved.

[0144] The log parsing method provided by the embodiments of this specification reduces the cost of using the large language model by not training the large language model when using it, and by using the log parsing task, the log query question, and the example semantic template as prompts for the large language model, the large language model has increased variable recognition and classification capabilities on the basis of the original template extraction, enhancing the task understanding ability of the large language model, thereby improving the accuracy of template extraction using the large language model. Moreover, this prompt method can also facilitate subsequent data analysis.

[0145] In practical applications, in the case where there is a log semantic template that matches the predicted semantic template, the specific implementation manner of determining the log group corresponding to the log to be parsed is as follows:

[0146] There are multiple of the log semantic templates;

[0147] Correspondingly, determining the log group corresponding to the log to be parsed according to the relationship between the predicted semantic template and the log semantic templates in the template pool includes:

[0148] Match the predicted semantic template with each log semantic template in the template pool;

[0149] When it is determined that there is a log semantic template in the template pool that matches the predicted semantic template, the log semantic template that matches the predicted semantic template is used as the target log semantic template;

[0150] Determine the log group corresponding to the log to be parsed according to the target log semantic template.

[0151] Among them, the log semantic template that matches the predicted semantic template can be understood as the log semantic template with the same number of word segments as the predicted semantic template and the corresponding word segments at each position are the same after word segment alignment.

[0152] In specific implementation, after the predicted semantic template corresponding to the log to be parsed is determined by the large language model, the log to be parsed is matched with each log semantic template in the template pool, so as to find the log semantic template that matches the predicted semantic template.

[0153] For example, as shown in Table 1, there are three log semantic templates in the template pool, and the three log semantic templates correspond to three log groups.

[0154]

[0155] The log group sequence is "Accept from node 4", and the above log syntax templates do not match this log group sequence. Using the large language model, the semantic template of this log group sequence is extracted, and after extraction, the log semantic template "Received from a certain node" is obtained. This log semantic template is the same as the second log semantic template "Received from a certain node" in the template pool. Therefore, it can be determined that the log to be parsed corresponds to log group 2.

[0156] The log parsing method provided by the embodiments of this specification matches the predicted semantic template with the log semantic template, and when there is a match, the log group corresponding to the matching log semantic template indicates that the current predicted semantic template and the log semantic template have the same semantics, but there are variant descriptions, so it is determined as the log group of the log to be parsed, making the clustering of logs more intelligent and improving the accuracy of template extraction and log parsing.

[0157] In practical applications, after the log group is determined, the template pool and the prefix tree can also be updated to ensure the accuracy of the next log parsing (or the log parsing of the next log). The specific implementation method is as follows:

[0158] After determining the log group corresponding to the log to be parsed, the following steps are further included:

[0159] Map the log to be parsed to the log group;

[0160] According to the predicted semantic template, determine the new semantic template corresponding to the log to be parsed;

[0161] Store the new semantic template as a new grammar template in the template pool, and associate the new grammar template with the target log semantic template;

[0162] Using the word segmentation included in the new grammar template as the update node, and the association relationship of the word segmentation in the new grammar template as the update path, update the prefix tree according to the update node and the update path.

[0163] Among them, the update node can be understood as a newly added node in the prefix tree, and the update path can be understood as the newly added path corresponding to the front-back relationship between the word segmentations in the new grammar template. Continuing with the above example, for example, a new grammar template is "Sendtopoint 5", then the newly added nodes are send, to, point, <*>. Since there are already send and to nodes in the prefix tree, then according to the front-back relationship between to and point, a new node is added after to in the prefix tree, and the word segmentation corresponding to the node is point. Then, according to the front-back relationship between point and <*>, a new node <*> is added after point, thus realizing the update of the prefix tree.

[0164] The log parsing method provided by the embodiments of this specification, after determining the log group as described above, maps the log to the determined log group, updates the log grammar template in the template pool, and updates the prefix tree according to the updated template pool, ensuring the accuracy of the log parsing of the next log.

[0165] In the case where there is no log semantic template that matches the predicted semantic template, it is possible to first determine whether there is a log grammar template in the template pool that loosely matches the log to be parsed when the log to be parsed is matched with the log grammar templates in the template pool.

[0166] In the case where there is no log grammar template that loosely matches the log to be parsed, the specific implementation manner of determining the log group corresponding to the log to be parsed is as follows:

[0167] After matching the predicted semantic template with each log semantic template in the template pool, the following steps are further included:

[0168] In the case where there is no log semantic template in the template pool that matches the predicted semantic template, use the predicted semantic template as a new log semantic template;

[0169] Determine the log group corresponding to the log to be parsed according to the new log semantic template.

[0170] Specifically, if there is no log semantic template in the template pool that matches the predicted semantic template, it means that the predicted semantic template is a new log semantic template.

[0171] Then, in order to prevent incorrect matching caused by the same semantics but different expressions, this predicted log semantic template can first be semantically similar identified with the log semantic template corresponding to the log grammar template that is loosely matched with the log tokenization sequence determined above through a large language model.

[0172] In the case where the large language model considers that this predicted log semantic template is semantically different from the log semantic template corresponding to the log grammar template that is loosely matched with the log tokenization sequence determined above, use this predicted semantic template as a new log semantic template, and create a corresponding log group according to this predicted semantic template, so as to determine the log group corresponding to the log to be parsed as the log group corresponding to this predicted semantic template.

[0173] The log parsing method provided by the embodiments of this specification improves the accuracy of log parsing by matching the predicted semantic template with the log semantic template, and in the case where there is no match, it indicates that there is no log semantic template with the same semantics as the current predicted semantic template in the template pool, and performs another semantic recognition and comparison through a large language model. In the case where the large language model gives a result of different semantics, use this predicted semantic template as a new log semantic template, so as to determine a new log group for the log to be parsed, ensuring the clustering accuracy of the newly classified logs and further improving the accuracy of log parsing.

[0174] In the case where there is no log grammar template that is loosely matched with the log to be parsed, that is, in the case where all log grammar templates do not match the log to be parsed, the specific implementation manner of determining the log group corresponding to the log to be parsed is as follows:

[0175] Determining the log group corresponding to the log to be parsed according to the relationship between the predicted semantic template and the log semantic templates in the template pool includes:

[0176] In the case where it is determined that there is no log semantic template in the template pool that matches the predicted semantic template, an additional log semantic template is determined according to the log semantic template corresponding to the predicted semantic template and the candidate grammar template.

[0177] According to the additional log semantic template, the log group corresponding to the log to be parsed is determined.

[0178] Specifically, the matching process can refer to the above implementation method. After matching, if the large language model determines that this predicted log semantic template has the same semantics as the log semantic template corresponding to the log grammar template that is loosely matched with the log tokenization sequence determined above, it indicates that this predicted semantic template is a variant semantic template of the log semantic template in the template pool.

[0179] Then, this predicted semantic template is merged with this log semantic template in terms of generality, and the merged log semantic template is used as the additional log semantic template, and the log group corresponding to this log semantic template in the template pool is determined as the log group of the log to be parsed.

[0180] The log parsing method provided by the embodiments of this specification improves the accuracy of log parsing by matching the predicted semantic template with the log semantic template, and in the case where there is no match, determining the log group corresponding to the matching log semantic template, which indicates that there is no log semantic template with the same semantics as the current predicted semantic template in the template pool, and performing another semantic recognition and comparison through the large language model. In the case where the large language model gives the result of the same semantics, the log semantic template with the same semantics is updated using the additional log semantic template, and the log group corresponding to the log semantic template with the same semantics is determined as the log group of the log to be parsed, increasing the clustering fault tolerance of the logs and further improving the accuracy of log parsing.

[0181] In practical applications, in the case where there is no log semantic template that matches the predicted semantic template, the template pool and the prefix tree can also be updated to facilitate the accurate parsing of subsequent logs. The specific implementation method is as follows:

[0182] After determining the log group corresponding to the log to be parsed, the following steps are further included:

[0183] Map the log to be parsed to the log group;

[0184] According to the log tokenization sequence, determine the additional grammar template corresponding to the log to be parsed;

[0185] Store the additional log semantic template and the additional grammar template in the template pool, and associate the additional grammar template with the additional log semantic template;

[0186] Using the word segmentation contained in the newly added grammar template as the update nodes, and using the association relationship of the word segmentation in the newly added grammar template as the update path, update the prefix tree according to the update nodes and the update path.

[0187] Specifically, after determining the log group corresponding to the log to be parsed, map the log to be parsed to the corresponding log group.

[0188] To ensure the accuracy of the next log parsing, the prefix tree and the template pool can be updated. Specifically, during implementation, first determine the log grammar template corresponding to the log word segmentation sequence according to each word segmentation and its corresponding position in the log word segmentation sequence (words are represented by the words themselves, and variables are represented by <*>), and then the template pool can be updated according to the newly determined log semantic template and log grammar template.

[0189] It should be noted that after performing semantic similarity recognition on the predicted log semantic template and the log semantic template corresponding to the log grammar template that is loosely matched with the log word segmentation sequence determined above using the large language model, regardless of whether there is a log grammar template that is loosely matched with the log to be parsed, the prefix tree can be updated using the new log semantic template and log word segmentation sequence. The difference is that if the new log semantic template is implemented by updating the original log semantic template, then the log grammar template corresponding to the original log semantic template and the newly added grammar template determined by the log group sequence will correspond to the new log semantic template, and the new log semantic template will correspond to the log group corresponding to the original log semantic template; while if the new log semantic template is implemented by adding a new log semantic template, then only the newly added grammar template determined by the log group sequence needs to be corresponding to the new log semantic template, and a new log group is added to correspond to the new log semantic template.

[0190] Then, after updating the template pool, update the prefix tree according to the newly added log grammar templates in the template pool according to the word segmentation and the relationship between the word segmentations. For the specific implementation method of updating the prefix tree, refer to the embodiments of the above specification.

[0191] The log parsing method provided by the embodiments of this specification ensures the accuracy of the log parsing of the next log by mapping the log to the determined log group after determining the log group, updating the log semantic templates and log grammar templates in the template pool, and updating the prefix tree according to the updated template pool.

[0192] The log parsing method provided by the embodiments of this specification first performs word segmentation on the log to be parsed to obtain a log word segmentation sequence, and improves the log parsing efficiency through the matching between word segments. Then, it matches the log to be parsed with a prefix tree containing information of log syntax templates, finds candidate syntax templates close to the log to be parsed to determine all possible log syntax templates that can correspond to the log to be parsed, and according to a preset matching strategy, matches the log word segmentation sequence with the candidate syntax templates to determine whether there is a candidate syntax template that matches the current log to be parsed and meets the preset matching conditions. In the case where there is no candidate syntax template that meets the preset matching conditions, it uses a large model to parse the log to be parsed and understand the semantics of the log to be parsed, achieving more accurate log parsing. Moreover, since the large language model is only used when no candidate syntax template that meets the matching conditions is found, the cost of using the large language model is reduced, thus making the log parsing cost of applying this solution lower.

[0193] The following is combined with the attached Figure 3 to further illustrate the log parsing method. See Figure 3 , Figure 3 is a schematic diagram of the data structure of a log parsing method provided according to an embodiment of this specification. As Figure 3 shown, Figure 3 contains three main data structures: log grouping, template pool, and prefix parsing tree.

[0194] Among them, there can be one or more log groupings. All logs in the same log grouping share the same log semantic template (such as Figure 3 log grouping 1 in shares "Receive<*> ", log grouping n shares "Send<*> "). Each log grouping records the IDs of all the logs it contains, as well as the vector description encoded by the large language model that describes the logs in the log grouping. Each log grouping has its own log semantic template extracted by the large language model and one or more log syntax templates generated through statistical word frequency analysis. These templates can assist in the traversal of a new log in the prefix tree and the process of matching the log to the log syntax template; each word segment in the log syntax template is in one-to-one alignment with the word segments of the log obtained after word segmentation of the original log, used to distinguish the static part of the log syntax template and the log variable part; while the log semantic template will place one or more words in a log under a variable placeholder. For example, for a log "start prefetching…0", its corresponding log syntax template is ["start", "prefetching…<*> "]; all log syntax templates are cataloged as a dictionary, with the number of word segments in the log syntax template as the keyword, and the corresponding value is the template list of the corresponding log syntax template.

[0195] The template pool consists of semantic templates (i.e., log semantic templates) and syntactic templates (i.e., log syntactic templates). One semantic template corresponds to multiple syntactic templates in terms of grammar obtained by counting word frequencies, and one log group corresponds to one syntactic template. Therefore, there is an association relationship between the semantic templates and syntactic templates in the template pool and the log groups.

[0196] That is to say, one log group corresponds to a template cluster in the template pool. This template cluster contains one log semantic template corresponding to this log group and multiple log syntactic templates corresponding to this log semantic template. The template pool consists of multiple template clusters corresponding to multiple log groups. For example, as Figure 3 shown, the template cluster corresponding to log group 1 contains one log semantic template "Receive<*>" and multiple log syntactic templates

[0197] {3, [[‘Receive’,‘<*>’,‘KB’],[‘Receive’,‘<*>’,‘BYTES’]]}.

[0198] The prefix parsing tree contains nodes and paths. Each node represents a word segment. The nodes include a root node (such as Figure 3 Root in Figure 3 ), non-leaf nodes (such as Figure 3 Send, Receive in

[0199] Specifically, the log parsing method includes the following steps:

[0200] (1) Preprocessing

[0201] After the server receives a new log, it is necessary to perform preprocessing of log content extraction and word segmentation on this log. [[ID=,‘<*>’,‘BYTES’]]}。

[0202] First, use a basic regular expression to extract the log content part (i.e., the log to be parsed) from the original log (i.e., the above-mentioned initial log to be parsed). Some implementable solutions are to extract according to domain knowledge. For example, use regular expressions to extract common variables such as IP (Internet Protocol, a network protocol) addresses and block IDs (Identity document, used to uniquely identify blocks). The implementation of the log parsing method in the embodiments of this specification is to retain the entire original log message (i.e., the above-mentioned log content part) to ensure that the subsequent large language model can master the complete context of the log.

[0203] Then, use spaces to split (i.e., tokenize) the above-mentioned extracted log content part to obtain the tokenized log (i.e., the above-mentioned log token sequence). Then, when it is necessary to use a large model to perform semantic analysis on the extracted log message in the future, use the large language model to perform anti-spelling, and then use the word segmentation method of the large language model itself for segmentation and semantic analysis.

[0204] Through this preprocessing method, the need for domain knowledge is minimized while maintaining a high level of log parsing efficiency.

[0205] (2) Clustering matching and tree search

[0206] After obtaining the tokenized log above, in the order of each word (i.e., the above-mentioned tokens) in the tokenized log, match each word in the log against the nodes in the prefix tree. And only after matching the previous word can the subsequent word matching continue. When matching backward in the prefix tree, the next match only matches the child nodes of the current node; match in this matching manner until all the tokens in the tokenized log are matched, or no further matching can be performed in the prefix tree; during the entire matching process, every time a node is reached, a path from the root node to this node will be generated, and this path will correspond to a log grammar template. Add all the log grammar templates corresponding to all the paths generated during the matching process to a list as the candidate list of log grammar templates (i.e., the list of candidate grammar templates). Then, evaluate these candidate log grammar templates to determine the most suitable log grammar template and the corresponding log grouping for this new log.

[0207] Some implementable evaluation methods are to use similarity metrics and specific thresholds related to the data set to determine whether there is a match. Different from these evaluation methods, the log parsing method in the embodiments of this specification classifies the matching results into three different categories: i) strict match, ii) loose match, and iii) no match.

[0208] First, for the log syntax template of each candidate log group, compare the number of words in the received log with the number of words in the log syntax template of this log group; if the number of words is inconsistent, it can be immediately determined that it is not a "strict match".

[0209] Then, determine whether it is a "loose match". Specifically, align the words in the log syntax template with the words in the tokenized log one by one. Among them, the log syntax template containing the "<*>" wildcard word can match any word at the corresponding position in the log. For example, the log syntax template corresponding to "start prefetching…<*>" is ["start","prefetching…<*>"], and this template can achieve a loose match with a log of two words starting with the word "start" and having any word in the second position.

[0210] Furthermore, after the loose match, use regular expressions to perform a more strict alignment match between the tokenized log and the elements other than "<*>" in the log syntax template. If all the words are matched after alignment, it means that the tokenized log and the currently matched log syntax template are in strict match; it should be noted that the matching process will stop when a strict match occurs.

[0211] In the case of a strict match, this new log will be directly mapped to the log group corresponding to the matched log syntax template.

[0212] In the case of no strict match, the large language model will be called for template extraction, and then the corresponding data structure will be updated.

[0213] (III) Template Extraction and Parse Tree Update

[0214] In the case of no strict match as described above, first use the large language model to perform semantic analysis on the tokenized log to obtain the new log semantic template corresponding to the tokenized log (i.e., the above predicted semantic template).

[0215] After that, match this new log semantic template with the log semantic templates in the template pool to determine whether there is a log semantic template in the template pool that is the same as this new log semantic template.

[0216] If there is a log semantic template in the template pool that is the same as the new log semantic template, it indicates that the current log can be clustered with the log group corresponding to the matching log semantic template. However, the log syntax template corresponding to this log semantic template is not comprehensive enough. Therefore, map this new log to the log group corresponding to the matching log semantic template, and use this new log semantic template as an updated log syntax template, adding it to the template cluster corresponding to this log group in the template pool. Then, update the above prefix parsing tree according to the updated template pool to ensure that there is a node path corresponding to this new log syntax template in the prefix parsing tree.

[0217] If there is no log semantic template in the template pool that is the same as the new log semantic template, but the above matching has been determined to be a loose match, then the large language model will be consulted again. The large language model determines whether this new log semantic template belongs to a log semantic template that can be loosely matched. If the large language model believes that there is a log semantic template that can be loosely matched with this new log semantic template, a more general, merged log semantic template will be generated to match this new log semantic template and each log syntax template in this log group. Then, update this log semantic template and its corresponding log syntax template, and add this log semantic template and its corresponding log syntax template to the template pool. If a log still fails to be successfully matched with the log semantic templates in the template pool after going through the above entire process, then create a new log group corresponding to this log semantic template, and use this log semantic template as a log syntax template corresponding to this log semantic template, and update the parsing tree accordingly according to this log syntax template.

[0218] Optionally, the process of using the large language model to perform semantic analysis on the tokenized log to obtain the new log semantic template corresponding to the tokenized log can utilize a more powerful large language model, or use labeled data to perform supervised learning fine-tuning on the large language model to improve the accuracy of the large language model's semantic analysis.

[0219] Preferably, the process of using the large language model to perform semantic analysis on the tokenized log to obtain the new log semantic template corresponding to the tokenized log can be specifically implemented by introducing a variable-aware prompting method.

[0220] In practical applications, identifying variable content in logs and classifying variables helps improve accuracy. By classifying these variables, not only can the accuracy of template extraction be improved, but it is also beneficial for subsequent data analysis tasks. Therefore, a way of asking questions to the large language model is constructed. These prompts now have a dual purpose: enabling the large language model to identify variables and classify them into one of the ten classifications outlined (this improved prompting method helps the large language model understand which part of the log should be a variable and what the logical reasoning behind classifying these variables is), and performing template extraction.

[0221] Specifically, the variable-aware prompt contains three elements: Instruction: used to describe the task for the large language model to perform, Demonstration: a set of examples, where an example is a sample question and the true answer, Query: the question that needs to be directly answered by the large language model. Among them, the examples can be constructed from the template pool. When calling the large language model for template extraction, multiple examples (such as 3) can be constructed from the template pool; or, for the identification and classification of variables in the log, ten more examples can be added on this basis, with each example representing a different type of log variable, as seed examples. The subsequent template extraction results will expand the template pool. In order to obtain these template samples, the cosine similarity between the vector description of the query log in the large language model and the vector description of each template existing in the template pool can be calculated, and then the top k samples can be selected as k task exemplars, as seed examples.

[0222] The log parsing method provided in the embodiments of this specification reduces the need for professional knowledge in log parsing through minimal preprocessing, while ensuring the efficiency of log parsing. In addition, the use of a prefix parsing tree for log template matching further improves the efficiency of log parsing. Moreover, during the log parsing process, the large language model is only called for log parsing in cases of non-matching or loose matching, which reduces the cost of using the large language model, optimizes resource consumption, and significantly improves the accuracy of log parsing through semantic analysis of the log parsing by calling the large language model. Among them, the large language model does not need to be pre-trained, but uses the large model through examples, further reducing manual consumption and increasing the versatility of this method in various different fields and log formats.

[0223] Corresponding to the above method embodiments, this specification also provides embodiments of a log parsing device. Refer to Figure 4 , Figure 4 which is a schematic structural diagram of a log parsing device provided according to an embodiment of this specification. As Figure 4 shown, the device includes:

[0224] The log word segmentation sequence obtaining module 402 is configured to determine the log to be parsed, perform word segmentation processing on the log to be parsed, and obtain a log word segmentation sequence;

[0225] The candidate grammar template determining module 404 is configured to match the log word segmentation sequence with the nodes in the prefix tree to determine the candidate grammar template corresponding to the log word segmentation sequence, where the prefix tree is constructed by nodes and paths, the nodes are the word segments in the log grammar template, the paths are the association relationships of the word segments in the log grammar template, and the log grammar template is determined according to the initial log;

[0226] The grammar template matching module 406 is configured to match the log word segmentation sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result;

[0227] The log parsing module 408 is configured to, when the matching result does not meet the preset matching condition, use a large language model to parse the log to be parsed to obtain a parsing result.

[0228] Optionally, the device further includes a prefix tree construction module, which is configured to:

[0229] Determine the initial log, perform word segmentation processing on the initial log, and obtain an initial log word segmentation sequence;

[0230] Input the initial log into a large language model to obtain a log semantic template corresponding to the initial log;

[0231] According to the initial log word segmentation sequence, determine a log grammar template corresponding to the log semantic template;

[0232] Use the word segments in the log grammar template as nodes and the association relationships of the word segments in the log grammar template as paths to construct a prefix tree.

[0233] Optionally, the device further includes an initial template storage module, which is configured to:

[0234] Establish a corresponding log group for the initial log, store the log semantic template and the log grammar template corresponding to the log semantic template in a template pool, and establish an association relationship between the log group and the log semantic template.

[0235] Optionally, the log word segmentation sequence obtaining module 402 is further configured to:

[0236] Determine the initial log to be parsed, extract the log content in the initial log to be parsed according to a preset regular expression, and determine it as the log to be parsed;

[0237] Perform word segmentation on the log to be parsed according to a preset segmentation method to obtain a log word segmentation sequence.

[0238] Optionally, the candidate grammar template determination module 404 is further configured to:

[0239] Sequentially match each log word in the log word segmentation sequence with the nodes in the prefix tree to determine target nodes, where the target nodes are any one or more nodes in the prefix tree;

[0240] Determine the candidate grammar template corresponding to the log word segmentation sequence according to the path between the root node and the target nodes in the prefix tree.

[0241] Optionally, the candidate grammar template determination module 404 is further configured to:

[0242] Match each log word in the log word segmentation sequence with the nodes in the prefix tree according to the order of each log word in the log word segmentation sequence and the front-back relationship between the nodes in the prefix tree;

[0243] Determine the nodes in the prefix tree that match each log word as target nodes;

[0244] Correspondingly, when there are multiple target nodes, the candidate grammar template determination module 404 is further configured to:

[0245] Determine the word segmentation corresponding to each path according to the paths between the root node and each target node in the prefix tree, and determine the candidate grammar template corresponding to the log word segmentation sequence according to the word segmentation corresponding to each path.

[0246] Optionally, there are multiple candidate grammar templates;

[0247] Correspondingly, the grammar template matching module 406 is further configured to:

[0248] Determine the number of the first words of the log words included in the log word segmentation sequence and the number of the second words of the words in each candidate grammar template;

[0249] Determine the candidate grammar templates with the same number of second words as the number of first words as the candidate grammar templates to be selected;

[0250] Align the log words with the words in the candidate grammar templates to be selected;

[0251] When it is determined that the log words after alignment do not match the words in the candidate grammar templates to be selected, obtain a first matching result.

[0252] Optionally, the device further includes a second grammar template matching module, which is configured to:

[0253] When it is determined that the log word segmentation after alignment matches the word segmentation in the candidate grammar template, a second matching result is obtained;

[0254] When the second matching result meets the preset matching condition, the log group corresponding to the candidate grammar template is determined as the log group corresponding to the log to be parsed.

[0255] Optionally, the device further includes a third grammar template matching module, configured to:

[0256] When the number of the first words is different from each number of the second words, a third matching result is obtained.

[0257] Optionally, the log parsing module 408 is further configured to:

[0258] When it is determined that the first matching result or the third matching result does not meet the preset matching condition, the log to be parsed is processed by using a large prediction model to obtain a predicted semantic template;

[0259] According to the relationship between the predicted semantic template and the log semantic templates in the template pool, the log group corresponding to the log to be parsed is determined, where the template pool includes the log grammar template and the log semantic templates corresponding to the log grammar template, and there are log groups with an associated relationship for the log semantic templates.

[0260] Optionally, there are multiple log semantic templates;

[0261] Correspondingly, the log parsing module 408 is further configured to:

[0262] Match the predicted semantic template with each log semantic template in the template pool;

[0263] When it is determined that there is a log semantic template in the template pool that matches the predicted semantic template, the log semantic template that matches the predicted semantic template is used as the target log semantic template;

[0264] According to the target log semantic template, the log group corresponding to the log to be parsed is determined.

[0265] Optionally, the device further includes a first sub-log parsing module, configured to:

[0266] When it is determined that there is no log semantic template in the template pool that matches the predicted semantic template, the predicted semantic template is used as a new log semantic template;

[0267] Determine the log group corresponding to the log to be parsed according to the newly added log semantic template.

[0268] Optionally, the device further includes a second sub-log parsing module, configured to:

[0269] In the case where it is determined that there is no log semantic template in the template pool that matches the predicted semantic template, determine a newly added log semantic template according to the log semantic template corresponding to the predicted semantic template and the candidate grammar template;

[0270] Determine the log group corresponding to the log to be parsed according to the newly added log semantic template.

[0271] Optionally, the device further includes a first prefix tree update module, configured to:

[0272] Map the log to be parsed to the log group;

[0273] Determine the newly added semantic template corresponding to the log to be parsed according to the predicted semantic template;

[0274] Store the newly added semantic template as a newly added grammar template in the template pool, and associate the newly added grammar template with the target log semantic template;

[0275] Use the word segmentation included in the newly added grammar template as the update node, use the association relationship of the word segmentation in the newly added grammar template as the update path, and update the prefix tree according to the update node and the update path.

[0276] Optionally, the device further includes a second prefix tree update module, configured to:

[0277] Map the log to be parsed to the log group;

[0278] Determine the newly added grammar template corresponding to the log to be parsed according to the log word segmentation sequence;

[0279] Store the newly added log semantic template and the newly added grammar template in the template pool, and associate the newly added grammar template with the newly added log semantic template;

[0280] Use the word segmentation included in the newly added grammar template as the update node, use the association relationship of the word segmentation in the newly added grammar template as the update path, and update the prefix tree according to the update node and the update path.

[0281] Optionally, the device further includes a third sub-log parsing module, configured to:

[0282] Determine a log parsing task and a log query problem according to the log to be parsed;

[0283] Determine a preset number of example semantic templates from the template pool according to the feature vector of the log to be parsed and the feature vector of the log semantic template in the template pool, where the example semantic template is a subset of the log semantic template;

[0284] Parse the log to be parsed by using a large language model according to the log parsing task, the log query problem, and the example semantic template.

[0285] The above is a schematic solution of a log parsing device according to an embodiment of the present specification. It should be noted that the technical solution of the log parsing device and the technical solution of the above log parsing method belong to the same concept. For the details not described in the technical solution of the log parsing device, reference can be made to the description of the technical solution of the above log parsing method.

[0286] Figure 5 FIG. shows a structural block diagram of a computing device 500 according to an embodiment of the present specification. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. The processor 520 is connected to the memory 510 through a bus 530, and a database 550 is used to store data.

[0287] The computing device 500 further includes an access device 540, and the access device 540 enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of wired or wireless network interfaces (for example, a network interface card (NIC)), such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0288] In one embodiment of this specification, the above components of the computing device 500 and Figure 5 other components not shown therein may also be connected to each other, for example, via a bus. It should be understood that Figure 5 the structural block diagram of the computing device shown is only for illustrative purposes and is not a limitation on the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0289] The computing device 500 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs, Personal Computers). The computing device 500 can also be a mobile or stationary server.

[0290] Among them, the processor 520 is used to execute the following computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above log parsing method are implemented.

[0291] The above is a schematic solution of a computing device in this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above log parsing method belong to the same concept. For the details not described in detail in the technical solution of the computing device, reference can be made to the description of the technical solution of the above log parsing method.

[0292] One embodiment of this specification also provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the above log parsing method are implemented.

[0293] The above is a schematic solution of a computer-readable storage medium in this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above log parsing method belong to the same concept. For the details not described in detail in the technical solution of the storage medium, reference can be made to the description of the technical solution of the above log parsing method.

[0294] One embodiment of this specification also provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by the processor, the steps of the above log parsing method are implemented.

[0295] The above is a schematic solution of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-mentioned log parsing method belong to the same concept. For the details not described in the technical solution of the computer program product, reference can be made to the description of the technical solution of the above-mentioned log parsing method.

[0296] The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0297] The computer instructions include computer program code, which may be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0298] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of this specification are not limited by the described action sequence, because according to the embodiments of this specification, certain steps may be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0299] In the above embodiments, the descriptions of the various embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0300] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the embodiments of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is only limited by the claims and their full scope and equivalents.

Claims

1. A log parsing method, comprising: Determine the log to be parsed, and perform word segmentation processing on the log to be parsed to obtain a log word segmentation sequence; Match the log word segmentation sequence with the nodes in the prefix tree to determine the candidate grammar templates corresponding to the log word segmentation sequence, wherein the prefix tree is constructed by nodes and paths, the nodes are the word segments in the log grammar template, and the paths are the association relationships of the word segments in the log grammar template, and the log grammar template is determined according to the initial log; Match the log word segmentation sequence with the candidate grammar templates according to a preset matching strategy to obtain a matching result; In the case where the matching result does not meet the preset matching condition, use a large language model to parse the log to be parsed to obtain a parsing result.

2. The log parsing method according to claim 1, before the step of matching the log word segmentation sequence with the nodes in the prefix tree to determine the candidate grammar templates corresponding to the log word segmentation sequence, further comprising: Determine the initial log, and perform word segmentation processing on the initial log to obtain an initial log word segmentation sequence; Input the initial log into a large language model to obtain the log semantic template corresponding to the initial log; Determine the log grammar template corresponding to the log semantic template according to the initial log word segmentation sequence; Use the word segments in the log grammar template as nodes, and use the association relationships of the word segments in the log grammar template as paths to construct a prefix tree.

3. The log parsing method according to claim 1 or 2, after the step of constructing the prefix tree, further comprising: Establish a corresponding log group for the initial log, store the log semantic template and the log grammar template corresponding to the log semantic template in a template pool, and establish an association relationship between the log group and the log semantic template.

4. The log parsing method according to claim 1 or 2, the step of determining the log to be parsed, and performing word segmentation processing on the log to be parsed to obtain a log word segmentation sequence, includes: Determine the initial log to be parsed, extract the log content in the initial log to be parsed according to a preset regular expression, and determine it as the log to be parsed; Perform word segmentation processing on the log to be parsed according to a preset segmentation method to obtain a log word segmentation sequence.

5. The log parsing method according to claim 1 or 2, the step of matching the log word segmentation sequence with the nodes in the prefix tree to determine the candidate grammar templates corresponding to the log word segmentation sequence, includes: Sequentially match each log word segment in the log word segmentation sequence with the nodes in the prefix tree to determine the target nodes, wherein the target nodes are any one or more nodes in the prefix tree; Determine the candidate grammar templates corresponding to the log word segmentation sequence according to the paths between the root node and the target nodes in the prefix tree.

6. The log parsing method according to claim 5, the step of sequentially matching each log word segment in the log word segmentation sequence with the nodes in the prefix tree to determine the target nodes, includes: Match each log token in the log token sequence according to the order of the log tokens and the forward and backward relationships between nodes in the prefix tree; Determine the nodes in the prefix tree that match the log tokens as target nodes; Correspondingly, when there are multiple target nodes, determining the candidate grammar template corresponding to the log token sequence according to the path between the root node and the target nodes in the prefix tree includes: Determine the tokens corresponding to each path according to the paths between the root node and each target node in the prefix tree, and determine the candidate grammar template corresponding to the log token sequence according to the tokens corresponding to each path.

7. The log parsing method according to claim 1 or 2, wherein the candidate grammar template includes multiple; Correspondingly, matching the log token sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result includes: Determine the first number of tokens of the log tokens included in the log token sequence and the second number of tokens of the tokens in each candidate grammar template; Determine the candidate grammar templates with the same second number of tokens as the first number of tokens as the candidate grammar templates to be selected; Align the log tokens with the tokens in the candidate grammar templates to be selected; In the case where it is determined that the log tokens after alignment do not match the tokens in the candidate grammar templates to be selected, obtain a first matching result.

8. The log parsing method according to claim 7, after aligning the log tokens with the tokens in the candidate grammar templates to be selected, further includes: In the case where it is determined that the log tokens after alignment match the tokens in the candidate grammar templates to be selected, obtain a second matching result; In the case where the second matching result meets a preset matching condition, determine the log group corresponding to the candidate grammar template to be the log group corresponding to the log to be parsed.

9. The log parsing method according to claim 7, after determining the first number of tokens of the log tokens included in the log token sequence and the second number of tokens of the tokens in each candidate grammar template, further includes: In the case where the first number of tokens is not the same as each second number of tokens, obtain a third matching result.

10. The log parsing method according to claim 9, in the case where the matching result does not meet a preset matching condition, parsing the log to be parsed using a large language model to obtain a parsing result, including: In the case where it is determined that the first matching result or the third matching result does not meet a preset matching condition, process the log to be parsed using a large prediction model to obtain a predicted semantic template; Determine the log group corresponding to the log to be parsed according to the relationship between the predicted semantic template and the log semantic templates in the template pool, where the template pool includes the log grammar template and the log semantic templates corresponding to the log grammar template, and there are log groups with an associated relationship for the log semantic templates.

11. The log parsing method according to claim 10, wherein the log semantic template includes multiple; Accordingly, determining the log group corresponding to the log to be parsed according to the relationship between the predicted semantic template and the log semantic templates in the template pool includes: Matching the predicted semantic template with each log semantic template in the template pool; When it is determined that there is a log semantic template in the template pool that matches the predicted semantic template, using the log semantic template that matches the predicted semantic template as the target log semantic template; Determining the log group corresponding to the log to be parsed according to the target log semantic template.

12. The log parsing method according to claim 11, after matching the predicted semantic template with each log semantic template in the template pool, further including: When it is determined that there is no log semantic template in the template pool that matches the predicted semantic template, using the predicted semantic template as a new log semantic template; Determining the log group corresponding to the log to be parsed according to the new log semantic template.

13. The log parsing method according to claim 10, determining the log group corresponding to the log to be parsed according to the relationship between the predicted semantic template and the log semantic templates in the template pool includes: When it is determined that there is no log semantic template in the template pool that matches the predicted semantic template, determining a new log semantic template according to the predicted semantic template and the log semantic template corresponding to the candidate grammar template; Determining the log group corresponding to the log to be parsed according to the new log semantic template.

14. The log parsing method according to claim 11, after determining the log group corresponding to the log to be parsed, further including: Mapping the log to be parsed to the log group; Determining a new grammar template corresponding to the log to be parsed according to the predicted semantic template; Storing the new grammar template in the template pool, and associating the new grammar template with the target log semantic template; Using the words included in the new grammar template as update nodes, using the association relationship of the words in the new grammar template as an update path, and updating the prefix tree according to the update nodes and the update path.

15. The log parsing method according to claim 12 or 13, after determining the log group corresponding to the log to be parsed, further including: Mapping the log to be parsed to the log group; Determining a new semantic template corresponding to the log to be parsed according to the predicted semantic template; Storing the new log semantic template and the new grammar template determined according to the new semantic template in the template pool, and associating the new grammar template with the new log semantic template; Using the words included in the new grammar template as update nodes, using the association relationship of the words in the new grammar template as an update path, and updating the prefix tree according to the update nodes and the update path.

16. The log parsing method according to claim 3, using a large language model to parse the log to be parsed, including: Determine a log parsing task and a log query problem according to the log to be parsed; Determine a preset number of example semantic templates from the template pool according to the feature vector of the log to be parsed and the feature vector of the log semantic template in the template pool, where the example semantic template is a subset of the log semantic template; Parse the log to be parsed by using a large language model according to the log parsing task, the log query problem, and the example semantic template.

17. A log parsing device, comprising: A log token sequence obtaining module, configured to determine a log to be parsed and perform tokenization processing on the log to be parsed to obtain a log token sequence; A candidate grammar template determining module, configured to match the log token sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log token sequence, where the prefix tree is constructed by nodes and paths, the nodes are tokens in a log grammar template, the paths are association relationships of tokens in the log grammar template, and the log grammar template is determined according to an initial log; A grammar template matching module, configured to match the log token sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result; A log parsing module, configured to parse the log to be parsed by using a large language model to obtain a parsing result when the matching result does not meet a preset matching condition.

18. A computing device, comprising: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the log parsing method according to any one of claims 1 to 16 are implemented.

19. A computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the log parsing method according to any one of claims 1 to 16 are implemented.

20. A computer program product, comprising computer programs / instructions. When the computer programs / instructions are executed by a processor, the steps of the log parsing method according to any one of claims 1 to 16 are implemented.

Images