Log parsing method and device

Abstract

Embodiments of the present disclosure provide a log parsing method and device. The log parsing method comprises: determining a log to be parsed, and carrying out tokenization processing on said log to obtain a log tokenization sequence; matching the log tokenization sequence with nodes in a prefix tree, and determining a candidate syntax template corresponding to the log tokenization sequence, wherein the prefix tree is constructed by the nodes and paths, the nodes are tokens in a log syntax template, each path represents an association relationship of tokens in the log syntax template, and the log syntax template is determined on the basis of an initial log; matching the log tokenization sequence with the candidate syntax template on the basis of a preset matching policy to obtain a matching result; and when the matching result does not satisfy a preset matching condition, using a large language model to parse said log, to obtain a parsing result. The method can improve the log parsing accuracy and reduce log parsing costs.

Description

[0001] Log Parsing Method and Apparatus This disclosure claims priority to Chinese Patent Application No. 202410139466.3, filed with the Chinese Patent Office on January 31, 2024, entitled "Log Parsing Method and Apparatus," the entire contents of which are incorporated herein by reference. Technical Field: Embodiments of the present disclosure relate to the field of log analysis technology, and more particularly, to a log parsing method. One or more embodiments of the present disclosure also relate to a log parsing apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background: Log data volumes are typically quite large. For example, for large-scale cloud services, it's common to generate trillions of log data rows daily. Log data plays an indispensable role in system diagnosis, security analysis, and performance tuning, necessitating data analysis. However, manual analysis of such large volumes of log data is inefficient, leading to the need for automated log data analysis. To better implement automated log analysis, log parsing technology is required to convert raw log data into a structured format for subsequent downstream analysis tasks. However, the complexity of modern systems and the diversity and volatility of logs pose significant challenges to existing automated log parsing technologies. Therefore, a log parsing solution with lower costs and higher accuracy is urgently needed to address these issues. SUMMARY OF THE INVENTION In light of this, embodiments of the present disclosure provide a log parsing method. One or more embodiments of the present disclosure also relate to a log parsing apparatus, a computing device, a computer-readable storage medium, and a computer program product to address technical deficiencies in the prior art. According to a first aspect of an embodiment of the present disclosure, a log parsing method is provided, comprising: determining a log to be parsed, and performing word segmentation processing on the log to be parsed to obtain a log word segmentation sequence; matching the log word segmentation sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log word segmentation sequence, wherein the prefix tree is constructed using nodes and paths, the nodes are words in a log grammar template, the paths are associations between words in the log grammar template, and the log grammar template is determined based on an initial log; matching the log word segmentation sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result; and if the matching result does not meet a preset matching condition, parsing the log to be parsed using a large language model to obtain a parsing result.According to a second aspect of an embodiment of the present disclosure, a log parsing apparatus is provided, comprising: a log word segmentation sequence obtaining module configured to determine a log to be parsed and perform word segmentation processing on the log to be parsed to obtain a log word segmentation sequence; a candidate grammar template determining module configured to match the log word segmentation sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log word segmentation sequence, wherein the prefix tree is constructed using nodes and paths, the nodes are words in a log grammar template, the paths are associations between words in the log grammar template, and the log grammar template is determined based on an initial log; a grammar template matching module configured to match the log word segmentation sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result; and a log parsing module configured to, if the matching result does not meet a preset matching condition, parse the log to be parsed using a large language model to obtain a parsing result. According to a third aspect of an embodiment of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions. When executed by the processor, the computer-executable instructions implement the steps of the log parsing method described above. According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, storing computer-executable instructions. When executed by the processor, the instructions implement the steps of the log parsing method described above. According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program / instructions. When executed by the processor, the computer program / instructions implement the steps of the log parsing method described above. An embodiment of the present disclosure provides a log parsing method, comprising: determining a log to be parsed, and performing word segmentation processing on the log to be parsed to obtain a log word segmentation sequence; matching the log word segmentation sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log word segmentation sequence, wherein the prefix tree is constructed using nodes and paths, the nodes are words in a log grammar template, the paths are associations between words in the log grammar template, and the log grammar template is determined based on an initial log; matching the log word segmentation sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result; and if the matching result does not meet a preset matching condition, parsing the log to be parsed using a large language model to obtain a parsing result.Specifically, the method first segments the log to be parsed to obtain a log segmentation sequence. By matching the segmentations, log parsing efficiency is improved. Subsequently, the log to be parsed is matched with a prefix tree containing information about log grammar templates to identify candidate grammar templates that are close to the log to be parsed. This method determines all possible log grammar templates that can correspond to the log to be parsed. Based on a preset matching strategy, the log segmentation sequence is matched with the candidate grammar templates to determine whether there is a candidate grammar template that matches the current log to be parsed and meets preset matching conditions. If no candidate grammar template meets the preset matching conditions, the log to be parsed is parsed using the large model to understand the semantics of the log to be parsed, thereby achieving more accurate log parsing. Furthermore, since the large language model is only used when no candidate grammar template that meets the matching conditions is found, the cost of using the large language model is reduced, thereby lowering the log parsing cost using this solution. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a diagram illustrating a specific application scenario of a log parsing method provided according to an embodiment of the present disclosure; Figure 2 is a flowchart of a log parsing method provided according to an embodiment of the present disclosure; Figure 3 is a diagram illustrating an example data structure of a log parsing method provided according to an embodiment of the present disclosure; Figure 4 is a schematic diagram illustrating the structure of a log parsing apparatus provided according to an embodiment of the present disclosure; and Figure 5 is a block diagram illustrating the structure of a computing device provided according to an embodiment of the present disclosure. The following description sets forth numerous specific details to facilitate a thorough understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art may make similar generalizations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific implementations disclosed below. The terminology used in one or more embodiments of the present disclosure is intended solely to describe specific embodiments and is not intended to limit the present disclosure. As used in one or more embodiments of the present disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be understood that although the terms "first," "second," and so on may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of the present disclosure.Depending on the context, the term "if" as used herein can be interpreted as "at the time of," "when," or "in response to a determination." Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data, etc.) involved in one or more embodiments of this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or deny. In one or more embodiments of this disclosure, a large model refers to a deep learning model with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. Large models, also known as foundation models, are pre-trained on large, unlabeled corpora to produce pre-trained models with over 100 million parameters. These models are adaptable to a wide range of downstream tasks and have good generalization capabilities. Examples include the Large Language Model (LLM) and the multimodal pre-training model. In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before being applied to various tasks. Large models can be widely used in fields such as natural language processing (NLP) and computer vision. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image captioning (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. Key application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. First, the terms used in one or more embodiments of this disclosure are explained.Log parsing: A technology that extracts log templates and variables from raw logs, thereby converting log data into structured data. Log templates can be understood as static keywords that describe the events corresponding to the log data, such as "receive," "send," and "node," and are typically expressed in natural language. Variables can be understood as dynamic variables within the log data, such as data representing IP (Internet Protocol) addresses and time. Prefix parse tree: A tree structure with a common prefix format used for log parsing, consisting of a root node, non-leaf nodes, and leaf nodes. Each node represents a word or placeholder. The path from the root node to any node is strung together in the order of the nodes to obtain a log template, also known as a log grammar template. For ease of explanation, the prefix parse tree in the following embodiments is abbreviated as the prefix tree, and this will not be discussed in detail. To ensure the smooth operation of cloud systems and improve their observability, they generate a large amount of operational data for analysis, such as log data, trace data, and metrics data. Log data is highly flexible and the most commonly used data for analysis. By analyzing log data, a wide range of anomalies and faults encountered during cloud system operation can be identified and diagnosed. However, the volume of log data is enormous. For large-scale cloud services, it is common to generate trillions of log lines daily. Therefore, manual analysis is inefficient, and automated log data analysis is urgently needed. To facilitate log output, logs come in a variety of formats. Therefore, in automated log analysis, it is crucial to automatically identify the changing content (i.e., variables) and the log template structure (i.e., log templates). When variables in each log line can be automatically identified and structured into high-dimensional metrics data (i.e., structured data), a variety of systematic methods for analyzing high-dimensional metrics data can be employed, enabling applications such as anomaly detection and root cause analysis. This helps cloud system personnel to quickly identify and detect system problems, nip them in the bud, and locate and solve problems in a timely manner.Log data is described in a manner very similar to natural language. Traditional automatic log parsing methods only consider the statistical variation of words in log data and fail to interpret the semantic information present in natural language. Consequently, these methods suffer from low accuracy. With the development of large language models, researchers have begun utilizing them for log parsing. However, because large language models require significant computing resources, the cost of using them is high. In light of this, we have innovatively developed a method that seamlessly integrates semantic analysis from large language models with traditional automatic log parsing methods (i.e., statistical machine learning algorithms for log parsing). This method leverages the strengths of both technologies, allowing statistical machine learning algorithms to utilize large language models to extract log patterns in specific, critical situations. This not only improves accuracy but also reduces the cost of using large language models. Specifically, this disclosure provides a log parsing method, which also includes a log parsing apparatus, a computing device, a computer-readable storage medium, and a computer program product, each of which is described in detail in the following embodiments. In practical applications, the log parsing method provided by the embodiments of the present disclosure can be applied to any application scenario requiring log parsing, including but not limited to mobile phone system log parsing, software execution log parsing, computer log parsing, and cloud system log parsing. For ease of understanding, the following explanation uses the log parsing method provided by the embodiments of the present disclosure applied to cloud system log parsing as an example. The following further describes the log parsing method provided by the embodiments of the present disclosure, using the application of the log parsing method provided by the embodiments of the present disclosure to cloud system log parsing as an example, with reference to FIG1 . FIG1 is a diagram illustrating a specific application scenario of the log parsing method provided by one embodiment of the present disclosure.As shown in FIG. 1 , FIG. 1 includes a client 102 and a cloud server 104. The client 102 displays a cloud system interface for providing cloud system usage services to users. The cloud server 104 is used to complete the task of log parsing logs generated by the operation of the cloud system. The client 102 includes but is not limited to mobile phones, tablet computers, laptops, desktop computers, etc. The cloud server 104 can be understood as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. In other feasible application scenarios (such as mobile phone system log parsing, local computer log parsing, etc.), the cloud server can be replaced by other servers, such as a physical server, a server cluster consisting of multiple physical servers, a distributed system, etc. In a specific implementation, a user operates the cloud system through a cloud system interface displayed on client 102. During the operation of the cloud system, logs to be parsed are generated. Client 102 sends the logs to be parsed to cloud server 104 of the cloud system. After receiving the logs to be parsed, cloud server 104 first performs word segmentation on the logs to be parsed to obtain a log word sequence after word segmentation. Cloud server 104 then matches the log word sequence with a prefix tree representing log grammar template information obtained by parsing the initial log. Specifically, the nodes of the prefix tree represent word segments in the log grammar template, and the paths represent the associations between the word segments. After matching, candidate grammar templates corresponding to the log word sequence are obtained. After obtaining the candidate grammar templates, the log word sequence is matched with the candidate grammar template using a preset matching strategy. If the matching results do not meet the preset matching conditions, that is, if none of the candidate grammar templates match the log word sequence, a large language model is used to perform semantic analysis on the logs to be parsed, thereby parsing the logs to be parsed. Get the log parsing results.In the log parsing method provided by the embodiments of the present disclosure, a cloud server segments the logs to be parsed, generated by users on a client using the cloud system, to obtain a log segmentation sequence. By matching the segmentations, log parsing efficiency is improved. The logs to be parsed are then matched against a prefix tree containing information about log grammar templates to identify candidate grammar templates that are similar to the logs to be parsed. This identifies all possible log grammar templates that could correspond to the logs to be parsed. Based on a preset matching strategy, the log segmentation sequence is matched against the candidate grammar templates to determine whether there are candidate grammar templates that match the logs to be parsed and meet preset matching criteria. If no candidate grammar templates meet the preset matching criteria, the logs to be parsed are parsed using a large language model to obtain a log parsing result. This helps understand the semantics of the logs to be parsed and achieves more accurate log parsing. Furthermore, since the large language model is only used when no candidate grammar templates meet the matching criteria, the cost of using the large language model is reduced, resulting in a lower cost for the cloud server to parse the logs of the cloud system. See Figure 2, which is a flowchart of a log parsing method provided according to an embodiment of the present disclosure, specifically including the following steps. Step 202: Determine the log to be parsed and perform word segmentation on the log to be parsed to obtain a log word segmentation sequence. The log to be parsed can be understood as the log to be parsed, word segmentation can be understood as the process of separating the individual words and variables contained in the log to be parsed, and the log word segmentation sequence can be understood as a word segmentation sequence obtained by segmenting the individual words, variables, and other elements in the log to be parsed, resulting in a consistent order of the individual words and variables in the log to be parsed. In practical applications, logs generated during system operation may contain invalid or interfering information that is meaningless or disruptive to log parsing. To ensure the accuracy of log parsing, preliminary processing of the original log to be parsed generated by the system is required. The specific implementation is as follows: Determine the log to be parsed, perform word segmentation on the log to be parsed, and obtain a log word segmentation sequence, including: determining an initial log to be parsed, extracting log content from the initial log to be parsed according to a preset regular expression, and determining the log to be parsed; and performing word segmentation on the log to be parsed according to a preset segmentation method to obtain a log word segmentation sequence.The initial log to be parsed can be understood as the original log generated during system operation and to be parsed. The preset regular expression can be understood as a regular expression used to extract the template portion and the variable portion of the log. In specific implementation, the preset regular expression can be set as required. The preset segmentation method can be understood as a preset method for segmenting the log to be parsed using preset delimiters. The preset delimiters include but are not limited to spaces, commas, semicolons, and tabs. Preferably, the preset segmentation method can be understood as a method for segmenting each word and each variable in the log to be parsed using spaces. This facilitates the subsequent semantic analysis of the log to be parsed using the large language model. The large language model is then used to reverse the reversed log to be parsed, and the word segmentation method of the large language model is then used to segment and perform semantic analysis on the reversed log to be parsed, thereby increasing the accuracy of the semantic analysis of the log to be parsed by the large language model. For example, a log to be parsed is "Receive 2". ?,, after being segmented by spaces, it becomes "Receive" and "2", or a log to be parsed "Send to node 4", after being segmented by spaces, it becomes "Send", "t.", "node", and "4". The log parsing method provided by the disclosed embodiments extracts log content using a preset regular expression and identifies the extracted log content as the log to be parsed. This eliminates interference information in the initial log to be parsed, ensuring the accuracy of the log to be parsed. The log to be parsed is then segmented according to the preset segmentation method to obtain a log segmentation sequence, providing a prerequisite for subsequent matching with a prefix tree. Step 204: Match the log segmentation sequence with the nodes in the prefix tree to determine the candidate grammar template corresponding to the log segmentation sequence. The prefix tree is constructed using nodes and paths, where the nodes are the segments in the log grammar template, and the paths are the associations between the segments in the log grammar template. The log grammar template is determined based on the initial log. Specifically, the prefix tree can be understood as a prefix parse tree. In specific implementations, the log word sequence of the log to be parsed is matched with a prefix tree to obtain all paths in the prefix tree that match the log word sequence, thereby determining the log grammar templates corresponding to all paths. In practical applications, a prefix tree is not yet available when parsing the first log. Therefore, a prefix tree can be constructed beforehand. The specific implementation is as follows: Before matching the log word sequence with nodes in the prefix tree and determining candidate grammar templates corresponding to the log word sequence, the process further includes: determining an initial log and performing word segmentation processing on the initial log to obtain an initial log word sequence; inputting the initial log into a large language model to obtain a log semantic template corresponding to the initial log; determining a log grammar template corresponding to the log semantic template based on the initial log word sequence; and constructing a prefix tree using the words in the log grammar template as nodes and the associations between the words in the log grammar template as paths. The initial log can be understood as the first log in the log parsing process. The log semantic template can be understood as the log template containing the semantic information of a log obtained by semantically analyzing the log using a large language model. The log syntax template can be understood as a log template that corresponds to the word segmentation sequence of the corresponding log, and each non-variable element corresponds to the same log word segmentation sequence.For example, the first log entry is "Receive from node 4," and the corresponding initial log word segmentation sequence is "Receive," "from," "node," and "4." Based on this log grouping sequence, a prefix tree containing the nodes "Receive," "from," and "node," and the path "Receive-from-node-4" can be established. The log parsing method provided by the disclosed embodiment uses a large language model in advance to construct a prefix tree based on the initial log word segmentation sequence, thereby ensuring the correspondence between each node and path in the prefix tree and the log, thereby ensuring the accuracy of log template matching based on the prefix tree. In actual applications, in order to reduce the total number of log templates, logs can be clustered and log groups can be established to facilitate subsequent statistics. The specific implementation method is as follows: After constructing the prefix tree, the method further includes: establishing a corresponding log group for the initial log, storing the log semantic template and the log syntax template corresponding to the log semantic template in a template pool, and establishing an association relationship between the log group and the log semantic template. Log groups can be understood as clustered groups of logs. Logs within a log group belong to the same type of log. The template pool can be understood as a storage pool containing the log semantic templates corresponding to each log group and the log grammar templates corresponding to each log semantic template. Specifically, each log group corresponds to a log semantic template, and all logs within a log group share the same log semantic template. A log semantic template can correspond to one or more log grammar templates. One or more logs within a log group correspond to the one or more log grammar templates corresponding to the log group. All log semantic templates corresponding to all log groups, and all log semantic templates can correspond to all log grammar templates, constituting the template pool. The log parsing method provided by the embodiments of the present disclosure ensures the accuracy of subsequent log clustering by establishing corresponding log groups for initial logs, storing the corresponding log semantic templates and log grammatical templates in a template pool, and associating the log groups with the log semantic templates and log grammatical templates. A prefix tree is constructed based on the initial log word segmentation sequence, ensuring the correspondence between each node and path in the prefix tree and the log, thereby ensuring the accuracy of log template matching based on the prefix tree. This method achieves fully automatic log template matching and log clustering, ensuring efficient log parsing.In practical applications, after constructing a prefix tree, the log word segmentation sequence can be matched with the nodes and paths in the prefix tree to determine the candidate grammatical template. The specific implementation method is as follows: Matching the log word segmentation sequence with the nodes in the prefix tree to determine the candidate grammatical template corresponding to the log word segmentation sequence includes: sequentially matching each log word in the log word segmentation sequence with the nodes in the prefix tree to determine the target node, wherein the target node is any one or more nodes in the prefix tree; and determining the candidate grammatical template corresponding to the log word segmentation sequence according to the path between the root node in the prefix tree and the target node. The root node in the prefix tree can be understood as the first node where matching starts. The word corresponding to the root node is usually a space, indicating the start. Usually, the root node does not perform the actual matching process, that is, the first word in the log word sequence is matched with the node next to the root node; the candidate grammar template can be understood as all possible log grammar templates that match the log word sequence, and the target node can be understood as all nodes passed by the path when all words in the log word sequence are matched or the node position where the next step of matching cannot be performed in the prefix tree. Specifically, a prefix tree consists of nodes and paths. Each node represents a word segment. Nodes include the root node, non-leaf nodes, and leaf nodes. A string of words formed by the path from the root node to a leaf node can correspond to a log syntax template in the template pool. Similarly, a string of words formed by the path from the root node to a non-leaf node can correspond to a log syntax template in the template pool. Since a log semantic template corresponds to multiple log syntax templates, and a log group corresponds to a log semantic template, a log group can correspond to multiple paths in the prefix parse tree. The log parsing method provided by the disclosed embodiments significantly improves the efficiency of determining candidate syntax templates by matching nodes of a log analysis sequence with a prefix tree, while ensuring the accuracy of the match with the log syntax templates in the template pool. This efficiency is achieved through the efficient search mechanism of the prefix tree itself.In practical applications, in order to match the subsequently determined path with the log word segmentation sequence, the matching can be performed strictly in order. The specific implementation method is as follows: sequentially matching each log word in the log word segmentation sequence with the nodes in the prefix tree to determine the target node includes: matching each log word with the nodes in the prefix tree according to the order of each log word in the log word segmentation sequence and the contextual relationship between the nodes in the prefix tree; determining the node in the prefix tree that matches each log word segmentation as the target node; accordingly, in the case where there are multiple target nodes, determining the candidate grammatical template corresponding to the log word segmentation sequence according to the path between the root node and the target node in the prefix tree includes: determining the word corresponding to each path according to the path between the root node and each target node in the prefix tree, and determining the candidate grammatical template corresponding to the log word segmentation sequence according to the word corresponding to each path. The order of the log words in the log word sequence can be understood as the order of their positions in the log word sequence, and the contextual relationship between nodes in the prefix tree can be understood as the contextual relationship between the positions represented by a node and the positions represented by other nodes in the prefix tree. In specific implementation, each log word in the log word sequence is matched sequentially with the nodes in the prefix tree, and all matching nodes are determined as target nodes. After multiple target nodes have been determined, the words strung together by the paths between the root node and each target node in the prefix tree can be determined as candidate grammar templates corresponding to the log word sequence. Specifically, the prefix tree contains nodes and paths. Each node represents a word segment. Nodes include the root node, non-leaf nodes, and leaf nodes. A string of words strung together along the path from the root node to a leaf node can correspond to a log syntax template in the template pool. A string of words strung together along the path from the root node to a non-leaf node can also correspond to a log syntax template in the template pool. Since a log semantic template corresponds to multiple log syntax templates, and a log group corresponds to a log semantic template, a log group can correspond to multiple paths in the prefix parse tree.For example, the root node of a prefix tree is "Root", the first-level nodes, that is, the child nodes of the root node are "Receive", "Send", and "<*>", the child node corresponding to "Receive" is "<*>", and the child node corresponding to "Send" is "<*>". Then, a log to be parsed "Receive from node 4" is matched with the prefix tree, and two candidate grammatical templates "Receive." and "<*>" can be obtained, among which "Receive" and "Send" are non-leaf nodes, and "<*>" is a leaf node. For another example, the root node of a prefix tree is "Root", the first-level node, that is, the child nodes of the root node are "Receive", "Send", and "<*>", the child nodes corresponding to "Receive" are "from", and "<*>", the child node corresponding to "Send" is "<*>", and the child node corresponding to the child node "from" of "Receive" is "<*>", then, a log to be parsed "Receive from node 4" is matched with the prefix tree, and two candidate grammatical templates "Receive from <*>", "Receive <*>", and "<*>" can be obtained. The log parsing method provided by the embodiments of the present disclosure sequentially matches the log word segmentation sequence with each node in the prefix tree, ensuring that the subsequently determined path matches the log word segmentation sequence, thereby improving the accuracy of matching the log word segmentation sequence with the log grammar templates in the template pool. Step 206: Match the log word segmentation sequence with the candidate grammar templates according to a preset matching strategy to obtain a matching result. The preset matching strategy can be understood as a preset strategy for performing quantitative and alignment matching between the log word segmentation sequence and each candidate grammar template. Matching results include, but are not limited to, loose match, strict match, and mismatch.In practical applications, quantity matching can be performed first, followed by alignment matching. The specific implementation methods for determining whether quantity matching is satisfied or alignment matching is not satisfied are as follows: The candidate grammar templates include multiple; accordingly, matching the log word segmentation sequence with the candidate grammar templates according to a preset matching strategy to obtain a matching result includes: determining the number of first words in the log word segmentation sequence and the number of second words in the word segmentation in each candidate grammar template; determining the candidate grammar template with the same number of second words as the first word as the candidate grammar template; aligning the log word segmentation with the word segmentation in the candidate grammar template; and obtaining a first matching result if it is determined that the aligned log word segmentation does not match the word segmentation in the candidate grammar template. The first word number can be understood as the number of word segmentation in the log word segmentation sequence, the second word number in each candidate grammar template can be understood as the number of word segmentation in the candidate grammar template corresponding to each candidate grammar template, and the first matching result can be understood as a loose match. Candidate grammar templates with inconsistent word counts are then eliminated, leaving the remaining candidate grammar templates as candidate grammar templates. Alignment matching is then performed. First, the log word segments in the log word segmentation sequence are aligned sequentially with the word segments in the candidate grammar template. The matching is then determined to determine whether the word segments at corresponding positions are consistent. If no candidate grammar template has a one-to-one correspondence between the word segments, the matching result is determined to be a first matching result, i.e., a loose match. The log parsing method provided in the disclosed embodiments obtains a first matching result, i.e., a loose match, when the number of word segments is consistent but the aligned log word segments do not match the word segments in the candidate grammar templates during the matching process between the log word segmentation sequence and the candidate grammar templates. This improves the template matching error tolerance and ensures more accurate subsequent log parsing of the log to be parsed. In actual applications, the specific implementation method of satisfying quantity matching and alignment matching is as follows: after aligning the log word segmentation with the word segmentation in the to-be-selected grammar template, it also includes: when it is determined that the aligned log word segmentation matches the word segmentation in the to-be-selected grammar template, obtaining a second matching result; specifically, following the description of the above implementation method, if there is a to-be-selected grammar template with consistent word segmentation, then it can be determined that the matching result is the second matching result, that is, a strict match, then the to-be-selected grammar template with consistent word segmentation is the log grammar template corresponding to the log to be parsed.The log parsing method provided by the disclosed embodiments obtains a second matching result (i.e., a strict match) during the matching process between a log word segmentation sequence and each candidate grammar template. If the number of segmentations is consistent and the aligned log word segments also match the segmentations in each candidate grammar template, the method directly determines the corresponding log grammar template without requiring the use of a large language model for subsequent operations, thereby improving the overall processing efficiency. In practical applications, if the second matching result satisfies a preset matching condition, the log group can be directly determined. The specific implementation method is as follows: After determining that the aligned log word segments match the segmentations in the candidate grammar template, the method further includes: if the second matching result satisfies the preset matching condition, determining the log group corresponding to the candidate grammar template as the log group corresponding to the log to be parsed. Specifically, if, during the process of determining the matching between the log word segmentation sequence and each candidate grammar template, the number of segmentations is consistent and the aligned log word segmentations also match the segmentations in each candidate grammar template, this indicates that the log grammar template of the log to be parsed exists in the template pool. Therefore, the log group corresponding to the matching candidate grammar template can be determined as the log group corresponding to the log to be parsed. The log parsing method provided in the disclosed embodiments maps the log to the corresponding log group when the second match result, i.e., a strict match, is determined. This ensures log parsing accuracy while improving log parsing efficiency, optimizing resource consumption, and reducing the cost of the entire log parsing process. In practical applications, the specific implementation method for not meeting the quantity match is as follows: After determining the first number of log word segments in the log word segmentation sequence and the second number of segmentations in each candidate grammar template, the method further includes: obtaining a third matching result when the first number of words and the second numbers of words are different. Specifically, following the description of the above implementation, when the number of segmented words in all candidate grammar templates is different from the number of segmented words included in the log segmented word sequence, a third matching result can be obtained, which can be understood as a mismatch.For example, a log word segmentation sequence is "End", and the candidate grammar templates are ["Send", "to", "node", "<*>"], ["Send", "<*>"], ["start", "prefetching", <*>"], the number of words in the log word segmentation sequence is 1, and the number of words in each candidate grammar template is not 1, that is, there is no candidate grammar template with the same number of words as the log word segmentation sequence. In this case, a third matching result of "no match" is obtained. The log parsing method provided by the embodiment of the present disclosure directly obtains the second matching result, i.e., no match, when matching the log word segmentation sequence with each candidate grammar template in the case of inconsistency in the number of words, without the need for subsequent matching with the prefix tree, thereby further improving the processing efficiency of the entire process. Step 208: When the matching result does not meet the preset matching condition, the log to be parsed is parsed using the large language model to obtain a parsing result. Among them, The preset matching condition can be understood as the existence of a strictly matching candidate grammatical template. If the matching result indicates that there is no strictly matching candidate grammatical template, it is necessary to call a large language model to parse the log. The specific implementation method is as follows: If the matching result does not meet the preset matching condition, the log to be parsed is parsed using the large language model to obtain a parsing result, including: if it is determined that the first matching result or the third matching result does not meet the preset matching condition, the log to be parsed is processed using the large prediction model to obtain a predicted semantic template; based on the relationship between the predicted semantic template and the log semantic template in the template pool, the log group corresponding to the log to be parsed is determined, wherein the template pool includes the log grammatical template and the log semantic template corresponding to the log grammatical template, and the log semantic template has an associated log group. The relationship between the predicted semantic template and the log semantic template includes a match or a mismatch. A match can be understood as the predicted semantic template and the log semantic template are completely consistent, and a mismatch can be understood as the predicted semantic template and the log semantic template have inconsistent words or sequences. Then, based on the match or mismatch relationship between the predicted semantic template and the log semantic template, That is, the log grouping corresponding to the log to be parsed can be determined according to different situations.The log parsing method provided by the embodiments of the present disclosure uses a large language model to predict a predicted semantic template corresponding to the log grouping sequence when the first matching result (i.e., a matching result of a candidate grammatical template that loosely matches the log word segmentation sequence) and the third matching result (i.e., a matching result of a candidate grammatical template that strictly matches the log word segmentation sequence) are obtained. The predicted semantic template is then matched with log semantic templates in a template pool. The accurate semantic analysis of the large language model improves intelligent analysis of the logs to be parsed, thereby increasing the accuracy of template extraction. Furthermore, this method reduces the occurrence of logs of the same category being unable to be clustered into the same log grouping due to different description methods, thereby improving the applicability and accuracy of log parsing across different systems and log formats. In practical applications, to reduce the resource consumption and labor costs associated with large language model training, large language models can be implemented through improved prompts. The specific implementation is as follows: Using the large language model to parse the log to be parsed includes: determining a log parsing task and a log query question based on the log to be parsed; determining a preset number of example semantic templates from the template pool based on the feature vector of the log to be parsed and the feature vectors of log semantic templates in the template pool, wherein the example semantic templates are a subset of the log semantic templates; and parsing the log to be parsed using the large language model based on the log parsing task, the log query question, and the example semantic templates. The log parsing task can be understood as describing the task to be performed by the large language model; the log query question can be understood as a question that needs to be directly answered by the large language model; the feature vector of the log to be parsed can be understood as a vector describing the log to be parsed and encoded by the large language model; the feature vector of the log semantic template can be understood as a vector describing the log semantic template and encoded by the large language model; and the example semantic template can be understood as a preset number of log semantic templates selected from the log semantic templates.Then, based on the log to be parsed, the log parsing task and log query problem corresponding to the log to be parsed can be determined. For example, the log to be parsed is "Receive-from-node-4". Then, based on the log to be parsed, the log parsing task can be determined as "Please extract the log template", and the log query problem can be determined as "Please extract the log template for the following log: Receive-from-node-4". For another example, the log to be parsed is "Send to node 2". Then, based on the log to be parsed, the log parsing task can be determined as "Please execute the log template extraction task" and the log query problem can be determined as "Please extract the log template for the following log: Receive-from-node-4". Determining a preset number of example semantic templates from the template pool based on the feature vector of the log to be parsed and the feature vectors of the log semantic templates in the template pool can be understood as calculating the cosine similarity between the vector description (i.e., feature vector) describing the log to be parsed and encoded by the large language model and the vector description (i.e., feature vector) describing each log semantic template in the template pool and encoded by the large language model, sorting the log semantic templates based on the cosine similarity, and selecting the top k log semantic templates as example semantic templates. k can be understood as the preset number mentioned above. In actual applications, the value of k can be determined based on actual needs. In specific implementations, identifying and classifying variables in logs helps improve the accuracy of log parsing. Furthermore, classifying these variables is also beneficial for subsequent data analysis tasks. Therefore, in the disclosed embodiments, a question text is constructed for the large language model, so that the prompt text can achieve the following: enabling the large language model to identify variables and classify them into one of the ten outlined categories (i.e., the prompt text can help the large language model understand which part of the log should be a variable and the logical reasoning behind the classification of these variables), as well as perform template extraction.Specifically, the prompt text can be designed as variable-aware prompt text. Variable-aware prompts contain three elements: an instruction (i.e., the log parsing task described above): used to describe the task to be performed by the large language model; a demonstration (i.e., the example semantic template described above): a set of examples, each of which is a question and a true answer; and a query (i.e., the log query question described above): a question that the large language model needs to answer directly. Examples can be constructed from a template pool. For instructions that call the large language model to extract templates, multiple examples (e.g., three) can be directly constructed from the template pool based on the log semantic template. Alternatively, for instructions that identify and classify variables in logs, multiple examples (e.g., ten) can be added to the multiple examples constructed based on the log semantic template. Each example represents a different type of log semantic template, corresponding to a variable type, and serves as a seed example. To obtain the seed examples, the cosine similarity between the vector description of the log to be parsed in the large language model and the vector description of each log semantic template in the template pool can be calculated. The log semantic templates are then sorted based on their cosine similarity. The first k log semantic templates are selected as k seed examples. Furthermore, after using a large language model to perform semantic analysis on the log word segmentation sequence and extract log semantic templates, the extracted log semantic templates will expand the template pool. Furthermore, after obtaining a preset number of seed examples, the seed examples are labeled with answers, and the seed examples and corresponding answers are used as demonstrations. The log parsing task, log query question, and demonstration are then input into the large language model to implement log parsing of the log to be parsed. The log parsing method provided in the disclosed embodiments reduces the cost of using a large language model by eliminating the need to train the large language model. Furthermore, by using the log parsing task, log query question, and example semantic templates as prompts for the large language model, the large language model enhances variable recognition and classification capabilities beyond its existing template extraction capabilities, strengthening its task understanding capabilities and thereby improving the accuracy of template extraction using the large language model. Furthermore, this prompting method also facilitates subsequent data analysis.In actual applications, when a log semantic template that matches a predicted semantic template exists, the specific implementation method for determining the log group corresponding to the log to be parsed is as follows: The log semantic templates include multiple; accordingly, determining the log group corresponding to the log to be parsed based on the relationship between the predicted semantic template and the log semantic templates in the template pool includes: matching the predicted semantic template with each log semantic template in the template pool; when a log semantic template matching the predicted semantic template is determined to exist in the template pool, using the log semantic template matching the predicted semantic template as the target log semantic template; and determining the log group corresponding to the log to be parsed based on the target log semantic template. The log semantic template that matches the predicted semantic template can be understood as a log semantic template that has the same number of segmentations as the predicted semantic template and the same segmentations corresponding to each position after segmentation alignment. In a specific implementation, after the predicted semantic template corresponding to the log to be parsed is determined using the large language model, the log to be parsed is matched with each log semantic template in the template pool to find a log semantic template that matches the predicted semantic template. For example, as shown in Table 1, the template pool contains three log semantic templates, and the three log semantic templates correspond to three log groups. The log group sequence is "Accept from node 4." None of the aforementioned log syntax templates match this log group sequence. Using a large language model, semantic template extraction is performed on this log group sequence, resulting in the log semantic template "Received from a certain node." This log semantic template is consistent with the second log semantic template "Received from a certain node" in the template pool. Therefore, the log to be parsed can be determined to correspond to log group 2. The log parsing method provided by the disclosed embodiments matches the predicted semantic template with the log semantic template. If a match exists, the log group corresponding to the matched log semantic template is identified as the log group to be parsed, indicating that the predicted semantic template and the log semantic template have the same semantics, but with a different description. This makes log clustering more intelligent and improves the accuracy of template extraction and log parsing. In actual applications, after the log grouping is determined, the template pool and the prefix tree can also be updated to ensure the accuracy of the next log parsing (or the log parsing of the next log). The specific implementation method is as follows: After determining the log grouping corresponding to the log to be parsed, it also includes: mapping the log to be parsed to the log grouping; determining the newly added semantic template corresponding to the log to be parsed based on the predicted semantic template; storing the newly added semantic template as a newly added syntax template in the template pool, and associating the newly added syntax template with the target log semantic template; using the participles contained in the newly added syntax template as update nodes, using the association relationship of the participles in the newly added syntax template as an update path, and updating the prefix tree according to the update node and the update path. Among them, the updated node can be understood as a newly added node in the prefix tree, and the updated path can be understood as a newly added path corresponding to the contextual relationship between the word segments in the newly added grammar template. Using the above example, for example, a newly added grammar template is "Send to point 5", then the newly added nodes are send, to, point, <*>. Since the prefix tree already has two nodes, send and to, then based on the contextual relationship between to and point, a new node is added after to in the prefix tree. The node corresponds to the word segment point. Then, based on the contextual relationship between point and <*>, another node <*> is added after point, thereby implementing the update of the prefix tree.The log parsing method provided by the disclosed embodiments ensures accurate log parsing of the next log entry by mapping the log entry to the determined log grouping after determining the log grouping, updating the log syntax template in the template pool, and updating the prefix tree based on the updated template pool. If no log semantic template matches the predicted semantic template, the method can first determine whether any log syntax template in the template pool loosely matches the log entry when matching the log entry to be parsed with the log syntax templates in the template pool. If no log syntax template loosely matches the log entry to be parsed, the method for determining the log grouping corresponding to the log entry to be parsed is specifically implemented as follows: After matching the predicted semantic template with each log semantic template in the template pool, the method further includes: if it is determined that no log semantic template matching the predicted semantic template exists in the template pool, using the predicted semantic template as a new log semantic template; and determining the log grouping corresponding to the log entry to be parsed based on the new log semantic template. Specifically, if no log semantic template matching the predicted semantic template exists in the template pool, the predicted semantic template is a new log semantic template. To prevent mismatches caused by semantically identical but different expressions, the predicted log semantic template can first be compared with the log semantic template corresponding to the log grammatical template determined above and loosely matching the log word segmentation sequence through a large language model for semantic similarity identification. If the large language model determines that the predicted log semantic template is semantically different from the log semantic template corresponding to the log grammatical template determined above and loosely matching the log word segmentation sequence, the predicted semantic template is used as a new log semantic template, and a corresponding log group is created based on the predicted semantic template. This newly created log group corresponding to the predicted semantic template is then determined as the log group for the log to be parsed. The log parsing method provided by the embodiments of the present disclosure improves log parsing accuracy by matching predicted semantic templates with log semantic templates. If no match exists, the logs corresponding to the matched log semantic template are grouped, indicating that no log semantic template with the same semantics exists in the template pool. A further semantic recognition comparison is performed using a large language model, thereby improving log parsing accuracy. If the large language model yields a semantically different result, the predicted semantic template is used as a new log semantic template, thereby determining a new log group for the log to be parsed. This ensures clustering accuracy for the newly classified logs and further improves log parsing accuracy.If no log syntax template loosely matches the log to be parsed, that is, if all log syntax templates do not match the log to be parsed, the specific implementation method for determining the log group corresponding to the log to be parsed is as follows: Determining the log group corresponding to the log to be parsed based on the relationship between the predicted semantic template and the log semantic templates in the template pool includes: if it is determined that no log semantic template matching the predicted semantic template exists in the template pool, determining a new log semantic template based on the predicted semantic template and the log semantic template corresponding to the candidate syntax template; and determining the log group corresponding to the log to be parsed based on the new log semantic template. Specifically, the matching process can be seen in the implementation described above. After matching, if the large language model determines that the predicted log semantic template is semantically identical to the log semantic template corresponding to the log syntax template loosely matching the log word segmentation sequence, then the predicted semantic template is a variant of the log semantic template in the template pool. Then, the predicted semantic template is universally merged with the log semantic template, and the merged log semantic template is used as a new log semantic template. The log group corresponding to the log semantic template in the template pool is determined as the log group for the log to be parsed. The log parsing method provided in the disclosed embodiments matches the predicted semantic template with the log semantic template. If no match exists, the log group corresponding to the matched log semantic template is assigned, indicating that there is no log semantic template with the same semantics as the current predicted semantic template in the template pool. Further semantic recognition and comparison are performed using a large language model to improve log parsing accuracy. If the large language model yields the same semantic result, the newly added log semantic template is used to update the log semantic template with the same semantics. The log group corresponding to the log semantic template with the same semantics is then assigned as the log group for the log to be parsed. This increases the log clustering fault tolerance and further improves log parsing accuracy.In actual applications, when there is no log semantic template that matches the predicted semantic template, the template pool and prefix tree can also be updated to facilitate accurate parsing of subsequent logs. The specific implementation method is as follows: After determining the log group corresponding to the log to be parsed, the method further includes: mapping the log to be parsed to the log group; determining the newly added grammatical template corresponding to the log to be parsed based on the log word sequence; storing the newly added log semantic template and the newly added grammatical template in the template pool, and associating the newly added grammatical template with the newly added log semantic template; using the word segments contained in the newly added grammatical template as the update node, using the association relationship of the word segments in the newly added grammatical template as the update path, and updating the prefix tree according to the update node and the update path. Specifically, after determining the log group corresponding to the log to be parsed as described above, the log to be parsed is mapped to the corresponding log group. To ensure the accuracy of the next log parsing, the prefix tree and template pool can be updated. In specific implementation, the log syntax template corresponding to the log word sequence is first determined based on each word and its corresponding position in the log word sequence (words are represented by the word itself, and variables are represented by <*>). Then, the template pool can be updated based on the newly determined log semantic template and log syntax template. It should be noted that after using the large language model to perform semantic similarity identification between the predicted log semantic template and the log semantic template corresponding to the log grammatical template determined to loosely match the log word segmentation sequence, the prefix tree can be updated using the new log semantic template and the log word segmentation sequence, regardless of whether a log grammatical template loosely matches the log to be parsed exists. The difference is that if the new log semantic template is implemented by updating the original log semantic template, then the log grammatical template corresponding to the original log semantic template and the new grammatical template determined by the log grouping sequence will correspond to the new log semantic template, and the new log semantic template will correspond to the log group corresponding to the original log semantic template. If the new log semantic template is implemented by adding a new log semantic template, then only the new grammatical template determined by the log grouping sequence needs to be associated with the new log semantic template, and a new log group needs to be added to correspond to the new log semantic template. Then, after the template pool is updated, the prefix tree can be updated by applying the newly added log syntax template in the template pool according to the segmentation and the relationship between segmentations. For the specific implementation of the prefix tree update, please refer to the above embodiment of the specification.The log parsing method provided by the embodiments of the present disclosure ensures the accuracy of log parsing for the next log by mapping the log to the determined log group after the log group is determined, updating the log semantic template and log syntax template in the template pool, and updating the prefix tree based on the updated template pool. The log parsing method provided by the disclosed embodiments first segments the log to be parsed to obtain a log word sequence. Matching between the word segments improves log parsing efficiency. The log to be parsed is then matched against a prefix tree containing information about log grammar templates to identify candidate grammar templates that are similar to the log to be parsed. This method identifies all possible log grammar templates that could correspond to the log to be parsed. Based on a preset matching strategy, the log word sequence is matched against the candidate grammar templates to determine whether there are candidate grammar templates that match the log to be parsed and meet preset matching conditions. If no candidate grammar templates meet the preset matching conditions, the log to be parsed is parsed using a large language model to understand the semantics of the log to be parsed, achieving more accurate log parsing. Furthermore, since the large language model is only used when no candidate grammar templates that meet the matching conditions are found, the cost of using the large language model is reduced, thereby lowering the log parsing cost of this solution. The log parsing method is further described below with reference to FIG3 . Referring to Figure 3, FIG3 illustrates an example data structure of a log parsing method provided according to an embodiment of the present disclosure. As shown in FIG3, FIG3 includes three main data structures: log group, template pool, and prefix parse tree. There can be one or more log groups. All logs in the same log group share the same log semantic template (for example, log group 1 shares "Receive <*>" and log group n shares "Send <*>" in FIG3). A log group records the IDs (identity documents, used to uniquely identify blocks) of all included logs, as well as vector descriptions of the logs in the log group, encoded by a large language model.Each log group has its own log semantic template extracted by a large language model and one or more log grammatical templates generated by statistical word frequency analysis. These templates assist in traversing a new log entry through the prefix tree and matching the log entry to the log grammatical template. After segmenting the log grammatical template with the original log entry, the resulting log grammatical templates are aligned to distinguish the static log grammatical template from the log variable portion. The log semantic template places one or more words in a log entry under a variable placeholder. For example, the log entry "start prefetching・"O" corresponds to the log grammatical template ["start", "prefetching ••• <*>"]. All log grammatical templates are cataloged into a dictionary, with the number of words in the log grammatical template as the key and the corresponding value as the template list of the corresponding log grammatical template. The template pool consists of semantic templates (i.e., log semantic templates) and grammatical templates (i.e., log grammatical templates). One semantic template corresponds to multiple grammatical templates derived by statistical word frequency analysis, while one log group corresponds to one grammatical template. Therefore, the semantic templates and syntactic templates in the template pool are associated with log groups. Specifically, each log group corresponds to a template cluster in the template pool. This template cluster contains a log semantic template corresponding to the log group and multiple log syntactic templates corresponding to the log semantic template. The template pool consists of multiple template clusters corresponding to multiple log groups. For example, as shown in Figure 3, the template cluster corresponding to log group 1 contains a log semantic template "Receive <*>" and multiple log syntactic templates {3,[['Receive', '<*>', 'KB'],['Receive', '<*>', 'BYTES']]}. oThe prefix parse tree consists of nodes and paths. Each node represents a word segment. Nodes include the root node (such as Root in Figure 3), non-leaf nodes (such as Send and Receive in Figure 3), and leaf nodes (such as <*> in Figure 3). A string of words along the path from the root node to a leaf node can correspond to a log syntax template in the template pool. Similarly, a string of words along the path from the root node to a non-leaf node can also correspond to a log syntax template in the template pool. Since a log semantic template corresponds to multiple log syntax templates, and a log group corresponds to a log semantic template, a log group can correspond to multiple paths in the prefix parse tree. In specific implementation, the log parsing method includes the following steps:

[0002] (1) Preprocessing: After a server receives a new log, it needs to perform log content extraction and word segmentation preprocessing. First, a basic regular expression is used to extract the log content portion (i.e., the log to be parsed) from the original log (i.e., the initial log to be parsed). Some feasible solutions involve extracting this content based on domain knowledge, such as using regular expressions to extract common variables such as IP addresses and block IDs. However, the log parsing method in the disclosed embodiment retains the entire original log message (i.e., the log content portion) to ensure that the subsequent large language model can grasp the complete context of the log. Then, the extracted log content portion is segmented (i.e., word segmentation) using spaces to obtain the segmented log (i.e., the log word segmentation sequence). When the large model is subsequently used to perform semantic analysis on the extracted log message, the large language model is used for reverse spelling, followed by word segmentation and semantic analysis using its own word segmentation method. This preprocessing method minimizes the need for domain knowledge while maintaining high log parsing efficiency.

[0003] (2) Cluster Matching and Tree Search After obtaining the segmented log, each word in the log is matched against the nodes in the prefix tree in the order of the words in the segmented log (i.e., the aforementioned segmented words). Subsequent word matching can only be continued after matching the previous word. When matching backward in the prefix tree, the next match only matches the child nodes of the current node. Matching is performed in this manner until all segmented words in the segmented log are matched, or no further matching position can be performed in the prefix tree. During the entire matching process, a path from the root node to each node is generated. This path corresponds to a log grammar template. All log grammar templates corresponding to all paths generated during the matching process are added to a list as a candidate list of log grammar templates (i.e., the candidate grammar template list). These candidate log grammar templates are then evaluated to determine the most suitable log grammar template for the new log and the corresponding log group. Some feasible evaluation methods use similarity metrics and specific thresholds related to the dataset to determine a match. Unlike these evaluation methods, the log parsing method in the disclosed embodiments categorizes matching results into three distinct categories: i) strict match, ii) loose match, and iii) mismatch. First, for each candidate log group's log grammar template, the number of words in the received log is compared with the number of words in the log grammar template for that log group. If the word counts do not match, it is immediately determined that there is no "strict match." Next, a loose match is determined by aligning the words in the log grammar template with the words in the segmented log. Words containing the "<*>" wildcard character in the log grammar template can match any word at the corresponding position in the log. For example, the log syntax template corresponding to "start prefetching", <*>" is ["start", "prefetching •••<*>"], which can achieve loose matching with a log that starts with the word "start" and has any word in the second position, a total of two words. Furthermore, after the loose matching, a regular expression is used to perform a stricter alignment match between the log after word segmentation and the elements other than "<*>" in the log syntax template. If all the words match after alignment, it means that the log after word segmentation is strictly matched with the currently matched log syntax template. It is worth noting that the matching process will stop when a strict match occurs.If a strict match occurs, the new log entry is directly mapped to the log group corresponding to the matching log syntax template. If a strict match does not occur, the large language model is used to extract the template and then update the corresponding data structure.

[0004] (3) Template Extraction and Parse Tree Update: In the absence of a strict match, the large language model is first used to perform semantic analysis on the segmented log to obtain a new log semantic template corresponding to the segmented log (i.e., the predicted semantic template). This new log semantic template is then matched against log semantic templates in the template pool to determine whether there exists a log semantic template in the template pool that is identical to the new log semantic template. If a log semantic template identical to the new log semantic template exists in the template pool, this indicates that the current log can be clustered with the log group corresponding to the matching log semantic template, but the log grammatical template corresponding to the new log semantic template is not comprehensive. Therefore, the new log is mapped to the log group corresponding to the matching log semantic template and added as an updated log grammatical template to the template cluster corresponding to the log group in the template pool. The prefix parse tree is then updated based on the updated template pool to ensure that the prefix parse tree contains a node path corresponding to the new log grammatical template. If the template pool doesn't contain a log semantic template identical to the new log semantic template, but the match is determined to be a loose match, the large language model is consulted again. The large language model determines whether the new log semantic template is a loose match with a particular log semantic template. If the large language model determines that a log semantic template exists that loosely matches the new log semantic template, a more general, merged log semantic template is generated to match the new log semantic template and each log syntax template in the log group. This log semantic template and its corresponding log syntax template are then updated and added to the template pool. If a log still fails to successfully match a log semantic template in the template pool after going through the entire process described above, a new log group is created corresponding to the log semantic template. This log semantic template is used as the log syntax template corresponding to the log semantic template, and the parse tree is updated accordingly. Optionally, the process of using a large language model to perform semantic analysis on the segmented log and obtain a new log semantic template corresponding to the segmented log can utilize a more powerful large language model, or fine-tune the large language model through supervised learning using labeled data to improve the accuracy of the semantic analysis of the large language model.Preferably, the process of using a large language model to perform semantic analysis on the segmented log and obtain a new log semantic template corresponding to the segmented log can be specifically implemented by introducing variable-aware prompts. In practical applications, identifying and classifying variables in logs helps improve accuracy. Classifying these variables not only improves the accuracy of template extraction but also facilitates subsequent data analysis tasks. Therefore, a method for questioning the large language model has been constructed. These prompts now serve a dual purpose: enabling the large language model to identify variables and classify them into one of the ten categories outlined in

[15] (this improved prompting method helps the large language model understand which parts of the log should be considered variables and the logical reasoning behind these variable classifications), and also enabling template extraction. Specifically, variable-aware prompts contain three elements: instructions: used to describe the task to be performed by the large language model; demonstrations: a set of examples, each of which is a question and a real answer; queries: questions that the large language model needs to answer directly. Examples can be constructed from a template pool. When calling a large language model for template extraction, multiple examples (for example, three) can be constructed from the template pool. Alternatively, for the identification and classification of variables in logs, ten more examples can be added to this basis. Each example represents a different type of log variable. These examples serve as seed examples. Subsequent template extraction results will expand the template pool. To obtain these template samples, the cosine similarity between the query log vector description in the large language model and the vector description of each template in the template pool can be calculated. The top k examples are then selected as k task examples, which serve as seed examples. The log parsing method provided by the embodiments of the present disclosure reduces the need for specialized knowledge for log parsing through minimal preprocessing, while ensuring log parsing efficiency. Furthermore, by utilizing a prefix parse tree to match log templates, log parsing efficiency is further improved. Furthermore, during the log parsing process, the large language model is invoked for log parsing only in cases of mismatch or loose match, reducing the cost of using the large language model and optimizing resource consumption. Furthermore, by invoking the large language model for semantic analysis during log parsing, log parsing accuracy is significantly improved. Furthermore, the large language model does not require pre-training; instead, it is utilized through examples, further reducing manual effort and increasing the versatility of the method across various fields and log formats.Corresponding to the above method embodiment, the present disclosure further provides an embodiment of a log parsing device. Referring to FIG. 4 , FIG. 4 is a schematic structural diagram of a log parsing device provided according to an embodiment of the present disclosure. As shown in FIG4 , the apparatus includes: a log word segmentation sequence obtaining module 402 configured to determine a log to be parsed and perform word segmentation processing on the log to be parsed to obtain a log word segmentation sequence; a candidate grammar template determining module 404 configured to match the log word segmentation sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log word segmentation sequence, wherein the prefix tree is constructed using nodes and paths, the nodes are words in a log grammar template, the paths are associations between words in the log grammar template, and the log grammar template is determined based on an initial log; a grammar template matching module 406 configured to match the log word segmentation sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result; and a log parsing module 408 configured to parse the log to be parsed using a large language model to obtain a parsing result if the matching result does not meet a preset matching condition. Optionally, the apparatus further includes a prefix tree construction module configured to: determine an initial log and perform word segmentation processing on the initial log to obtain an initial log word segmentation sequence; input the initial log into a large language model to obtain a log semantic template corresponding to the initial log; determine a log grammar template corresponding to the log semantic template based on the initial log word segmentation sequence; and construct a prefix tree using the word segments in the log grammar template as nodes and the associations between the word segments in the log grammar template as paths. Optionally, the apparatus further includes an initial template storage module configured to: create a corresponding log group for the initial log, store the log semantic template and the log grammar template corresponding to the log semantic template in a template pool, and establish an association between the log group and the log semantic template. Optionally, the log word segmentation sequence obtaining module 402 is further configured to: determine an initial log to be parsed, extract log content in the initial log to be parsed according to a preset regular expression, and determine it as a log to be parsed; and perform word segmentation processing on the log to be parsed according to a preset segmentation method to obtain a log word segmentation sequence.Optionally, the candidate grammar template determination module 404 is further configured to: sequentially match each log word in the log word sequence with a node in a prefix tree to determine a target node, where the target node is any one or more nodes in the prefix tree; and determine a candidate grammar template corresponding to the log word sequence based on the path between the root node in the prefix tree and the target node. Optionally, the candidate grammar template determination module 404 is further configured to: match each log word with a node in the prefix tree according to the order of each log word in the log word sequence and the contextual relationship between nodes in the prefix tree; and determine a node in the prefix tree that matches each log word as the target node. Accordingly, if there are multiple target nodes, the candidate grammar template determination module 404 is further configured to: determine a word corresponding to each path based on the path between the root node in the prefix tree and each target node, and determine a candidate grammar template corresponding to the log word sequence based on the word corresponding to each path. Optionally, the candidate grammar templates include multiple ones; accordingly, the grammar template matching module 406 is further configured to: determine the number of first words in the log word segmentation contained in the log word segmentation sequence, and the number of second words in the word segmentation in each candidate grammar template; determine the candidate grammar template having the same number of second words as the first number as the candidate grammar template; align the log word segmentation with the word segmentation in the candidate grammar template; and obtain a first matching result if it is determined that the aligned log word segmentation does not match the word segmentation in the candidate grammar template. Optionally, the apparatus further includes a second grammar template matching module configured to: obtain a second matching result if it is determined that the aligned log word segmentation matches the word segmentation in the candidate grammar template; and determine the log group corresponding to the candidate grammar template as the log group corresponding to the log to be parsed if the second matching result satisfies a preset matching condition. Optionally, the apparatus further includes a third grammar template matching module configured to: obtain a third matching result if the first number of words and each second number of words are different.Optionally, the log parsing module 408 is further configured to: if it is determined that the first matching result or the third matching result does not meet the preset matching condition, process the log to be parsed using a large prediction model to obtain a predicted semantic template; determine a log group corresponding to the log to be parsed based on the relationship between the predicted semantic template and log semantic templates in a template pool, wherein the template pool includes the log syntax template and a log semantic template corresponding to the log syntax template, and the log semantic template has an associated log group. Optionally, the log semantic template includes multiple log semantic templates; accordingly, the log parsing module 408 is further configured to: match the predicted semantic template with each log semantic template in the template pool; if it is determined that a log semantic template matching the predicted semantic template exists in the template pool, use the log semantic template matching the predicted semantic template as a target log semantic template; and determine a log group corresponding to the log to be parsed based on the target log semantic template. Optionally, the apparatus further includes a first sub-log parsing module configured to: if it is determined that no log semantic template matching the predicted semantic template exists in the template pool, use the predicted semantic template as a new log semantic template; and determine a log group corresponding to the log to be parsed based on the new log semantic template. Optionally, the apparatus further includes a second sub-log parsing module configured to: if it is determined that no log semantic template matching the predicted semantic template exists in the template pool, determine a new log semantic template based on the log semantic template corresponding to the predicted semantic template and the candidate syntax template; and determine a log group corresponding to the log to be parsed based on the new log semantic template. Optionally, the device further includes a first prefix tree update module, which is configured to: map the log to be parsed to the log group; determine the newly added semantic template corresponding to the log to be parsed based on the predicted semantic template; store the newly added semantic template as a newly added syntax template in the template pool, and associate the newly added syntax template with the target log semantic template; use the segmentation contained in the newly added syntax template as an update node, use the association relationship of the segmentation in the newly added syntax template as an update path, and update the prefix tree according to the update node and the update path.Optionally, the device further includes a second prefix tree updating module configured to: map the log to be parsed to the log group; determine a new grammatical template corresponding to the log to be parsed based on the log word sequence; store the new log semantic template and the new grammatical template in the template pool, and associate the new grammatical template with the new log semantic template; use the word segments contained in the new grammatical template as update nodes, use the association relationship between the word segments in the new grammatical template as an update path, and update the prefix tree based on the update nodes and update path. Optionally, the device further includes a third sub-log parsing module configured to: determine a log parsing task and a log query question based on the log to be parsed; determine a preset number of example semantic templates from the template pool based on the feature vector of the log to be parsed and the feature vectors of the log semantic templates in the template pool, wherein the example semantic templates are a subset of the log semantic templates; and parse the log to be parsed using a large language model based on the log parsing task, the log query question, and the example semantic templates. The above is an exemplary solution of a log parsing device according to this embodiment. It should be noted that the technical solution of the log parsing device and the technical solution of the log parsing method described above share the same concept. For details not described in detail in the technical solution of the log parsing device, please refer to the description of the technical solution of the log parsing method described above. Figure 5 shows a block diagram of a computing device 500 according to one embodiment of the present disclosure. The components of computing device 500 include, but are not limited to, a memory 510 and a processor 520. Processor 520 is connected to memory 510 via a bus 530, and a database 550 is used to store data. Computing device 500 also includes an access device 540 that enables computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet.The access device 540 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a Near Field Communication (NFC). oIn one embodiment of the present disclosure, the aforementioned components of computing device 500, as well as other components not shown in FIG. 5 , may also be connected to one another, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 5 is for illustrative purposes only and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed. Computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 500 may also be a mobile or stationary server. Processor 520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the log parsing method described above. The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solution of the log parsing method described above. Any details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the log parsing method described above. An embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these computer-executable instructions implement the steps of the log parsing method described above. The above is a schematic diagram of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the log parsing method described above. Any details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the log parsing method described above. An embodiment of the present disclosure also provides a computer program product comprising a computer program / instructions. When executed by a processor, these computer program / instructions implement the steps of the log parsing method described above. The above is a schematic diagram of a computer program product of this embodiment. It should be noted that the technical solution of this computer program product is based on the same concept as the technical solution of the log parsing method described above. Any details not described in detail in the technical solution of the computer program product can be found in the description of the technical solution of the log parsing method described above. The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims.In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program code, which may be in source code form, object code form, executable files, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electrical carrier signals or telecommunication signals. It should be noted that, for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present disclosure are not limited by the order of the actions described, as certain steps may be performed in a different order or simultaneously, depending on the embodiments of the present disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules described are not necessarily required for the embodiments of the present disclosure. In the above embodiments, the description of each embodiment has its own emphasis. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. The preferred embodiments disclosed above are merely intended to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and variations are possible based on the content of the embodiments of the present disclosure. The present disclosure selects and describes these embodiments in detail to better explain the principles and practical applications of the embodiments of the present disclosure, thereby enabling those skilled in the art to better understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims

Claims 1. A log parsing method, comprising: Determine the log to be parsed, and perform word segmentation processing on the log to be parsed to obtain a log word segmentation sequence; Matching the log word segmentation sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log word segmentation sequence, wherein the prefix tree is constructed using nodes and paths, the nodes are words in the log grammar template, the paths are associations between words in the log grammar template, and the log grammar template is determined based on the initial log; matching the log word segmentation sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result; When the matching result does not meet the preset matching condition, the log to be parsed is parsed using the large language model to obtain a parsing result.

2. The log parsing method according to claim 1, further comprising: Determine an initial log, and perform word segmentation processing on the initial log to obtain an initial log word segmentation sequence; Inputting the initial log into a large language model to obtain a log semantic template corresponding to the initial log; determining a log grammatical template corresponding to the log semantic template based on the initial log word segmentation sequence; The word segments in the log grammar template are used as nodes, and the association relationship between the word segments in the log grammar template is used as a path to construct a prefix tree.

3. The log parsing method according to claim 1 or 2, further comprising, after constructing the prefix tree: A corresponding log group is created for the initial log, the log semantic template and the log syntax template corresponding to the log semantic template are stored in a template pool, and an association relationship is established between the log group and the log semantic template.

4. The log parsing method according to any one of claims 1 to 3, wherein determining the log to be parsed and performing word segmentation processing on the log to be parsed to obtain a log word segmentation sequence comprises: Determine an initial log to be parsed, extract log content in the initial log to be parsed according to a preset regular expression, and determine it as the log to be parsed; The log to be parsed is segmented according to a preset segmentation method to obtain a log segmentation sequence.

5. The log parsing method according to any one of claims 1 to 4, wherein matching the log word segmentation sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log word segmentation sequence comprises: Sequentially matching each log word in the log word segmentation sequence with a node in a prefix tree to determine a target node, where the target node is any one or more nodes in the prefix tree; and determining a candidate grammar template corresponding to the log word segmentation sequence based on a path between a root node in the prefix tree and the target node.

6. The log parsing method according to claim 5, wherein sequentially matching each log word in the log word sequence with a node in a prefix tree to determine a target node comprises: According to the order of each log segmentation in the log segmentation sequence and the contextual relationship between nodes in the prefix tree, each log segmentation is matched with a node in the prefix tree; a node in the prefix tree that matches each log segmentation is determined as a target node; accordingly, in the case where there are multiple target nodes, determining a candidate grammatical template corresponding to the log segmentation sequence based on a path between a root node and the target node in the prefix tree includes: determining a segmentation corresponding to each path between the root node and each target node in the prefix tree, and determining a candidate grammatical template corresponding to the log segmentation sequence based on the segmentation corresponding to each path.

7. The log parsing method according to any one of claims 1 to 6, wherein the candidate grammatical templates include a plurality of; Accordingly, the matching of the log word segmentation sequence with the candidate grammar template according to the preset matching strategy to obtain a matching result includes: Determining the number of first words in the log word segmentation sequence and the number of second words in each candidate grammar template; Determine a candidate grammar template having the same number of second words as the first words as a grammar template to be selected; align the log segmentation words with the segmentation words in the grammar template to be selected; When it is determined that the aligned log word segmentation does not match the word segmentation in the to-be-selected grammar template, a first matching result is obtained.

8. The log parsing method according to claim 7, further comprising: after aligning the log segmentation words with the segmentation words in the candidate grammar template; When it is determined that the aligned log word segmentation matches the word segmentation in the candidate grammar template, obtaining a second matching result; In a case where the second matching result satisfies a preset matching condition, the log group corresponding to the candidate grammar template is determined as the log group corresponding to the log to be parsed.

9. The log parsing method according to claim 7 or 8, further comprising: after determining the number of first words in the log word segmentation sequence and the number of second words in each candidate grammar template; When the number of the first words is different from the number of each second word, a third matching result is obtained.

10. The log parsing method according to claim 9, wherein when the matching result does not meet the preset matching condition, the large language model is used to parse the log to be parsed to obtain the parsing result, comprising: When it is determined that the first matching result or the third matching result does not meet the preset matching condition, the log to be parsed is processed using a large prediction model to obtain a prediction semantic template; and a log group corresponding to the log to be parsed is determined based on a relationship between the prediction semantic template and a log semantic template in a template pool, wherein the template pool includes the log syntax template and a log semantic template corresponding to the log syntax template, and the log semantic template has an associated log group.

11. The log parsing method according to claim 10, wherein the log semantic templates include a plurality; accordingly, determining the log group corresponding to the log to be parsed based on the relationship between the predicted semantic template and the log semantic templates in the template pool comprises: Matching the predicted semantic template with each log semantic template in the template pool; In a case where it is determined that a log semantic template matching the predicted semantic template exists in the template pool, taking the log semantic template matching the predicted semantic template as a target log semantic template; Determine the log group corresponding to the log to be parsed according to the target log semantic template.

12. The log parsing method according to claim 11, further comprising: after matching the predicted semantic template with each log semantic template in the template pool; In the case where it is determined that the log semantic template that matches the predicted semantic template does not exist in the template pool, using the predicted semantic template as a new log semantic template; Determine the log group corresponding to the log to be parsed according to the newly added log semantic template.

13. The log parsing method according to claim 10 or 11, wherein determining the log group corresponding to the log to be parsed based on the relationship between the predicted semantic template and the log semantic templates in the template pool comprises: If it is determined that no log semantic template matching the predicted semantic template exists in the template pool, a new log semantic template is determined based on the predicted semantic template and the log semantic template corresponding to the candidate syntax template; and a log group corresponding to the log to be parsed is determined based on the new log semantic template.

14. The log parsing method according to claim 11, wherein the log group corresponding to the log to be parsed is determined After that, it also includes: Mapping the log to be parsed to the log group; Determining a new syntax template corresponding to the log to be parsed according to the predicted semantic template; Storing the newly added syntax template in the template pool, and associating the newly added syntax template with the target log semantic template; The word segments included in the newly added grammar template are used as update nodes, the association relationship between the word segments in the newly added grammar template is used as an update path, and the prefix tree is updated according to the update nodes and the update path.

15. The log parsing method according to claim 12 or 13, further comprising, after determining the log group corresponding to the log to be parsed: Mapping the log to be parsed to the log group; Determining a new semantic template corresponding to the log to be parsed according to the predicted semantic template; storing the newly added log semantic template and the newly added syntax template determined according to the newly added semantic template in the template pool, and associating the newly added syntax template with the newly added log semantic template; The word segments included in the newly added grammar template are used as update nodes, the association relationship between the word segments in the newly added grammar template is used as an update path, and the prefix tree is updated according to the update nodes and the update path.

16. The log parsing method according to claim 3, wherein the parsing the log to be parsed using a large language model comprises: Determine log parsing tasks and log query problems based on the log to be parsed; Based on the feature vector of the log to be parsed and the feature vectors of the log semantic templates in the template pool, a preset number of example semantic templates are determined from the template pool, wherein the example semantic templates are a subset of the log semantic templates; and based on the log parsing task, the log query question, and the example semantic templates, the log to be parsed is parsed using a large language model.

17. A log parsing device, comprising: a log word segmentation sequence acquisition module configured to determine a log to be parsed, and perform word segmentation processing on the log to be parsed to obtain a log word segmentation sequence; The candidate grammar template determination module is configured to match the log word segmentation sequence with nodes in a prefix tree to determine a candidate grammar template corresponding to the log word segmentation sequence, wherein the prefix tree is constructed by nodes and paths, the nodes are words in the log grammar template, the paths are associations between words in the log grammar template, and the log grammar template is determined based on the initial log; the grammar template matching module is configured to match the log word segmentation sequence with the candidate grammar template according to a preset matching strategy to obtain a matching result; the log parsing module is configured to parse the log to be parsed using a large language model to obtain a parsing result if the matching result does not meet the preset matching condition.

18. A computing device comprising: memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the log parsing method according to any one of claims 1 to 16 are implemented.

19. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the log parsing method according to any one of claims 1 to 16.

20. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the log parsing method according to any one of claims 1 to 16.

Images