Large language model log analysis method and device based on relation inference

Abstract

The invention discloses a large language model log analysis method based on relation inference, which comprises the following steps of: by taking a prefix analysis tree as a core, preprocessing an input log, and traversing the prefix tree to realize template matching; and if the matching is successful, outputting the structured log, and if the matching is not successful, evaluating the relationship between the unmatched log and other logs from grammar and semantic two dimensions. In the grammar level, the grammar similarity between unmatched logs and a candidate log set is calculated, in the semantic level, the causal strength and the number of shared entities of the unmatched logs and logs in a historical log window are evaluated through LLM, and semantic association logs are screened out in combination with semantic composite scores; taking the screened log with similar grammar and semantic correlation as an example, constructing a complete prompt comprising a task instruction, a demonstration example and a to-be-analyzed log, and calling a large language model to generate a structured template of an unmatched log; and the newly generated template is adapted to the prefix analysis tree, and the tree structure is dynamically updated and perfected, so that more comprehensive template support is provided for subsequent log analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software, in particular to a log analysis method, and specifically to a large language model log analysis method and device based on relationship inference. BACKGROUND

[0002] With the continuous growth of the scale and complexity of modern software systems, logs have become a key data source for understanding runtime behavior, supporting anomaly detection, fault prediction and root cause diagnosis. Raw logs are often semi-structured, while most log analysis tools require structured input. Therefore, it is necessary to parse the logs to convert the raw log messages into structured templates and parameters. In recent years, large language models have shown good potential in the field of log analysis due to their powerful natural language understanding and generation capabilities.

[0003] Existing log analysis methods have obvious limitations. Syntax-based methods rely on the syntactic structure of logs, and although they are efficient, they have weak generalization ability. Semantics-based methods capture semantic features through neural network models to improve parsing accuracy, but they require a large amount of resources for training and have poor adaptability to newly deployed systems or evolving log formats. Large language model-based methods avoid the need for large-scale annotation and training, but are easily influenced by surface bias in prompt construction, overemphasizing surface token patterns while ignoring underlying semantics. They also have difficulty handling the long-tail distribution of system logs, and generally ignore the semantic association and causal dependence between logs, resulting in limited template generation accuracy. In addition, the way large language models are called line by line also brings high computational cost, making it difficult to meet the efficient parsing needs of large-scale logs. SUMMARY

[0004] To solve the technical problems of existing log analysis methods, such as surface bias in prompt construction, weak processing ability for long-tail log distribution, insufficient utilization of cross-log semantic association, and low computational efficiency, the present application provides a large language model log analysis method and device based on relationship inference, which overcomes the deficiencies of the existing technology by inferring relationships from both syntax and semantics, meeting the efficient parsing needs of large-scale logs.

[0005] According to one aspect of the present application, a large language model log analysis method based on relationship inference is provided, comprising: Taking the prefix parse tree as the core, the input log is preprocessed and the prefix parse tree is traversed to realize template matching; When no template is matched, the relationship between the unmatched log and the logs in the candidate log set and the historical log window is inferred from the syntax and semantic dimensions; wherein, the syntax similarity between the unmatched log and the logs in the candidate log set is calculated in the syntax dimension to screen out the syntax similar logs; in the semantic dimension, the causal strength and the number of shared entities between the unmatched log and the logs in the historical log window are inferred by the large language model, and then the semantic related logs are screened out in combination with the semantic composite score; The syntax similar logs and the semantic related logs screened out are used as demonstration examples to construct a complete prompt containing task instructions, demonstration examples and logs to be parsed, and a large language model is called to generate a structured template of the unmatched log. The newly generated template is adapted to the prefix parse tree to dynamically update the tree structure.

[0006] As a further technical solution, the method further comprises: in the offline stage, constructing a candidate log set by clustering and stratified sampling of historical system logs.

[0007] As a further technical solution, the template matching is implemented by traversing the prefix parse tree after the input log is preprocessed, comprising: The tokens in the log token sequence obtained by preprocessing are sequentially input into the prefix parse tree for node matching, wherein the first token is matched with the child nodes of the root node, when there is a corresponding normal token node or a wildcard node that can match the current token, the current node is entered, and then the next token in the token sequence is matched with the child nodes of the current node, and the token-by-token traversal matching operation is executed in this way. When the traversal operation terminates, if the current token cannot be matched with any child node of the internal node, and the leaf node is not reached, it is determined that the current input log cannot find a matched template in the existing prefix parse tree.

[0008] As a further technical solution, the prefix parse tree is composed of a root node, internal nodes and leaf nodes: the root node is the starting point of log parsing; the internal nodes include normal token nodes representing constants and wildcard nodes for matching any token; the leaf node uniquely corresponds to a stored log template, and the path token from the root node to the leaf node forms a complete log template string.

[0009] As a further technical solution, if a leaf node can be reached in the traversal process, it indicates that the current input log is successfully matched to the template corresponding to the leaf node.

[0010] As a further technical solution, the construction of the complete prompt containing task instructions, demonstration examples and logs to be parsed comprises: The prompt word is constructed based on context learning and follows a "task instruction-demonstration example-query" structure: first, the task instruction is clear; then, the filtered syntax similar logs and the semantic related logs are organized as demonstration examples in the form of "log example-corresponding structured template" pairs; and finally, the unmatched logs are embedded at the end of the prompt as queries.

[0011] As a further technical solution, the newly generated template is adapted to the prefix parse tree, and the tree structure is dynamically updated, including: All leaf nodes under the terminal node of the prefix parse tree are extracted, and the templates corresponding to all leaf nodes constitute a candidate template set; The similarity of the newly generated template and each candidate template is calculated using the longest common subsequence length; When the similarity of a certain candidate template exceeds a threshold, the corresponding fixed token in the current candidate template is replaced with a wildcard to adapt to the new template; When the similarity of all candidate templates is lower than the threshold, a node path corresponding to the token sequence of the newly generated template is added under the terminal node, and a leaf node is added at the end of the path to store the new template.

[0012] According to an aspect of the present application, a large language model log parsing device based on relationship inference is provided, comprising: A prefix parse tree parsing module is used to implement template matching by traversing the prefix parse tree after preprocessing the input log with the prefix parse tree as the core; A log relationship inference module is used to infer the relationship between the unmatched log and the logs in the candidate log set and the historical log window from the syntax and semantic dimensions when no template is matched; wherein, the syntax similarity between the unmatched log and the logs in the candidate log set is calculated in the syntax dimension to filter out the syntax similar logs; and the causality strength and the number of shared entities between the unmatched log and the logs in the historical log window are inferred by the large language model in the semantic dimension, and then the semantic related logs are filtered out by combining the semantic composite score; A log template generation module is used to construct a complete prompt containing a task instruction, demonstration examples, and a log to be parsed by taking the filtered syntax similar logs and semantic related logs as demonstration examples, and to generate a structured template of the unmatched log by calling a large language model; A prefix parse tree tree update module is used to adapt the newly generated template to the prefix parse tree and dynamically update the tree structure.

[0013] According to an aspect of the present application, a computing device is provided, comprising a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the steps of the log parsing method of the large language model based on relationship inference.

[0014] According to an aspect of the present application, a computer storage medium is provided, and the storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute the steps of the log parsing method of the large language model based on relationship inference.

[0015] Compared with the prior art, the present application has the following advantages: The log parsing method and device of the large language model based on relationship inference provided by the present application aim to solve the technical problems of the existing log parsing method, such as surface bias of prompt construction, weak processing capability of long-tail log distribution, insufficient utilization of cross-log semantic association, and low computing efficiency. The core process includes: taking the prefix parsing tree as the core, pre-processing the input log, and then traversing the prefix parsing tree to realize fast template matching. If the matching is successful, the structured log is directly output, and if the matching is not successful, the subsequent process is entered; the unmatched log will infer its relationship with other logs in the candidate log set from the syntax and semantics. Among them, the syntax level calculates the syntax similarity of the unmatched log and the logs in the candidate log set, the semantic level infers the causal strength and the number of shared entities of the unmatched log and the logs in the historical log window through the large language model, and then combines the semantic composite score to screen out the semantically related logs; the syntax similar logs and the semantically related logs screened out are used as examples to construct a complete prompt containing task instructions, demonstration examples and logs to be parsed, and a large language model is called to generate a structured template of the unmatched log; finally, the newly generated template is adapted to the prefix parsing tree, and the tree structure is perfected through dynamic updating, providing more comprehensive template support for subsequent log parsing. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or prior art description will be briefly described as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 A flowchart of a log parsing method of a large language model based on relationship inference provided by an embodiment of the present application is shown; Figure 2 A main framework diagram of a log parsing method of a large language model based on relationship inference provided by an embodiment of the present application is shown; Figure 3 A structural diagram of a computing device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] The terms "comprise", "comprising", "include", "including", "have", "having" and any variations thereof in the specification and in the claims are intended to cover both the express stated features or steps and also those that are equivalent or similar thereto in function, operation or effect, unless expressly stated otherwise. The terms "comprise", "comprising", "include", "including", "have", "having" and any variations thereof are not intended to exclude any step or feature, whether explicitly stated or not.

[0019] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application. In addition, the technical features in each embodiment or in a single embodiment provided by the present application can be combined with each other at will to form new technical solutions, and the combination is not restricted by the order of steps and / or the mode of structural composition, but should be based on the realization by a person of ordinary skill in the art. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope of the present application.

[0020] Reference is made to Figure 1 For the purpose of description, the present application provides a large language model log analysis method based on relationship inference, which comprises the following steps:

[0021] Step S101: In the offline stage, a candidate log set with diversity and representativeness is constructed by clustering and stratified sampling of historical system logs.

[0022] First, the historical system logs are preprocessed. Each log entry is converted to lowercase and then processed using the regular expression "([^\w]+)" to retain all consecutive non-alphanumeric symbols (such as " / ", ";", "=", etc.) as independent tokens, forming a token set containing syntactic content and structural separators. Next, a lightweight signature is generated for each log entry. The top-k high-frequency tokens of the log are sorted and concatenated with the symbols retained in the preprocessing stage. This signature allows for quick differentiation of different template types. Then, an O(N) single-pass bucketing strategy is used to group logs with the same signature into the same cluster, achieving high-precision template clustering. Finally, a tiered quota sampling strategy is employed, allocating representative log slots according to the size of each cluster. While prioritizing the coverage of dominant templates to ensure universality, sampling positions are reserved for a few low-frequency templates, ultimately constructing a candidate log set that combines diversity and representativeness. In this embodiment, the tiered sampling number for the candidate log pool is set to 32 entries. This number can be adaptively adjusted based on the total amount of historical logs and is not the only limitation of this invention.

[0023] Step S102: Using the prefix parse tree as the core, preprocess the input log and then traverse the prefix parse tree to achieve fast template matching. If a match is successful, output the structured log directly; otherwise, proceed to the next step.

[0024] Before parsing the input logs using a prefix parse tree, the same preprocessing operations described above must be performed to generate a token sequence. Then, the prefix parse tree matching phase begins. The prefix parse tree consists of a root node, a set of internal nodes, and a set of leaf nodes. The root node serves as the starting point for all log parsing. The internal nodes contain both ordinary token nodes representing fixed text fragments of the log and special wildcard nodes used to match any token. The leaf node uniquely corresponds to a stored log template. The tokens of all nodes traversed from the root node to any leaf node are concatenated in order to form the complete log template string corresponding to that leaf node.

[0025] The specific process of prefix parse tree parsing is as follows: Starting from the root node of the prefix parse tree, the tokens in the preprocessed log token sequence are input into the prefix parse tree in sequence for node matching. That is, the first token is matched with the child nodes of the root node. If a corresponding ordinary token node or a matching "" exists, the matching is performed. If the token is a wildcard node, the token sequence is matched with the child nodes of the wildcard node, and the token sequence is matched with the child nodes of the wildcard node in a loop. If the token sequence is finally matched with a leaf node in the traversal process, it is indicated that the current input log is successfully matched with the log template corresponding to the leaf node, and the structured log template corresponding to the leaf node is directly output. If the traversal operation is terminated, the current token cannot be matched with any child node of the internal node, and a leaf node is not reached, it is determined that the current input log cannot be matched with the existing prefix parsing tree, and the unmatched log is forwarded to a subsequent process.

[0026] Step S103: The unmatched log infers its relationship with other logs from the syntax and semantics. The syntax similarity between the unmatched log and the logs in the candidate log set is calculated to filter out the syntax-related logs; the causality strength and the number of shared entities between the unmatched log and the logs in the historical log window are inferred by the large language model, and the most relevant example is selected by combining the semantic composite score, that is, the semantic-related log. The other logs here include the logs in the candidate log set and the logs in the historical log window.

[0027] For the unmatched log in S102, the relationship is inferred from the syntax and semantics to filter out high-relevance demonstration examples. The syntax relationship inference directly calls the candidate log set constructed offline in S101, and calculates the syntax similarity between the unmatched log and the candidate log by using the Jaccard coefficient: the tokens of the unmatched log and the candidate log are obtained by tokenizing the unmatched log and the candidate log, respectively. The token sets and are obtained by tokenizing the unmatched log and the candidate log, respectively. The ratio of the number of intersection elements to the number of union elements is calculated by the formula: , The top-m syntax-similar logs are finally selected according to the similarity results.

[0028] In this embodiment, the semantic relationship inference is performed on the 3 logs before the unmatched log to form a historical log window. This design can not only make the logs close in time more likely to have semantic associations (such as causal dependence and entity sharing), but also avoid the logical loop problem caused by relying on future logs that have not been parsed. Then, a large language model is introduced as a reasoning engine to construct a structured template and input the pair of the unmatched log and the log in the historical log window, guide the large language model to analyze the semantic connection of the two from the event logic, state dependence and other dimensions, and finally output <start>log_idl, log_id2, causal_strength, entity_count <end>The formatted results are structured. `causal_strength` is a continuous value between 0 and 1, representing the strength of the causal relationship between logs; `entity_count` is an integer, quantifying the number of shared entities (such as user ID, IP address, service name, etc.) between logs; `log_id1` and `log_id2` represent the ID numbers of the two logs, respectively. In this embodiment, the large language model used is GPT-4o mini, and the temperature parameter is set to 0 to ensure more stable output. Other large language models can also be used in practical applications.

[0029] To comprehensively infer semantic relevance, a semantic composite score (SCS) needs to be calculated. First, the entity_count (denoted as e) is normalized: , This is because, statistically, over 90% of log pairs share no more than 3 entities; further, using the formula SCS=c+ +c× (where c represents causal strength) The score calculation reflects both the independent contribution of causality and entity relationships, as well as their synergistic effect, ultimately selecting the top-n semantically relevant logs, which, together with syntactically similar logs, constitute the demonstration example. In this example, the number of syntactically similar logs m=3, the historical log window size k=3, and the number of semantically relevant logs n=1. That is, three log examples are selected at the syntactic level, and at the semantic level, one semantically related log example is selected from the top three historical logs.

[0030] Step S104: Using the selected logs with similar syntax and semantic relevance as examples, construct a complete prompt containing task instructions, demonstration examples, and logs to be parsed, and call the large language model to generate a structured template for the unmatched logs.

[0031] Context-based learning is used to construct prompt words, and a large language model is invoked to generate structured templates for unmatched logs. The prompt words strictly follow a three-part structure: "instruction-demonstration example-query." First, the task instruction is clearly defined, specifying "identify dynamic variables in the log (such as session ID, IP address, status code, task number, etc.) and replace them with..." The core requirements include "fully preserving constant parts (including formatting and spelling errors)," while providing examples of common variable types to guide the model in distinguishing between dynamic and static elements. Then, the syntactically similar and semantically relevant logs filtered in S103 are organized into pairs of "log example - corresponding structured template" as demonstration examples, providing the model with dual references of syntactic structure and semantic background. Finally, the unmatched logs from S102 are used as queries and embedded at the end of the prompt. The structured template of the unmatched logs is then called from the large language model to ensure that all dynamic variables are "fully preserved." "Replacement, static text completely retains original features. In this embodiment, GPT-4o mini is also used to generate log templates, and the temperature parameter is set to 0."

[0032] Step S105: Adapt the newly generated template into the prefix parsing tree, and improve the tree structure through dynamic updates to provide more comprehensive template support for subsequent log parsing.

[0033] First, extract all leaf nodes under the terminal node in the prefix parsing tree for the unmatched logs in S102. The templates corresponding to these leaf nodes constitute the candidate template set. Then, calculate the new template generated in S104. ) and each candidate template ( The similarity between the sequences is calculated using the length of the longest common subsequence (LCS), as follows: , Where Lg and Lc are the token sequences of the new template and candidate template, respectively; in this embodiment, they are set as follows: The threshold is 0.8. If the similarity exceeds the threshold, it means the existing template needs to be adjusted to accommodate unmatched logs. The corresponding fixed token in the candidate template needs to be replaced with " Wildcards are used to adapt to the new template; if the similarity is below the threshold, it means that no template can approximately match the unmatched log. Therefore, a new node path corresponding to the token sequence of the new template is added under the terminal node, and a leaf node is added at the end of the path to store the new template; after the adaptation is completed, the coverage of the prefix parse tree is expanded, providing more comprehensive template support for subsequent log parsing.

[0034] Figure 2 This paper illustrates the main framework of a large language model log parsing device based on relation inference provided by an embodiment of the present invention, which fully presents the entire process logic for parsing input logs.

[0035] During the offline phase, the candidate sampling module clusters and performs hierarchical sampling on historical system logs to construct a candidate log set that is both diverse and representative.

[0036] The input log, as the raw data to be parsed, first enters the prefix parse tree parsing module. By traversing and matching token by token, it determines whether the input log can match an existing template. If a match is successful, the structured log is directly output; if a match fails, the unmatched log is passed to the subsequent relation inference module.

[0037] In the log relationship inference module, unmatched logs trigger two parallel processes: syntactic relationship inference and semantic relationship inference. The syntactic relationship inference module, based on an offline-built candidate log set, uses the Jaccard coefficient to filter log examples with similar syntactic structures to the unmatched logs. The semantic relationship inference module, based on a historical log window, uses a large language model to infer the "causal strength" and "number of shared entities" between logs, thereby filtering log examples with high semantic relevance. Ultimately, syntactically similar log examples and semantically related log examples together serve as "demonstration examples" to participate in the construction of subsequent prompt words.

[0038] Then, in the log template generation module, the "instruction", the "demonstration example" obtained from the above filtering, and the "question" are combined into a complete prompt word, which is then input into the large language model module to generate the corresponding log template.

[0039] In the prefix tree update module, the newly generated log template will update the prefix parse tree in reverse. By adapting to existing templates or adding new template paths, the structure of the prefix parse tree is optimized so that it can cover more log types and provide more comprehensive template support for the parsing of subsequent input logs.

[0040] This invention provides a non-volatile computer storage medium storing at least one executable instruction that can execute a large language model log parsing method based on relation inference as described in the above embodiments.

[0041] Executable instructions can specifically be used to cause the processor to perform the following operations: First, during the offline phase, a candidate log set that is both diverse and representative is constructed by clustering and hierarchical sampling of historical system logs. Using a prefix parse tree as the core, the input log is preprocessed and then the prefix parse tree is traversed to achieve fast template matching. If a match is successful, the structured log is output directly; otherwise, the process continues to the next step. Unmatched logs will have their relationships with other logs inferred from both syntactic and semantic dimensions. At the syntactic level, the syntactic similarity between unmatched logs and the candidate log set will be calculated. At the semantic level, a large language model will be used to infer the causal strength and the number of shared entities between unmatched logs and logs in the historical log window. Finally, a semantic composite score will be used to select the most relevant examples. Using the selected logs with similar syntax and semantic relevance as examples, we construct a complete prompt containing task instructions, demonstration examples, and logs to be parsed, and call the large language model to generate a structured template for the unmatched logs. Finally, the newly generated template is adapted into the prefix parsing tree, and the tree structure is improved through dynamic updates, providing more comprehensive template support for subsequent log parsing.

[0042] Figure 3 The structural schematic diagram of the computing device embodiment of the present application is shown, and the specific embodiment of the present application does not limit the specific implementation of the computing device.

[0043] As shown in the figure, the computing device can include: Figure 3 a processor, a communications interface, a memory, and a communications bus.

[0044] Among them, the processor, the communications interface, and the memory complete the communication with each other through the communications bus. The communications interface is used to communicate with network elements of other devices, such as clients or other servers. The processor is used to execute programs, and specifically can execute the related steps in the above-mentioned one social media content propagation prediction method embodiment.

[0045] Specifically, the program can include program code, which includes computer operation instructions.

[0046] The processor can be a central processing unit CPU, or an application specific integrated circuit ASIC, or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the server can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0047] The memory is used to store programs. The memory can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0048] The program can specifically be used to cause the processor to perform the following operations: In the offline stage, by performing a word segmentation operation on the historical system log, then clustering the log according to the high-frequency token and single-pass bucketing strategy, then performing hierarchical quota sampling, a candidate log set with diversity and representativeness is constructed; Load the prefix parse tree data stored in the memory, and perform a traversal matching operation on the input log in the order of the token sequence after preprocessing. If the matching is successful, the structured log is directly output, and if the matching is not successful, the subsequent process is entered; ​Unmatched logs will have their relationships with other logs inferred from both syntactic and semantic dimensions. Candidate log set data is used, and syntactic similarity is calculated using the Jaccard coefficient formula. Semantically, a large language model is used to infer the causal strength and the number of shared entities between unmatched logs and logs in the historical log window. Finally, a semantic composite scoring formula is used to calculate the most relevant examples. Using the selected logs with similar syntax and semantic relevance as examples, we construct a complete prompt containing task instructions, demonstration examples, and logs to be parsed, and call the large language model to generate a structured template for the unmatched logs. Finally, the newly generated template is adapted into the prefix parse tree, and the updated prefix parse tree data is written to storage, providing more comprehensive template support for subsequent log parsing.

[0049] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0050] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0051] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in fewer than all features of the single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0052] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0053] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, any of the claimed embodiments can be used in any combination.

[0054] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0055] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps not listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for carrying out a certain function. The functions of the separate means can be carried out by one specific apparatus or means. The use of the terms 'first','second' and 'third' and the like in the description is only for distinguishing between similar elements, and is not meant to denote a physical or chronological priority. The terms 'comprises', 'comprising', 'includes', 'including', 'contains', 'containing' or similar are not intended to exclude the presence of elements or steps not listed in a claim. The terms 'a', 'an' and 'the' are not intended to be limiting in the sense of the specification and the patent claims.< / end> < / start> ​

Claims

1. A log parsing method for large language models based on relation inference, characterized in that, include: Using the prefix parse tree as the core, template matching is achieved by traversing the prefix parse tree after preprocessing the input logs; When no template is matched, the relationship between the unmatched logs and the logs in the candidate log set and the logs in the historical log window is inferred from both syntactic and semantic dimensions. Specifically, in the syntactic dimension, the syntactic similarity between the unmatched logs and the logs in the candidate log set is calculated to filter out syntactically similar logs. In the semantic dimension, the causal strength and the number of shared entities between the unmatched logs and the logs in the historical log window are inferred through a large language model, and then semantically related logs are filtered out by combining the semantic composite score. Using the selected grammatically similar and semantically related logs as demonstration examples, a complete prompt containing task instructions, demonstration examples, and logs to be parsed is constructed, and a large language model is called to generate a structured template for the unmatched logs. The newly generated template is adapted into the prefix parsing tree, and the tree structure is dynamically updated.

2. The log parsing method for a large language model based on relation inference according to claim 1, characterized in that, The method further includes: during the offline phase, constructing a candidate log set by clustering and hierarchical sampling of historical system logs.

3. The log parsing method for a large language model based on relation inference according to claim 1, characterized in that, The process of preprocessing the input logs and then traversing the prefix parse tree to achieve template matching includes: The tokens in the preprocessed log token sequence are sequentially input into the prefix parse tree for node matching. The first token is matched with the child nodes of the root node. If there is a corresponding ordinary token node or a wildcard node that can match the current token, the current node is entered. Then the next token in the token sequence is matched with the child nodes of the current node. This token-by-token traversal matching operation is performed in a loop. When the traversal operation terminates, if the current token cannot match any child node of the internal node and has not reached a leaf node, it is determined that the current input log cannot find a matching template in the existing prefix parsing tree.

4. The log parsing method for a large language model based on relation inference according to claim 3, characterized in that, The prefix parse tree consists of a root node, internal nodes, and leaf nodes: the root node is the starting point for log parsing; the internal nodes include ordinary token nodes representing constants and wildcard nodes used to match any token; each leaf node uniquely corresponds to a stored log template, and the path tokens from the root node to the leaf node are concatenated to form a complete log template string.

5. The log parsing method for a large language model based on relation inference according to claim 3, characterized in that, During the traversal, if a leaf node is reached, it indicates that the current input log has successfully matched the template corresponding to that leaf node.

6. The log parsing method for a large language model based on relation inference according to claim 1, characterized in that, The construct includes complete prompts such as task instructions, demonstration examples, and logs to be parsed, including: The prompt words are constructed based on context learning, and the prompt words follow the structure of "task instruction-demonstration example-query": first, the task instruction is clarified; then, the selected grammatically similar logs and semantically related logs are organized into demonstration examples in pairs of "log example-corresponding structured template"; finally, the unmatched logs are used as queries and embedded at the end of the prompt.

7. The log parsing method for a large language model based on relation inference according to claim 1, characterized in that, The step of adapting the newly generated template to the prefix parsing tree and dynamically updating the tree structure includes: Extract all leaf nodes under the terminal node of the unmatched log in the prefix parsing tree, and the templates corresponding to all leaf nodes constitute a candidate template set. The similarity between the newly generated template and each candidate template is calculated using the length of the longest common subsequence. When the similarity of a candidate template exceeds a threshold, the corresponding fixed token in the current candidate template is replaced with a wildcard to adapt to the new template; If the similarity of all candidate templates is below the threshold, add a new node path corresponding to the newly generated template token sequence under the termination node, and add a leaf node at the end of the path to store the new template.

8. A log parsing device for a large language model based on relation inference, characterized in that, include: The prefix parse tree parsing module is used to perform template matching by traversing the prefix parse tree after preprocessing the input logs, with the prefix parse tree as the core. The log relationship inference module is used to infer the relationship between unmatched logs and logs in the candidate log set and the historical log window from both syntactic and semantic dimensions when no template is matched. Specifically, in the syntactic dimension, the syntactic similarity between unmatched logs and logs in the candidate log set is calculated to filter out syntactically similar logs; in the semantic dimension, the causal strength and the number of shared entities between unmatched logs and logs in the historical log window are inferred through a large language model, and then semantic composite scores are combined to filter out semantically related logs. The log template generation module is used to construct a complete prompt containing task instructions, demonstration examples, and logs to be parsed by using filtered syntactically similar and semantically related logs as demonstration examples, and calling a large language model to generate a structured template for unmatched logs. The prefix parse tree update module is used to adapt newly generated templates into the prefix parse tree and dynamically update the tree structure.

9. A computing device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of a large language model log parsing method based on relation inference as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The storage medium stores at least one executable instruction that causes the processor to perform the steps of a large language model log parsing method based on relation inference as described in any one of claims 1-7.

Images