A large language model log analysis method and device based on relationship inference

Abstract

The application discloses a large language model log analysis method based on relationship inference, comprising: taking a prefix parsing tree as the core, pre-processing the input log, and then traversing the prefix tree to realize template matching. If the matching is successful, the structured log is output, if the matching is unsuccessful, the relationship between the unmatched log and other logs is evaluated from the syntax and semantics. The syntax similarity between the unmatched log and the candidate log set is calculated at the syntax level, the causality strength and the number of shared entities between the unmatched log and the logs in the historical log window are evaluated through LLM at the semantic level, and the semantic correlation log is screened out by combining the semantic composite score; the screened syntax similar and semantic related log is taken as an example to construct a complete prompt containing a task instruction, a demonstration example and a log to be parsed, and a large language model is called to generate a structured template of the unmatched log; the newly generated template is adapted to the prefix parsing tree, the tree structure is perfected through dynamic updating, and more comprehensive template support is provided for subsequent log analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software, in particular to a log analysis method, and specifically to a large language model log analysis method and device based on relationship inference. BACKGROUND

[0002] With the continuous growth of the scale and complexity of modern software systems, logs have become a key data source for understanding runtime behavior, supporting anomaly detection, fault prediction, and root cause diagnosis. Raw logs are often semi-structured, while most log analysis tools require structured input. Therefore, it is necessary to parse the logs to convert the raw log messages into structured templates and parameters. In recent years, large language models have shown good potential in the field of log analysis due to their powerful natural language understanding and generation capabilities.

[0003] Existing log analysis methods have obvious limitations. Syntax-based methods rely on the syntactic structure of logs, and although they are efficient, they have weak generalization ability. Semantics-based methods capture semantic features through neural network models to improve parsing accuracy, but they require a large amount of resources for training and have poor adaptability to newly deployed systems or evolving log formats. Large language model-based methods avoid the need for large-scale annotation and training, but are easily influenced by surface bias in prompt construction, focusing too much on surface token patterns and ignoring underlying semantics. They also have difficulty handling the long-tail distribution of system logs, and generally ignore the semantic association and causal dependence between logs, limiting the accuracy of template generation. In addition, the way large language models are called line by line also brings high computational cost, making it difficult to meet the efficient parsing needs of large-scale logs. SUMMARY

[0004] To solve the technical problems of existing log analysis methods, such as surface bias in prompt construction, weak processing ability for long-tail log distribution, insufficient use of cross-log semantic association, and low computational efficiency, the present application provides a large language model log analysis method and device based on relationship inference, which overcomes the deficiencies of the existing technology by inferring relationships from both syntax and semantics, meeting the efficient parsing needs of large-scale logs.

[0005] According to one aspect of the present application, a large language model log analysis method based on relationship inference is provided, comprising:

[0006] Taking the prefix parse tree as the core, the input log is preprocessed and the prefix parse tree is traversed to implement template matching;

[0007] When no template is matched, the relationship between the unmatched log and the logs in the candidate log set and the historical log window is inferred from the syntax and semantic dimensions; wherein, the syntax similarity between the unmatched log and the logs in the candidate log set is calculated in the syntax dimension, and the syntax similar logs are screened out; in the semantic dimension, the causal strength and the number of shared entities between the unmatched log and the logs in the historical log window are inferred by the large language model, and then the semantic related logs are screened out in combination with the semantic composite score;

[0008] The screened syntax similar logs and the semantic related logs are used as demonstration examples to construct a complete prompt containing task instructions, demonstration examples and logs to be parsed, and a large language model is called to generate a structured template of the unmatched log;

[0009] The newly generated template is adapted to the prefix parse tree, and the tree structure is dynamically updated.

[0010] As a further technical solution, the method further comprises: in the offline stage, constructing a candidate log set by clustering and hierarchical sampling of historical system logs.

[0011] As a further technical solution, the template matching is implemented by traversing the prefix parse tree after the input log is preprocessed, comprising:

[0012] The tokens in the log token sequence obtained by preprocessing are sequentially input into the prefix parse tree for node matching, wherein the first token is matched with the child nodes of the root node, and when there is a corresponding normal token node or a wildcard node that can match the current token, the current node is entered, and then the next token in the token sequence is matched with the child nodes of the current node, and the token-by-token traversal matching operation is executed in this way.

[0013] When the traversal operation terminates, if the current token cannot be matched with any child node of the internal node, and the leaf node has not been reached, it is determined that the current input log cannot find a matched template in the existing prefix parse tree.

[0014] As a further technical solution, the prefix parse tree is composed of a root node, internal nodes and leaf nodes: the root node is the starting point of log parsing; the internal nodes include normal token nodes representing constants and wildcard nodes for matching any token; the leaf node uniquely corresponds to a stored log template, and the path token from the root node to the leaf node forms a complete log template string.

[0015] As a further technical solution, if a leaf node can be reached in the traversal process, it indicates that the current input log is successfully matched with the template corresponding to the leaf node.

[0016] As a further technical solution, the construction includes task instructions, demonstration examples, and complete prompts for logs to be parsed, including:

[0017] The context-based learning constructs a prompt word, which follows the structure of "task instruction-demonstration example-query": first, the task instruction is clear; then, the filtered syntaxically similar logs and semantically related logs are organized as demonstration examples in the form of pairs of "log examples-corresponding structured templates"; finally, the unmatched logs are embedded at the end of the prompt as queries.

[0018] As a further technical solution, the new generated template is adapted to the prefix parse tree, and the tree structure is dynamically updated, including:

[0019] Extract all leaf nodes under the terminal node of the prefix parse tree for the unmatched log, and the templates corresponding to all leaf nodes constitute a candidate template set;

[0020] The similarity between the newly generated template and each candidate template is calculated using the longest common subsequence length;

[0021] When the similarity of a certain candidate template exceeds the threshold, replace the corresponding fixed token in the current candidate template with a wildcard to adapt the new template;

[0022] When the similarity of all candidate templates is lower than the threshold, add a node path corresponding to the token sequence of the newly generated template under the terminal node, and add a leaf node at the end of the path to store the new template.

[0023] According to an aspect of the present application, a large language model log parsing device based on relationship inference is provided, comprising:

[0024] The prefix parse tree parsing module is used to traverse the prefix parse tree to realize template matching after preprocessing the input log with the prefix parse tree as the core;

[0025] The log relationship inference module is used to infer the relationship between the unmatched log and the logs in the candidate log set and the historical log window from the syntax and semantic dimensions when no template is matched; wherein, the syntax similarity between the unmatched log and the logs in the candidate log set is calculated in the syntax dimension to filter out syntaxically similar logs; in the semantic dimension, the causality strength and the number of shared entities between the unmatched log and the logs in the historical log window are inferred by the large language model, and then the semantic related logs are selected by combining the semantic composite score;

[0026] The log template generation module is used to construct a complete prompt including task instructions, demonstration examples, and logs to be parsed by taking the filtered syntaxically similar logs and semantically related logs as demonstration examples, and to generate the structured template of the unmatched log by calling the large language model;

[0027] The prefix parsing tree tree updating module is configured to adapt a newly generated template to the prefix parsing tree, and dynamically update the tree structure.

[0028] According to an aspect of the present application, a computing device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus;

[0029] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the steps of the large language model log parsing method based on relationship inference.

[0030] According to an aspect of the present application, a computer storage medium is provided, and the storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute the steps of the large language model log parsing method based on relationship inference.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] The large language model log parsing method and device based on relationship inference provided by the present application aim to solve the technical problems of the existing log parsing methods, such as surface bias of prompt construction, weak processing capability of long-tail log distribution, insufficient utilization of cross-log semantic association, and low computing efficiency. The core process includes: taking the prefix parsing tree as the core, pre-processing the input log, and then traversing the prefix parsing tree to realize fast template matching. If the matching is successful, the structured log is directly output, and if the matching is not successful, the subsequent process is entered; the unmatched log will infer its relationship with other logs in the candidate log set from the syntax and semantics. Among them, the syntax level calculates the syntax similarity of the unmatched log and the logs in the candidate log set, the semantic level infers the cause-effect strength and the number of shared entities of the unmatched log and the logs in the historical log window through the large language model, and then combines the semantic composite score to screen out the semantically related logs; the syntax similar logs and the semantically related logs screened out are used as examples to construct a complete prompt containing task instructions, demonstration examples and logs to be parsed, and a large language model is called to generate a structured template of the unmatched log; finally, the newly generated template is adapted to the prefix parsing tree, and the tree structure is dynamically updated to perfect the tree structure, providing more comprehensive template support for subsequent log parsing. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0034] Figure 1 The diagram illustrates a flowchart of a large language model log parsing method based on relation inference provided by an embodiment of the present invention;

[0035] Figure 2 This diagram illustrates the main framework of a log parsing method for a large language model based on relation inference, as provided in an embodiment of the present invention.

[0036] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present invention is shown. Detailed Implementation

[0037] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0038] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0039] See Figure 1 As explained, this invention provides a method for parsing large language model logs based on relation inference, including the following steps:

[0040] Step S101: In the offline phase, a candidate log set with both diversity and representativeness is constructed by clustering and hierarchical sampling of historical system logs.

[0041] First, the historical system logs are preprocessed. Each log entry is converted to lowercase and then processed using the regular expression "([^\w]+)" to retain all consecutive non-alphanumeric symbols (such as " / ", ";", "=", etc.) as independent tokens, forming a token set containing syntactic content and structural separators. Next, a lightweight signature is generated for each log entry. The top-k high-frequency tokens of the log are sorted and concatenated with the symbols retained in the preprocessing stage. This signature allows for quick differentiation of different template types. Then, an O(N) single-pass bucketing strategy is used to group logs with the same signature into the same cluster, achieving high-precision template clustering. Finally, a tiered quota sampling strategy is employed, allocating representative log slots according to the size of each cluster. While prioritizing the coverage of dominant templates to ensure universality, sampling positions are reserved for a few low-frequency templates, ultimately constructing a candidate log set that combines diversity and representativeness. In this embodiment, the tiered sampling number for the candidate log pool is set to 32 entries. This number can be adaptively adjusted based on the total amount of historical logs and is not the only limitation of this invention.

[0042] Step S102: Using the prefix parse tree as the core, preprocess the input log and then traverse the prefix parse tree to achieve fast template matching. If a match is successful, output the structured log directly; otherwise, proceed to the next step.

[0043] Before parsing the input logs using a prefix parse tree, the same preprocessing operations described above must be performed to generate a token sequence. Then, the prefix parse tree matching phase begins. The prefix parse tree consists of a root node, a set of internal nodes, and a set of leaf nodes. The root node serves as the starting point for all log parsing. The internal nodes contain both ordinary token nodes representing fixed text fragments of the log and special wildcard nodes used to match any token. The leaf node uniquely corresponds to a stored log template. The tokens of all nodes traversed from the root node to any leaf node are concatenated in order to form the complete log template string corresponding to that leaf node.

[0044] The specific process of prefix parse tree parsing is as follows: Starting from the root node of the prefix parse tree, the tokens in the preprocessed log token sequence are input into the prefix parse tree in sequence for node matching. That is, the first token is matched with the child nodes of the root node. If a corresponding ordinary token node or a matching "" exists, the matching is performed. "If a wildcard node is encountered, the process proceeds to that node, and the next token in the token sequence is used to match the child nodes of the current node. This token-by-token traversal matching operation is repeated. If a leaf node is eventually reached during the traversal, it indicates that the current input log has successfully matched the log template corresponding to that leaf node, and the structured log template corresponding to that leaf node is directly output. If the traversal operation terminates, and the current token cannot match any child node of the internal node, and a leaf node has not been reached, it is determined that the current input log cannot find a matching template in the existing prefix parsing tree, and the unmatched log will be forwarded to the subsequent process."

[0045] Step S103: The relationship between unmatched logs and other logs will be inferred from both syntactic and semantic dimensions. At the syntactic level, the syntactic similarity between the unmatched logs and logs in the candidate log set is calculated to filter out syntactically related logs. At the semantic level, a large language model is used to infer the causal strength and the number of shared entities between the unmatched logs and logs in the historical log window. This is then combined with a semantic composite score to filter out the most relevant examples, i.e., semantically related logs. The other logs here include logs in the candidate log set and logs in the historical log window.

[0046] For the unmatched logs in S102, relation inference is performed from both syntactic and semantic dimensions to filter highly relevant demonstration examples. Syntactic relation inference directly calls the candidate log set built offline in S101, using the Jaccard coefficient to calculate the syntactic similarity between the unmatched logs and the candidate logs: [The remaining text appears to be incomplete and requires further context.] With candidate logs Each word is segmented to obtain a token set. and Through the formula:

[0047] ,

[0048] Calculate the ratio of the number of elements in the intersection to the number of elements in the union of the two sets. Because special characters are preserved as independent tokens during word segmentation, the system can accurately capture the fine-grained structural patterns of the logs. Finally, the top-m syntactically similar logs are selected based on the similarity results.

[0049] In this embodiment, semantic relationship inference uses the three log entries preceding the unmatched log entry to form a historical log window. This design is advantageous because logs from similar times are more likely to have semantic connections (such as causal dependencies or entity sharing), and it also avoids the logical loop problem caused by relying on unresolved future log entries. Then, a large language model is introduced as the inference engine. A structured template is constructed, and the log pairs from the unmatched log entry and the historical log window are input. The large language model is guided to analyze the semantic connections between the two from dimensions such as event logic and state dependencies, ultimately outputting the result. <start>log_idl, log_id2, causal_strength, entity_count <end>The formatted results are structured. `causal_strength` is a continuous value between 0 and 1, representing the strength of the causal relationship between logs; `entity_count` is an integer, quantifying the number of shared entities (such as user ID, IP address, service name, etc.) between logs; `log_id1` and `log_id2` represent the ID numbers of the two logs, respectively. In this embodiment, the large language model used is GPT-4o mini, and the temperature parameter is set to 0 to ensure more stable output. Other large language models can also be used in practical applications.

[0050] To comprehensively infer semantic relevance, a semantic composite score (SCS) needs to be calculated. First, the entity_count (denoted as e) is normalized:

[0051] ,

[0052] This is because, statistically, over 90% of log pairs share no more than 3 entities; further, using the formula SCS=c+ +c× (where c represents causal strength) The score calculation reflects both the independent contribution of causality and entity relationships, as well as their synergistic effect, ultimately selecting the top-n semantically relevant logs, which, together with syntactically similar logs, constitute the demonstration example. In this example, the number of syntactically similar logs m=3, the historical log window size k=3, and the number of semantically relevant logs n=1. That is, three log examples are selected at the syntactic level, and at the semantic level, one semantically related log example is selected from the top three historical logs.

[0053] Step S104: Using the selected logs with similar syntax and semantic relevance as examples, construct a complete prompt containing task instructions, demonstration examples, and logs to be parsed, and call the large language model to generate a structured template for the unmatched logs.

[0054] Context-based learning is used to construct prompt words, and a large language model is invoked to generate structured templates for unmatched logs. The prompt words strictly follow a three-part structure: "instruction-demonstration example-query." First, the task instruction is clearly defined, specifying "identify dynamic variables in the log (such as session ID, IP address, status code, task number, etc.) and replace them with..." core requirements such as "retain the constant part (including format, spelling errors) completely", and provide examples of common variable types to guide the model to distinguish dynamic and static elements; then the S103 filtered syntax similar logs and semantic related logs are organized in the form of pairs of "log example-corresponding structured template" as demonstration examples, providing the model with both syntax structure and semantic background references; finally, the unmatched logs in S102 are taken as queries, and the end of the prompt is embedded. The large language model is called to output the structured template of the unmatched log, ensuring that dynamic variables are replaced with , and static text is completely retained with original features. In this embodiment, GPT-4o mini is also used to generate log templates, and the temperature parameter is set to 0.

[0055] Step S105: Adapt the newly generated template to the prefix parse tree, and perfect the tree structure by dynamic updating to provide more comprehensive template support for subsequent log parsing.

[0056] First, extract all leaf nodes under the terminal node of the prefix parse tree for the unmatched log in S102, and the templates corresponding to these leaf nodes constitute a candidate template set; then calculate the similarity between the new template generated in S104 ( ) and each candidate template ( ), using the longest common subsequence (LCS) length calculation, the formula is:

[0057] ,

[0058] where Lg and Lc are the token sequences of the new template and the candidate template, respectively; in this embodiment, the threshold value of is set to 0.8, if the similarity exceeds the threshold value, it means that the existing template needs to be adjusted to adapt to the unmatched log. The corresponding fixed token in the candidate template needs to be replaced with wildcard to adapt to the new template; if the similarity is lower than the threshold value, it means that no template can approximately match the unmatched log. Therefore, a node path corresponding to the token sequence of the new template is added under the terminal node, and a leaf node is added at the end of the path to store the new template; after the adaptation is completed, the coverage of the prefix parse tree is expanded, providing more comprehensive template support for subsequent log parsing.

[0059] Figure 2 Fig. 1 shows the main framework of a large language model log parsing device based on relationship inference according to an embodiment of the present application, which fully presents the whole process logic of parsing the input log.

[0060] In the offline stage, the candidate sampling module is used to cluster and stratified sample the historical system logs to construct a candidate log set with diversity and representativeness.

[0061] The input log as raw data to be parsed first enters the prefix parsing tree parsing module. By means of token-by-token traversal matching, it is judged whether the input log can match the existing template. If the matching is successful, the structured log is directly output; if the matching fails, the unmatching log is transferred to the subsequent relationship inference module.

[0062] In the log relationship inference module, the unmatching log triggers two parallel processes of syntax relationship inference and semantic relationship inference. Among them, the syntax relationship inference module screens out log examples similar in syntax structure to the unmatching log based on the candidate log set constructed offline through the Jaccard coefficient; the semantic relationship inference module screens out log examples with high semantic correlation based on the historical log window through the large language model to infer the "causal strength" and "number of shared entities" between logs. Finally, the syntax similar log examples and the semantic related log examples jointly serve as "demonstration examples" to participate in the construction of subsequent prompt words.

[0063] Then, in the log template generation module, the "instruction", the "demonstration example" screened above, and the "question" are combined into a complete prompt word, which is input into the large language model module to generate a corresponding log template.

[0064] In the prefix tree update module, the newly generated log template reversely updates the prefix parsing tree, optimizes the structure of the prefix parsing tree by adapting the existing template or adding a template path, so that it can cover more log types and provide more comprehensive template support for the parsing of subsequent input logs.

[0065] The embodiment of the application provides a non-volatile computer storage medium, which stores at least one executable instruction. The computer executable instruction can execute the relationship inference-based large language model log parsing method in the above embodiment.

[0066] The executable instruction can be specifically used to make the processor perform the following operations:

[0067] First, in the offline stage, a candidate log set with diversity and representativeness is constructed by clustering and hierarchical sampling of historical system logs;

[0068] The prefix parsing tree is used as the core, and the input log is preprocessed and then traversed in the prefix parsing tree to realize fast template matching. If the matching is successful, the structured log is directly output; if the matching is unsuccessful, the subsequent process is entered;

[0069] The unmatched log will infer its relationship with other logs from the syntax and semantic dimensions. The syntax level calculates the syntax similarity of the unmatched log with the candidate log set, the semantic level infers the causal strength and the number of shared entities of the unmatched log with the logs in the historical log window through the large language model, and then combines the semantic composite score to screen out the most relevant examples;

[0070] The screened syntax similar and semantic related logs are used as examples to construct a complete prompt containing task instructions, demonstration examples and logs to be parsed, and a large language model is called to generate a structured template of the unmatched log;

[0071] Finally, the newly generated template is adapted to the prefix parse tree, and the tree structure is perfected through dynamic updating to provide more comprehensive template support for subsequent log parsing.

[0072] Figure 3 The structure schematic diagram of the computing device embodiment of the present application is shown, and the specific implementation of the computing device is not limited in the specific embodiment of the present application.

[0073] As shown in Figure 3 , the computing device can include:

[0074] a processor, a communications interface, a memory, and a communications bus.

[0075] Among them: the processor, the communications interface, and the memory complete the communication among each other through the communications bus. The communications interface is used for communication with network elements such as other devices, clients or other servers, etc. The processor is used for executing programs, and specifically can execute related steps in the above-mentioned one social media content propagation prediction method embodiment.

[0076] Specifically, the program can include program code, and the program code includes computer operation instructions.

[0077] The processor can be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the server can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0078] The memory is used to store programs. The memory can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0079] The program can specifically be used to cause the processor to perform the following operations:

[0080] In the offline stage, by performing a word segmentation operation on the historical system log, then clustering the log according to the high-frequency token and single-pass bucketing strategy, and then performing hierarchical quota sampling, a candidate log set with diversity and representativeness is constructed;

[0081] The prefix parse tree data stored in the memory is loaded, and the input log is preprocessed and then traversed and matched in the order of the token sequence. If the matching is successful, the structured log is directly output, and if the matching is not successful, the subsequent process is entered;

[0082] The unmatched log will infer its relationship with other logs from the syntax and semantics dimensions. The candidate log set data is called to calculate the syntax similarity through the Jaccard coefficient formula, and on the semantic level, the causal strength and the number of shared entities of the unmatched log and the logs in the historical log window are inferred by calling a large language model, and then the most relevant example is calculated through a semantic composite scoring formula;

[0083] The screened syntax similar and semantic related logs are taken as examples to construct a complete prompt containing task instructions, demonstration examples and logs to be parsed, and a large language model is called to generate a structured template of the unmatched log;

[0084] Finally, the newly generated template is adapted to the prefix parse tree, and the updated prefix parse tree data is written into the storage to provide more comprehensive template support for subsequent log parsing.

[0085] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with these teachings, based on the description as provided herein. In addition, the present embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings as described herein, and any references below to specific languages are provided for disclosure of enablement only.

[0086] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known methods, structures and techniques are not shown in detail in order not to obscure the understanding of the present specification.

[0087] Similarly, it is to be understood that the embodiments of the application can be alternately grouped together in a single embodiment, figure, or description thereof for the purpose of brevity and understanding in the foregoing detailed description of the exemplary embodiments of the application, various features of the embodiments of the application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the application requires more features than are explicitly recited in each claim. Rather, inventive aspects lie in less than all features of the single embodiments disclosed. Thus, the claims following, in which each claim recites only a few specific combinations of features, are to be understood as including all possible combinations of features present in one or more embodiments of the application. Each claim is hereby defined to include what is individually presented in the claim following, what is reasonably supported by the disclosure of the application, and what composition of matter or device is recited in the claim following.

[0088] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and further can be divided into multiple sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and all the processes or units of any method or apparatus disclosed in the specification can be adopted, except that at least some of such features and / or processes or units are mutually exclusive. Unless explicitly stated, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar function.

[0089] Furthermore, those skilled in the art will appreciate that a combination of features of different embodiments can mean being within the scope of the application and forming a different embodiment. For example, any one of the claimed embodiments can be used in any combination.

[0090] The various component embodiments of the application can be implemented in hardware, or as software modules running in one or more processors, or combinations thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some or all of the components according to the embodiments of the application. The application can also be implemented as a program for executing the part or all of the methods described herein on a computer or a processor (for example, a computer program and a computer program product). Such a program implementing the application can be stored on a computer readable medium, or can have the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0091] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps not listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for carrying out a certain function. The functions of the separate means can be carried out by one specific apparatus or means. The use of the terms 'first','second' and 'third' and the like in the description is only for distinguishing between similar elements, and is not meant to denote a physical or chronological priority. The terms 'comprise', 'comprising', 'comprises', 'including', 'include', 'includes' and the like are not used in their exclusive or exhaustive sense.< / end> < / start>

Claims

1. A log parsing method for large language models based on relation inference, characterized in that, include: Using a prefix parse tree as its core, the input log is preprocessed and then traversed to achieve template matching. This includes: inputting the tokens in the preprocessed log token sequence into the prefix parse tree sequentially for node matching. The first token is matched against the child nodes of the root node. If a corresponding ordinary token node or a wildcard node that matches the current token exists, the current node is entered, and the next token in the token sequence is matched against the child nodes of the current node. This token-by-token traversal matching operation is performed in a loop. When the traversal operation terminates, if the current token cannot match any child node of an internal node and a leaf node has not been reached, it is determined that the current input log cannot find a matching template in the existing prefix parse tree. When no template is matched, the relationship between the unmatched logs and logs in the candidate log set and the historical log window is inferred from both syntactic and semantic dimensions. Specifically, syntactic similarity is calculated between the unmatched logs and the candidate logs to filter out syntactically similar logs. Semantically, a large language model is used to infer the causal strength and the number of shared entities between the unmatched logs and the historical logs, and semantic composite scoring is combined to filter out semantically relevant logs. For unmatched logs, relationship inference is performed from both syntactic and semantic dimensions, and highly relevant demonstration examples are selected. Syntactic relationship inference directly calls the offline-built candidate log set, and the Jaccard coefficient is used to calculate the syntactic similarity between the unmatched logs and the candidate logs. With candidate logs Each word is segmented to obtain a token set. and Through the formula: Calculate the ratio of the number of elements in the intersection to the number of elements in the union of the two, and finally select the top-m syntactically similar logs based on the similarity results; Using the selected grammatically similar and semantically related logs as demonstration examples, a complete prompt containing task instructions, demonstration examples, and logs to be parsed is constructed, and a large language model is called to generate a structured template for the unmatched logs. The newly generated templates are adapted to the prefix parse tree, and the tree structure is dynamically updated. This includes: extracting all leaf nodes under the terminal node of the unmatched log in the prefix parse tree, and the templates corresponding to these leaf nodes constitute a candidate template set; then calculating the similarity between the generated new template and each candidate template, using the longest common subsequence length. If the similarity exceeds a threshold, it means that the existing template needs to be adjusted to adapt to the unmatched log, and the corresponding fixed token in the candidate template is replaced with the wildcard "<*>" to adapt to the new template; if the similarity is below the threshold, it means that no template can approximately match the unmatched log; adding a node path corresponding to the token sequence of the new template under the terminal node, and adding a leaf node at the end of the path to store the new template; after the adaptation is completed, the coverage of the prefix parse tree is expanded, providing more comprehensive template support for subsequent log parsing.

2. The log parsing method for a large language model based on relation inference according to claim 1, characterized in that, The method further includes: during the offline phase, constructing a candidate log set by clustering and hierarchical sampling of historical system logs.

3. The log parsing method for a large language model based on relation inference according to claim 1, characterized in that, The prefix parse tree consists of a root node, internal nodes, and leaf nodes: the root node is the starting point for log parsing; the internal nodes include ordinary token nodes representing constants and wildcard nodes used to match any token; each leaf node uniquely corresponds to a stored log template, and the path tokens from the root node to the leaf node are concatenated to form a complete log template string.

4. The log parsing method for a large language model based on relation inference according to claim 1, characterized in that, During the traversal, if a leaf node is reached, it indicates that the current input log has successfully matched the template corresponding to that leaf node.

5. The log parsing method for a large language model based on relation inference according to claim 1, characterized in that, The construct includes complete prompts such as task instructions, demonstration examples, and logs to be parsed, including: The prompt words are constructed based on context learning, and the prompt words follow the structure of "task instruction-demonstration example-query": first, the task instruction is clarified; then, the selected grammatically similar logs and semantically related logs are organized into demonstration examples in pairs of "log example-corresponding structured template"; finally, the unmatched logs are used as queries and embedded at the end of the prompt.

6. A log parsing device for a large language model based on relation inference, characterized in that, include: The prefix parse tree parsing module is used to perform template matching by traversing the prefix parse tree after preprocessing the input log. This includes: inputting the tokens in the preprocessed log token sequence into the prefix parse tree in sequence for node matching. The first token is matched with the child nodes of the root node. If a corresponding ordinary token node or a wildcard node that matches the current token exists, the current node is entered, and the next token in the token sequence is matched with the child nodes of the current node. This token-by-token traversal matching operation is performed in a loop. When the traversal operation terminates, if the current token cannot match any child node of the internal node and has not reached a leaf node, it is determined that the current input log cannot find a matching template in the existing prefix parse tree. The log relationship inference module is used to infer the relationship between unmatched logs and logs in the candidate log set and historical log window from both syntactic and semantic dimensions when no template is matched. Specifically, in the syntactic dimension, it calculates the syntactic similarity between unmatched logs and logs in the candidate log set, filtering out syntactically similar logs. In the semantic dimension, it infers the causal strength and the number of shared entities between unmatched logs and logs in the historical log window using a large language model, and then combines this with a semantic composite score to filter out semantically relevant logs. For unmatched logs, it performs relationship inference from both syntactic and semantic dimensions, filtering highly relevant demonstration examples. The syntactic relationship inference directly calls the offline-built candidate log set, using the Jaccard coefficient to calculate the syntactic similarity between unmatched logs and candidate logs. With candidate logs Each word is segmented to obtain a token set. and Through the formula: Calculate the ratio of the number of elements in the intersection to the number of elements in the union of the two, and finally select the top-m syntactically similar logs based on the similarity results; The log template generation module is used to construct a complete prompt containing task instructions, demonstration examples, and logs to be parsed by using filtered syntactically similar and semantically related logs as demonstration examples, and calling a large language model to generate a structured template for unmatched logs. The prefix parse tree update module is used to adapt newly generated templates to the prefix parse tree and dynamically update the tree structure. This includes: extracting all leaf nodes under the terminal node of the unmatched log in the prefix parse tree, and the templates corresponding to these leaf nodes constitute a candidate template set; then calculating the similarity between the generated new template and each candidate template, using the longest common subsequence length; if the similarity exceeds a threshold, it means that the existing template needs to be adjusted to adapt to the unmatched log, and the corresponding fixed token in the candidate template is replaced with the wildcard "<*>" to adapt to the new template; if the similarity is below the threshold, it means that no template can approximately match the unmatched log; adding a node path corresponding to the token sequence of the new template under the terminal node, and adding a leaf node at the end of the path to store the new template; after the adaptation is completed, the coverage of the prefix parse tree is expanded, providing more comprehensive template support for subsequent log parsing.

7. A computing device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of a large language model log parsing method based on relation inference as described in any one of claims 1-5.

8. A computer storage medium, characterized in that, The storage medium stores at least one executable instruction that causes the processor to perform the steps of a large language model log parsing method based on relation inference as described in any one of claims 1-5.

Images