Log analysis method and device based on large language model

Abstract

The invention provides a log analysis method and device based on a large language model, and the method comprises the steps: determining a sample log and an analysis requirement, inputting the sample log and the analysis requirement into a trained large language model, obtaining a log analysis step corresponding to the sample log based on the large language model, determining a sample log and a result structure corresponding to the analysis requirement based on the log analysis step; determining a representative log data set for generating a log template from the to-be-analyzed log data set; inputting the sample log, the analysis requirement, the log analysis step, the result structure and the representative log data set into a trained large language model to obtain a log template analysis result corresponding to the representative log data set; wherein the log template analysis result comprises a name attribute, a description attribute, a format attribute and a sample attribute; and storing the log template analysis result as template data based on the prefix tree data structure. The accuracy and efficiency of log analysis are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a log parsing method and device based on a large language model. Background Art

[0002] Logs are files that record important information such as system operation status, operation records, error messages, etc. They are important data sources in the fields of operation and maintenance management, troubleshooting, security monitoring, etc. With the rapid development of social information technology, the IT systems of modern enterprises, government departments and other social organizations are becoming more and more complex, and the amount of log data generated is also increasing rapidly.

[0003] Log parsing is to parse the structure of the log, determine its fixed pattern and variable part, and extract the log template. Existing log parsing methods usually rely on predefined rules or templates. These methods work well when processing logs with fixed formats and simple contents, but they are often unable to cope with large-scale log data that is complex, changeable, and has inconsistent formats. For example, existing log parsing methods use traditional rule matching methods, which require manual writing and maintenance of a large number of rules, lacking flexibility. As the system and business change, the rules also need to be updated frequently, which leads to high maintenance costs and low efficiency. In addition, existing log parsing methods have limited ability to identify complex log patterns and are prone to parsing errors, resulting in low log parsing accuracy. In addition to the above, existing log parsing methods cannot deeply parse log content when performing log parsing, and are limited to identifying fixed patterns and variable parts. In summary, log parsing methods in the prior art have technical problems of low parsing accuracy and low parsing efficiency. Therefore, how to improve the accuracy and efficiency of log parsing is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a log parsing method and device based on a large language model to eliminate the work of writing and maintaining rules in the log parsing process, improve the efficiency of log parsing, and use a thinking chain prompt project to improve the accuracy of log parsing.

[0005] One aspect of the present invention provides a log parsing method based on a large language model, the log parsing method comprising:

[0006] Determine a sample log and a parsing requirement, input the sample log and the parsing requirement into a trained large language model, obtain a log parsing step corresponding to the sample log based on the trained large language model, and determine a result structure corresponding to the sample log based on the log parsing step;

[0007] Determine a representative log data set for generating a log template from the log data set to be parsed;

[0008] Input the sample log, parsing requirements, log parsing steps, result structure and the representative log data set into the trained large language model to obtain a log template parsing result corresponding to the representative log data set; wherein the log template parsing result includes a name attribute, a description attribute, a format attribute and a sample attribute;

[0009] The log template parsing result is stored as template data based on the prefix tree data structure.

[0010] In some embodiments of the present invention, storing the log template parsing result as template data based on a prefix tree data structure includes:

[0011] Determining non-matching information through a regular expression matching algorithm based on the representative log data set and the log template parsing result;

[0012] Repairing the log template parsing result based on the mismatch information to obtain a repaired log template parsing result;

[0013] The repaired log template parsing result is stored as template data based on the prefix tree data structure.

[0014] In some embodiments of the present invention, determining a representative log data set for generating a log template from the log data set to be parsed includes:

[0015] A representative log data set for generating a log template is determined from the log data set to be parsed based on the similarity of each log data.

[0016] In some embodiments of the present invention, determining a representative log data set for generating a log template from the log data set to be parsed based on the similarity of each log data includes:

[0017] Dividing the plurality of log data in the to-be-parsed log data set into a plurality of subsets based on the length of the log data;

[0018] Performing similarity calculation on each log data in each of the subsets;

[0019] A representative log data set is generated based on the log data in each of the subsets whose similarity value is less than a preset threshold.

[0020] In some embodiments of the present invention, determining a representative log data set for generating a log template from the log data set to be parsed includes:

[0021] A representative log data set for generating a log template is determined from the log data set to be parsed based on a systematic sampling method.

[0022] In some embodiments of the present invention, the method further comprises:

[0023] Based on the template data, each log data in the to-be-parsed log data set is parsed online or offline.

[0024] In some embodiments of the present invention, when each log data in the to-be-parsed log data set is parsed online based on the template data, the log parsing method further includes:

[0025] determining first log data that does not match the template data;

[0026] Obtaining a log template parsing result corresponding to the first log data through the large language model based on the first log data;

[0027] The stored template data is updated based on the log template parsing result corresponding to the first log data.

[0028] Correspondingly, the present invention also discloses a log parsing system based on a large language model, including a processor, a memory and a computer program stored in the memory, wherein the processor is used to execute the computer program, and when the computer program is executed, the system implements the steps of the method described in any of the above embodiments.

[0029] According to another aspect of the present invention, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0030] According to yet another aspect of the present invention, a computer program product is disclosed, including a computer program, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0031] The log parsing method and device based on the large language model disclosed in the present invention firstly provide the log parsing steps for the parsing requirements based on the large language model, then generate the log template parsing results according to the received sample logs, parsing requirements, log parsing steps, result structure and the representative log data set, and the log template parsing results are stored as prefix trees. The method implements log parsing based on the large language model, which can automatically learn complex patterns and features from the data, thereby eliminating the workload of rule writing and maintenance in the log parsing process, thereby improving the efficiency of log parsing; in addition, the large language model is trained with a large amount of diversified data, can adapt to log data of different types and formats, has strong generalization ability and pattern recognition ability, and can improve the accuracy of log parsing.

[0032] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0033] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are only for illustrating the principles of the present invention. In order to facilitate the illustration and description of some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings:

[0035] Figure 1 The figure is a flow chart of a log parsing method based on a large language model according to an embodiment of the present invention.

[0036] Figure 2 The module calling flow chart of the log parsing system based on the large language model according to an embodiment of the present invention.

[0037] Figure 3 This is a composition structure diagram of the large language model thinking chain prompt words according to an embodiment of the present invention.

[0038] Figure 4 The figure is a flow chart of log parsing performed by a large language model according to an embodiment of the present invention.

[0039] Figure 5 The figure is a flow chart of determining a representative log data set based on a similarity calculation method according to an embodiment of the present invention.

[0040] Figure 6 The figure is a flow chart of log template repair according to an embodiment of the present invention.

[0041] Figure 7 The figure is a flow chart of storing a log template as a prefix tree according to an embodiment of the present invention.

[0042] Figure 8 FIG. 4 is a schematic diagram of a prefix tree according to an embodiment of the present invention.

[0043] Fig. 9 The figure is a schematic diagram of a process of updating a prefix tree in one embodiment of the present invention.

[0044] Fig.10 The figure is a flowchart of persistent storage of a prefix tree according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0046] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0047] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0048] It should also be noted here that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection with an intermediary, not only to a wired connection, but also to a wireless connection, and the specific connection can be changed based on the actual application scenario.

[0049] Large Language Models (LLMs) are a type of model that uses deep learning technology to complete a variety of natural language processing tasks by training on large-scale text data. The training of large language models usually involves billions to hundreds of billions of parameters, and has powerful language understanding and generation capabilities. It can be used for the analysis and processing of complex text data, which brings new possibilities for log parsing. The inventor found in the study that the application of large language models in log parsing realizes automatic recognition, which can eliminate the writing and maintenance of rules in the log parsing process, improve the efficiency of log parsing, and can use the thinking chain prompt engineering to improve the accuracy of log parsing.

[0050] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0051] Figure 1 FIG. 1 is a flow chart of a log parsing method based on a large language model according to an embodiment of the present invention. Figure 1 As shown, the log parsing method at least includes steps S10 to S40.

[0052] Step S10: Determine a sample log and analysis requirements, input the sample log and analysis requirements into a trained large language model, obtain a log analysis step corresponding to the sample log based on the trained large language model, and determine a result structure corresponding to the sample log based on the log analysis step.

[0053] Step S20: determining a representative log data set for generating a log template from the log data set to be parsed.

[0054] Step S30: Input the sample log, parsing requirements, log parsing steps, result structure and the representative log data set into the trained large language model to obtain the log template parsing result corresponding to the representative log data set; wherein the log template parsing result includes name attributes, description attributes, format attributes and sample attributes.

[0055] Step S40: storing the log template parsing result as template data based on the prefix tree data structure.

[0056] For a large language model, prompts directly affect the output and behavior of the model; the model understands the requirements of the task based on the provided prompts, and analyzes and solves the problem based on the content of the understanding. When using a large language model, in order to obtain high-quality and expected output, this application needs to construct appropriate prompts to guide the large language model; that is, this application is based on a large language model (such as GLM-4) and stimulates the ability of the large language model in the field of log parsing through a chain of thought prompt engineering. The Chain-of-Thought (coT) prompt of the large language model is an improved prompt technology. Compared with simple prompts, the chain of thought prompts can provide specific analysis and thinking steps for the large language model, which can be applied to any task that can be solved by language, such as mathematical problems, common sense reasoning, and symbolic operations. This application uses chain of thought prompts to enable the large language model to enhance its parsing ability and improve the accuracy of the log parsing results through intermediate step-by-step parsing steps before outputting the final result.

[0057] In the above step S10, a thought chain prompt is generated through a question-answer-question structure. First, a sample log is given and parsing requirements are proposed so that the large language model knows the specific task; then a log parsing step is constructed to guide the thinking analysis of the large language model (the log parsing step is as follows Figure 4 As shown), combined with sample logs, and define the result structure to guide the large language model to generate the results we need; finally, a specific representative log data set is given for the large language model to parse. Exemplarily, the result structure can be defined in JSON format, so that each log template output by the large language model is represented by a JSON object. The specific examples of thinking chain prompts are as follows:

[0058]

[0059]

[0060]

[0061] Figure 3 FIG. 1 is a composition structure diagram of a large language model thought chain prompt word according to an embodiment of the present invention. Figure 3 As shown, the thought chain prompts include parsing requirements, sample logs, parsing steps, result structures, and representative log data sets. The parsing requirements enable the large language model to know the specific tasks; the sample logs and result structures are used as learning samples to guide the large language model to perform log parsing. Exemplarily, the result structure includes two dimensions, the first dimension is the structure of the results returned by the large language model, and the second dimension is the parsing regulations for the log fields; Exemplarily, the result returned by the large language model is a JSON structure, and the JSON structure includes the name, description, format, and sample of the log template, that is, the log template parsing result specifically includes the name attribute, description attribute, format attribute, and sample attribute.

[0062] When determining the sample logs, specifically, the sample logs can be randomly selected from the log data set to be parsed based on the random sampling method; it is understandable that if there are too few sample logs, the result structure cannot be well defined, and if there are too many sample logs, it is easy to cause hallucinations in the large language model, causing the large language model to directly use the parsing results of the sample logs as the parsing results of the representative log data set, thereby reducing the quality of the log parsing results; therefore, 2-3 log data can be selected from the log data set to be parsed as sample logs based on the random sampling method.

[0063] In a specific embodiment, a self-prompting method can be used to obtain steps for guiding the large language model to perform log parsing. The prompt words for obtaining the log parsing steps based on the self-prompting method are shown in the following table:

[0064]

[0065] Based on the above prompt words, the large language model can give specific log parsing steps, such as understanding the log structure, identifying fixed patterns, determining variable parts, building templates, and verifying templates.

[0066] In the above step S20, a representative log data set is generated based on the log data set to be parsed, and the log template generated based on the representative log data set is used to parse each log in the log data set to be parsed. The selected representative log data should be as diverse as possible, so that the log sample can contain more log templates, and the log template parsed by the large language model is also more comprehensive.

[0067] In addition, if the number of log data in the log data set to be parsed is not large, the log data set to be parsed as a whole can also be used as a representative log data set; at this time, when generating a log template based on a large language model, the log data set to be parsed is traversed, and all the logs to be parsed are sent to the large language model for parsing to obtain the log template corresponding to each log data. However, the number of logs generated by IT systems in today's era is increasing. Faced with the number of logs in the millions or even tens of millions, it takes a long time and consumes more resources to traverse all the data in the log data set to be parsed and send it to the large language model to complete the log template parsing. Therefore, when there are more log data in the log data set to be parsed, a variety of log samples can be selected from the log data set to be parsed and sent to the large language model to generate a variety of log templates. Exemplarily, a representative log data set for generating a log template can be determined from the log data set to be parsed based on the similarity of each log data.

[0068] Furthermore, based on the similarity of each log data, a representative log data set for generating a log template is determined from the log data set to be parsed, specifically including: dividing the multiple log data in the log data set to be parsed into multiple subsets based on the length of the log data; performing similarity calculation on each log data in each of the subsets; and generating a representative log data set based on the log data in each of the subsets whose similarity value is less than a preset threshold.

[0069] Figure 5 FIG. 1 is a flow chart of determining a representative log data set based on a similarity calculation method according to a specific embodiment of the present invention. Figure 5 As shown, firstly, the log data set to be parsed is divided according to the log length, that is, the log data of the same length is divided into the same set, thereby obtaining multiple subsets; since the parameter length of the variable part in the log data is not fixed, the length here refers to the length of the field list formed after the log is split according to the space. In this embodiment, the logs belonging to the same log template have the same length, and the embodiment based on the division of subsets can reduce the data scale of the log set and facilitate subsequent operations; in addition, if the data scale of the log subset after division is still large, it is necessary to process the logs in the log subset in batches to reduce the processing time and processing complexity.

[0070] Further calculate the similarity of all logs in each log subset, and initialize the representative log data set to an empty set; logs with lower similarity are more likely to belong to different log templates, and selecting log data with lower similarity can increase the diversity of representative log data. When calculating the similarity, the log subset can be converted into a feature matrix through the TF-IDF algorithm, and the cosine similarity is used to calculate the similarity between logs. In addition, in order to reduce the complexity of the calculation, the feature matrix can be first reduced in dimension. In a specific embodiment of determining a representative log data set, log data can be read one by one. If the similarity between the current log data and all logs in the representative log data set is lower than a threshold, the log data is added to the representative log data set. Based on the above steps, all log subsets are traversed, and the representative log data obtained in each traversal is spliced ​​to obtain the final representative log data set. In addition, if there are too many logs in the representative log data set, it means that the threshold is set too high, so that logs belonging to the same log template are added to the set; at this time, the threshold can be lowered and the representative log data set can be traversed; since the similarity of logs belonging to the same log template will be higher than the threshold, one or several logs are retained and the remaining logs are removed from the representative log data set.

[0071] In embodiments where the log data processing efficiency is high or the number of logs in the log data set to be parsed is huge, a representative log data set for generating a log template can also be determined from the log data set to be parsed based on a systematic sampling method. In this embodiment, the log data set to be parsed is first divided according to the log length, and then the systematic sampling method is used to obtain representative log data for each divided log subset; finally, all representative log data are spliced ​​to obtain the final representative log data set. The systematic sampling method is also called equidistant sampling, which extracts samples at a certain interval and is an effective sampling method; considering that the positions of log data belonging to the same log template in the log subset are more likely to be adjacent or close, especially in batch tasks, periodic tasks, and logs triggered by specific events, for example, during network attacks or system failures, the system may generate a large number of error logs with the same template; and the systematic sampling method is a sampling method that ensures the diversity of samples in the representative log data set on the basis of improving processing efficiency.

[0072] In the above step S30, the sample log, parsing requirements, log parsing steps, result structure and the representative log data set are all input into the large language model, so that the large language model, under the guidance of the parsing steps and the result structure, thinks and analyzes according to the parsing steps, forms a transparent thinking chain to parse the specific log and returns the parsing result in the specified format according to the result structure. Compared with the traditional log parsing method, the log parsing method based on the large language model of the present application does not require domain knowledge to write regular expressions, but only needs to list examples to guide the large language model; for example, for the parsing of time, the parsing is guided by giving examples in the result structure of the prompt word: the time field 13:14:15 can be parsed as the time field, or parsed as the three fields of hour:minute:second.

[0073] In some embodiments of the present invention, the log template parsing result is stored as template data based on a prefix tree data structure, which may specifically include: determining non-matching information through a regular expression matching algorithm based on the representative log data set and the log template parsing result; repairing the log template parsing result based on the non-matching information to obtain a repaired log template parsing result; and storing the repaired log template parsing result as template data based on the prefix tree data structure.

[0074] For the log template JSON object list output by the large language model, the format (format attribute) may not match the example (sample attribute) due to deviation; for such JSON objects, the template automatic repair algorithm can correct the format attribute to match the example attribute. The flowchart of the template automatic repair algorithm is as follows Figure 6 As shown in the figure, the matching rule between the log template format attribute and the log is the matching rule of the regular expression. The variables wrapped in <::> in the log template will be replaced with ".*?" in the regular expression to match any character in a non-greedy way. For example, for the log: "[Mon Dec0514:11:43 2005][error][client 141.154.18.244]Directory index forbidden byrule: / var / www / html / ", the large language model can parse this log and return either a matching log template or an unmatched log template. The two unmatched log templates are as follows:

[0075] [<:week:><:month:><:day:><:time:><:year:>][<:log_level:>][client<:client_ip:>]Directory index forbidden by rule:<:directory_path:>

[0076] [<:week:><:month:><:day:><:time:><:year:>][<:log_level:>][client<:client_ip:>]]Directory index forbidden by rule:<:directory_path:>

[0077] For the first mismatched template mentioned above, the reason for the mismatch is the redundant space before the right bracket. The error point is [client<:client_ip:>], where there is an extra space before the right bracket. The specific reason is that <:client ip:> will match the characters 141.154.18.244] in the log. The next character of the log template is ']', and the next character of the log is 'D' in Directory, which will make the log mismatch with the log template generated by the large language model.

[0078] For the second template that does not match, the reason for the mismatch is the lack of space and the presence of redundant characters. The error is that there is no space before the characters [client<:client_ip:>]], and there is a redundant character ']' after it. For the above two errors, the following log template can be repaired through the template automatic repair algorithm to match the log:

[0079] [<:week:><:month:><:day:><:time:><:year:>][<:log_level:>][client<:client_ip:>]Directory index forbidden by rule:<:directory_path:>.

[0080] For example, Figure 6This is a flow chart of log template repair according to an embodiment of the present invention. In this embodiment, the template automatic repair algorithm adopts the idea of ​​double pointers; in the initial state, the log character pointer points to the first character of the log, and the template character pointer points to the first character of the template. If the log character is equal to the template character, the template character pointer and the log character pointer are moved to point to the next character at the same time, and the log and template are traversed to find the mismatch and repair them according to different situations. The matching rules between templates and logs are the matching rules of regular expressions, and the variables wrapped in <::> in the template will be replaced with “.*?” in the regular expression to non-greedily match any character as few times as possible. Common situations where template characters and log characters do not match and the corresponding repair methods are as follows:

[0081] 1. The current log character is a blank character:

[0082] 1.1 The current template character is equal to the next non-blank log character; the error reason is that the log template lacks a blank character. You need to add a blank character to the log template and move the log character pointer to point to the next log character. The template character pointer does not move.

[0083] 1.2 The current template character is not equal to the next non-blank log character; the error reason is that redundant characters appear in the log template. The redundant characters need to be deleted and the template character pointer is moved to point to the next template character. The log character pointer remains unchanged.

[0084] 2 The current log character is not a blank character:

[0085] 2.1 The current template character is a blank character:

[0086] 2.1.1 The current log character is equal to the next non-blank template character; the error reason is that redundant blank characters appear in the log template. The redundant blank characters need to be deleted and the template character pointer is moved to point to the next template character. The log character pointer does not move.

[0087] 2.1.2 The current log character is not equal to the next non-blank template character; the error reason is that the log template is missing characters. You need to add the missing characters to the log template and move the log character pointer to point to the next log character. The template character pointer does not move.

[0088] 2.2 The current template character is not a blank character:

[0089] 2.2.1 The current template character is an independent special character. Special characters are defined as non-letters, numbers, and underscores in the log, usually used to wrap other characters, such as the right bracket ']' and the right parenthesis ')'. Independence means that there are spaces before and after the character. Large language models may add spaces before these characters. For example, in the field [141.154.18.244] in the log, the log template may add spaces before ']' to change it to [141.154.18.244], which will cause the log characters to not match the template characters. In this case, you need to delete the extra spaces and move the template character pointer to point to the next non-blank template character. The log character pointer does not move.

[0090] 2.2.2 The current template character is not an independent special character; the error reason is that the template character change is not equal to the log character. The template character needs to be changed to the log character, and the template character pointer and the log character pointer need to be moved to point to the next character at the same time.

[0091] In the above embodiment, the log template output by the large language model is repaired based on the template automatic repair algorithm, thereby improving the matching degree between the log template and the log data.

[0092] In addition, after the present application obtains the log template based on the large language model, it can also perform online or offline analysis on each log data in the set of log data to be analyzed based on the template data. In addition, when performing online analysis on each log data in the set of log data to be analyzed based on the template data, the log analysis method also includes: determining the first log data that does not match the template data; obtaining a log template analysis result corresponding to the first log data based on the large language model based on the first log data; and updating the stored template data based on the log template analysis result corresponding to the first log data. In this embodiment, if it is determined that the log data does not match all log templates, a log template matching the log data is further generated based on the large language model and stored.

[0093] In the above step S40, the log template is stored using a prefix tree, and the log can be matched efficiently later. When multiple log templates have the same prefix, the storage space can be saved by sharing the common prefix.

[0094] Two classes are needed to construct a prefix tree for storing log templates; one is the tree node class TrieNode, whose attributes include children and template; the children attribute is a dictionary type, which is used to store the child nodes of the current node; the key of the dictionary is the string corresponding to the child node (the field of the format attribute separated by spaces in the above embodiment), and the value of the dictionary is the "TrieNode" instance of the child node. The other is the prefix tree class Trie, which is used to store and retrieve template sets; the prefix tree is an ordered tree, in which each node contains a string of the format attribute separated by spaces, and the strings on the path from the root node to the leaf node are connected to correspond to a template; the Trie class implements the basic functions of the prefix tree, including insertion, query and traversal; to determine whether a log matches an existing template in the prefix tree, the matching function of the prefix tree can be used to determine; if it does not match, a thinking chain prompt word is constructed and sent to the large language model to obtain a new log template; when a new log template is added, the method of inserting a node is called to add the new log template to the Trie. In order to achieve offline and online parsing, the prefix tree needs to be converted into JSON format and stored persistently in a file. When the log data needs to be analyzed online, the tree structure is directly read from the file to generate the tree. The prefix tree persistence access process is as follows: Fig.10 shown.

[0095] The prefix tree corresponding to a specific log template is as follows Figure 8 As shown in the figure, the log template corresponding to the prefix tree is: 1.<:time:><:hostname:>sshd(<:module:>)[<:pid:>]:check pass; user unknown; 2.<:time:><:hostname:>su(<:module:>)[<:pid:>]:session opened for user<:user:>; 3.<:time:><:hostname:>su(<:module:>)[<:pid:>]:session closed for user<:user:>.

[0096] In addition, when performing online parsing based on the prefix tree, first determine whether the log matches a log template in the prefix tree. If it matches, use the matching template as the log template for the log data. The matching rules are: first split the log string into a parts list according to spaces, use the root node of the prefix tree as the current node, and recursively match it; traverse the child nodes of the current node to determine whether there is a child node that matches the current part. If not, determine that the log string does not match the prefix tree. If so, move the current node to the child node that matches the current part and move the current part to the next part; if there is no next part, it means that all parts of the log have been processed. At this time, the template attribute of the current node is not empty, and the match is determined to be successful. If there is no match, build a thinking chain prompt word to send a request to the large language model to obtain the template JSON object of the log, and update the log template to the prefix tree to complete the online update of the log template. The process of updating the prefix tree is as follows: Fig. 9 shown.

[0097] In addition, the log template JSON object list is generated by parsing the log collection by the large language model. Each item in the list is a JSON object containing the name, description, format, and sample of the log template. When storing the log template, traverse the list and insert the log template into the prefix tree. Through the prefix tree storage, the log template based on the prefix tree storage can further improve the reading efficiency when parsing the log data offline and online.

[0098] Figure 7 FIG. 1 is a flow chart of storing a log template as a prefix tree according to an embodiment of the present invention. Figure 7 As shown, when storing the log template, first split the "format" in the template JSON object by spaces to form a field list, starting with the root node of the tree as the current node; determine whether the current field is in the child nodes of the current node; if so, move the current node to the child node that matches the current field; if not, create a child node with the current field as the key, and then move the current node. Determine whether there is the next field in the field list, if so, move the current field to the next field, if not, assign the JSON object of the log template to the template attribute of the current node. Determine whether there is the next template JSON object in the log template JSON object list, if so, repeat the above operation, if not, persist the prefix tree.

[0099] In addition, in order to build a prefix tree in an offline state and use the prefix tree to parse logs in an online state, the following methods need to be further defined in the prefix tree:

[0100] 1. insert method. This method receives the log template JSON object and its format attribute as parameters, and is used to insert the log template into the prefix tree. First, split the format attribute into a parts list by spaces; then traverse the parts, and for each part in the parts list, if the part is in the children of the current node, move the current node to the child node where the part is located; if the part is not in the children of the current node, first create a new TrieNode and add it to the children, and then move the current node. After traversing the parts list, assign the log template JSON object to the template attribute of the current node.

[0101] 2.match_log method. This method receives a log string as a parameter and determines whether the log matches the template contained in the prefix tree. The matching rules are as follows: first, split the log string into a parts list according to spaces, and recursively match it with the root node of the prefix tree as the current node; traverse the child nodes of the current node to determine whether there is a child node that matches the current part. If not, it is determined that the log string does not match the prefix tree. If so, move the current node to the child node that matches the current part and move the current part to the next part. If there is no next part, it means that all parts of the log have been processed. At this time, the template attribute of the current node is not empty, and the match is determined to be successful.

[0102] 3.save_to_file method. This method receives a file path as a parameter and is used to persist the prefix tree to the specified file.

[0103] 4. load_from_file method. This method receives a file path as a parameter and is used to read and generate a prefix tree from the specified file.

[0104] Correspondingly, the present invention also provides a log parsing system based on a large language model, including a processor, a memory and a computer program stored in the memory, wherein the processor is used to execute the computer program, and when the computer program is executed, the system implements the steps of the method described in any of the above embodiments.

[0105] Figure 2 FIG. 1 is a module call flow chart of a log parsing system based on a large language model according to an embodiment of the present invention. Figure 2As shown, the system can realize online parsing and offline parsing; during offline parsing, the log data set to be parsed is first loaded based on the log loading module, and then the representative log data set is determined based on the log sample selection module, and then the log template is parsed based on the large language model, and the log template parsing result output by the large language model is repaired through the template automatic repair module, and the repaired log template is further sent to the prefix tree storage module for storage. During online parsing, the stored prefix tree is loaded based on the prefix tree loading module, and then the log data is matched with the log template in the prefix tree based on the log matching module. If the log data does not match all log templates, the log data is sent to the large language module for parsing to obtain the log template corresponding to the log data, and similarly, the template is repaired based on the template automatic repair module, and the repaired log template is further stored through the prefix tree storage module.

[0106] Through the above embodiments, it can be found that the present application uses a diversity sample selection algorithm based on similarity calculation to obtain a diversity of log samples to generate a representative log data set, which is used as the log to be parsed each time sent to the large language model for parsing the log template. The thinking chain prompts for log parsing include parsing requirements, sample logs, parsing steps, result structure, and representative log data set; that is, through the parsing steps and result structure in the prompts, the thinking analysis of the large language model is guided, and the log template parsing results expected by the user are given.

[0107] In addition, the log parsing of the present invention is divided into two scenarios: offline and online. For large-scale existing logs, log templates can be generated in batches through offline log parsing and stored persistently through prefix trees. For instantly generated logs, log templates are obtained by loading prefix trees, and online log parsing is performed and the prefix tree can be updated in time. For deviated log templates generated by large language models, they can be corrected through the log template automatic repair algorithm and then inserted into the prefix tree after correction.

[0108] The embodiments of the present invention also provide a computer-readable storage medium and a computer program product, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0109] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0110] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0111] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0112] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A log parsing method based on a large language model, characterized in that: The log parsing method comprises: Determine a sample log and a parsing requirement, input the sample log and the parsing requirement into a trained large language model, obtain a log parsing step corresponding to the sample log based on the trained large language model, and determine a result structure corresponding to the sample log based on the log parsing step; Determine a representative log data set for generating a log template from the log data set to be parsed; Input the sample log, parsing requirements, log parsing steps, result structure and the representative log data set into the trained large language model to obtain a log template parsing result corresponding to the representative log data set; wherein the log template parsing result includes a name attribute, a description attribute, a format attribute and a sample attribute; The log template parsing result is stored as template data based on the prefix tree data structure.

2. The log parsing method based on a large language model according to claim 1, characterized in that: The log template parsing result is stored as template data based on the prefix tree data structure, including: Determining non-matching information through a regular expression matching algorithm based on the representative log data set and the log template parsing result; Repairing the log template parsing result based on the mismatch information to obtain a repaired log template parsing result; The repaired log template parsing result is stored as template data based on the prefix tree data structure.

3. The log parsing method based on a large language model according to claim 1, characterized in that: Determine a representative log data set for generating a log template from the log data set to be parsed, including: A representative log data set for generating a log template is determined from the log data set to be parsed based on the similarity of each log data.

4. The log parsing method based on a large language model according to claim 3 is characterized in that: Determining a representative log data set for generating a log template from the log data set to be parsed based on the similarity of each log data, including: Dividing the plurality of log data in the to-be-parsed log data set into a plurality of subsets based on the length of the log data; Performing similarity calculation on each log data in each of the subsets; A representative log data set is generated based on the log data in each of the subsets whose similarity value is less than a preset threshold.

5. The log parsing method based on a large language model according to claim 1, characterized in that: Determine a representative log data set for generating a log template from the log data set to be parsed, including: A representative log data set for generating a log template is determined from the log data set to be parsed based on a systematic sampling method.

6. The log parsing method based on a large language model according to claim 1, characterized in that: The method further comprises: Based on the template data, each log data in the to-be-parsed log data set is parsed online or offline.

7. The log parsing method based on a large language model according to claim 6, characterized in that: When each log data in the to-be-parsed log data set is parsed online based on the template data, the log parsing method further includes: determining first log data that does not match the template data; Obtaining a log template parsing result corresponding to the first log data through the large language model based on the first log data; The stored template data is updated based on the log template parsing result corresponding to the first log data.

8. A log parsing system based on a large language model, comprising a processor, a memory, and a computer program stored in the memory, characterized in that: The processor is used to execute the computer program. When the computer program is executed, the system implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images